Trace
01What was the question?
A panel can end on the right frame and still have jumped on the way there
Screenshot tests compare where two builds end up. In the recorded session at the top of this story the broken build opens its panel 80 px to the right of its button and slides home in 240 ms, and the last frame of the two builds is the same, pixel for pixel. So I built Trace to watch the whole way. These are the sixteen frames of that opening, the working build first and the broken one second, above the distance between the two panels, in px.
- 80px
- 58
- 39
- 25
- 17
- 11
- 7.7
- 5.3
- 3.5
- 2.3
- 1.4
- 0.8
- 0.4
- 0.2
- 0
- 0
02Who needs the answer?
An engineer changing a component, and so far nobody but me
02Who needs the answer?
An engineer changing a component, and so far nobody but me
They have a branch that touches a menu, a dialog or a drawer. They write down what the user does and what must stay true, and Trace replays that on both builds. I wrote every component it has run on.
The report it writes opens on one sentence and the requirement that broke, then both builds at that moment.
Under them is every requirement, held or broken on each build.
03What did you expect?
I expected requirements to catch bugs that comparing two runs would miss, and that did not hold
The control that flags any difference found fewer bugs than Trace
The struck sentence is from my first report, on 6 November 2026. Requirements judge each build without looking at the other, so I expected them to find bugs a comparison could not see. The same day a second reviewer found that the control read only 3 of the recorded channels, and once it read all of them it caught all 18 seeded bugs, the same as Trace.
04What was the first design?
I threw out a timeline that showed everything and stated nothing, and the page now opens on the failure
04What was the first design?
I threw out a timeline that showed everything and stated nothing, and the page now opens on the failure
The report was settled early, once I threw out a timeline that showed everything and stated nothing. The project page took four passes in four days, and the first, on 6 November, led with a headline and four tiles of numbers.
The replay sat under the tiles, below the first screen.
Now the page opens on the failure, the film at the top of this story, and further down each build is judged on its own against what must stay true.
A pair of builds can give four verdicts, and only one of them is a regression.
05What broke it?
A second reviewer found 25 issues, and then 7 regressions in my fixes
The reviewer worked from a clean clone with none of my conclusions, and one of the regressions could delete a user's files.
The score I liked was 18 of 18 seeded bugs caught, and it could not carry what I had put on it. I wrote those 31 builds and tuned against them, and the first run caught 12.
06How was it checked?
I checked it on unseen builds, random sessions and a cut-down failure
06How was it checked?
I checked it on unseen builds, random sessions and a cut-down failure
A fourth family of builds was written after the oracle stopped changing, labelled before any detector saw it, and run once. Trace caught all 6 of its seeded bugs, and wrongly flagged 1 of its 4 clean builds.
The reducer cut the 14 actions behind this page to 4, and those 4 reproduced the failure in 10 of 10 fresh runs.
Random sessions on builds I had labelled correct raised 2 flags in 295, and both were a real bug in my own fixture.
07What came of it?
It catches the bugs I seeded, and what it adds is the clean builds it leaves alone
0 of 13
clean builds Trace flagged on the corpus I tuned, every one a legitimate change
9 of 13
flagged by a plain comparison, which caught all 18 seeded bugs too
On builds I wrote, requirements catch the seeded bugs and leave most legitimate changes alone. The numbers cannot say how it does on anyone else's code.
08What would you do next?
Next is measuring the final oracle on builds it has never seen
That needs a new family of builds written after the last change. Then I would fix the stuck dialog so that correct means correct, let a requirement state a design range, and watch someone other than me use it.