Your first frame
IMPORTANT
This step
- Teaches: each trial in turn, levels, open checks, and carried.
- Frames:
upgrade.jsonandrung1.jsontorung6.json: dependencies, at the grain of one changelog line. - After: In plain words, the idea without the terms. This is the first step of the ladder.
Gadfly asks one question about one decision: does the decision rest on the evidence you think it rests on? It cannot look inside the reader, so it changes the evidence in fixed ways, asks again, and compares what comes back.
This page climbs one real experiment a step at a time. A dependency bot opens a pull request that bumps a made-up library, quill, from 2.4.1 to 3.0.0. Our code calls quill.retry_after(), and quill 3.0.0's changelog removes it.
The reviewer is Gemma 4 26B, served on this machine by eGemma. Does its decision to hold the upgrade rest on that changelog line?
Each step below is a frame in examples/ (rung1.json to rung6.json, each the one before with one more trial), read in five fresh runs by Gemma. The predictions were written in examples/PREDICTIONS.md before the first call, and every record is kept in examples/runs/. So every output on this page is read back from those records, and reading it needs no model.
1. The evidence and the question
The frame is the whole experiment, written down before any call. Here it is in words, in the order a run uses it, each span linked to its line in the file:
The claim
| Part | In this frame |
|---|---|
| Relation | the decision to hold the upgrade rests on the changelog line that removes retry_after() |
| Grain | one changelog line |
| Promise | the reviewer holds the upgrade while the changelog removes a function our code calls; without that line they merge or say unclear, never hold for another reason |
What the reader is shown, the anchor marked *:
| Span | Kind | Text |
|---|---|---|
use | code | src/billing/retry.py calls quill.retry_after(response) to wait before retrying a 429 response. |
get | code | src/billing/client.py calls quill.get(url, timeout=10) for every request. |
removed * | changelog | quill 3.0.0 changelog, Breaking: retry_after() is removed; use Response.backoff() instead. |
http3 | changelog | quill 3.0.0 changelog, Added: HTTP/3 support. |
leak | changelog | quill 3.0.0 changelog, Fixed: a memory leak in the connection pool. |
typos | changelog | quill 3.0.0 changelog, Docs: corrected typos in the README. |
ci | ci | CI on this pull request: all 214 tests passed; no test exercises the retry path. |
What the reader is asked
| Part | In this frame |
|---|---|
| Question | Dependabot opened pull request #212: bump quill from 2.4.1 to 3.0.0 in requirements.txt. Can it be merged as is? |
| Answers | merge, hold or unclear |
| Instruction | You review dependency upgrade pull requests for this repository. Use only the evidence given. Answer merge if the upgrade can be merged without changing our code, hold if our code must change before it can be merged, unclear if the evidence does not decide. |
The trials, each asked 5 times, 40 calls in all:
| Trial | What changes | What it tests |
|---|---|---|
baseline | nothing | the answer the others are read against |
repeat | nothing, asked again | whether that answer is stable |
ablation | cuts removed | whether the answer needs the anchor |
sham-1 | cuts leak | whether any cut of the same kind moves it |
sham-2 | cuts typos | whether any cut of the same kind moves it |
presentation | renames backoff to pause, quill to heron, retry_after to wait_hint | whether the wording, not the meaning, drives it |
bypass | cuts use | whether a hidden fallback still carries it |
control | cuts removed, use | whether the reader can move at all |
How the answers are read
| Rule | In this frame |
|---|---|
| Without the anchor | merge or unclear may come back |
| A level | 4 of 5 answers alike |
| Carriage | all-checks, so a category waits for the checks it needs |
| Materialized | the answer is the form the recipient receives |
| Battery | egemma |
The setting, hashed with the rest:
| Part | In this frame |
|---|---|
| Role | the reviewer of dependency upgrades |
| Locus | the evidence a reviewer has on the pull request: our code, the library's changelog, the CI result |
| Purpose | merge an upgrade only when our code needs no change for it |
| Trajectory | pull request, evidence, the reviewer's decision, the merge button |
| Crossing | the decision posted on the pull request |
| Baseline | the pull request's evidence as it stands; nothing changed |
- The claim (
relation,grain,promise) says what is being claimed, at which unit, and what should come back. It is hashed with everything else, the setting included, so a frame cannot be reworded after its runs: records from another wording are refused. - The evidence (
body.spans) is one sentence per span. Itskindgroups spans for the sham trial (section 4). - The anchor (
anchors), marked*, is the span the decision is supposed to rest on: the changelog line that removesretry_after(). - The question (
inputs) is what the reader is asked, and the answers and the instruction (observe) say what each answer means. Define every label in the instruction. A reader left to guess what a label means answers a different question.
What Gemma receives per call is the instruction, then the spans one per line, a line ---, and the question. It must answer with one label. Each trial is asked quorum times (5) and needs agree (4) answers alike; that answer is the trial's level.
2. Cut the line: ablation
Rung 1 declares only the ablation, which removes the anchor.
$ uv run python examples/read.py examples/rung1.json
quill-upgrade-rung-1: 5 kept runs, read by google/gemma-4-26B-A4B-it-qat-q4_0-gguf@d1c082be
trial pr
baseline hold 25
ablation merge 25
run 1: unresolved
run 2: unresolved
run 3: unresolved
run 4: unresolved
run 5: unresolvedEvery call held while the line was there, and merged once it was gone. That move is necessary but not enough to call the line load-bearing. The verdict says so, and lists what it still needs:
$ uv run python examples/read.py \
examples/rung1.json --verdict 1
pr: unresolved
levels: baseline hold, ablation merge
evidence: ablation moved it hold -> merge
evidence: not carried while 3 carriage checks are open
open: no hidden fallback (no bypass trial was declared)
open: presentation invariance (no presentation trial was declared)
open: specificity (no sham trial was declared)Each open: line is a check no trial has made yet. Under the default carriage: all-checks, a finding is not carried while one is open.
3. Ask again: repeat
A level is worth comparing only if it is stable. Repeat asks the unchanged evidence again; if it gives another level, the verdict is unresolved before anything else is read.
$ uv run python examples/read.py examples/rung2.json
quill-upgrade-rung-2: 5 kept runs, read by google/gemma-4-26B-A4B-it-qat-q4_0-gguf@d1c082be
trial pr
baseline hold 25
repeat hold 25
ablation merge 25
run 1: unresolved
run 2: unresolved
run 3: unresolved
run 4: unresolved
run 5: unresolvedGemma gave the same answer on all 25 calls. On clear evidence this reader is close to deterministic; Quorum shows where it is not.
4. Cut something harmless: shams
Perhaps any cut moves the reader. A sham removes a span of the anchor's kind instead (here another changelog line, the closest in length), which should change nothing. Two shams are drawn.
$ uv run python examples/read.py examples/rung3.json
quill-upgrade-rung-3: 5 kept runs, read by google/gemma-4-26B-A4B-it-qat-q4_0-gguf@d1c082be
trial pr
baseline hold 25
repeat hold 25
ablation merge 25
sham-1 hold 25
sham-2 hold 25
run 1: unresolved
run 2: unresolved
run 3: unresolved
run 4: unresolved
run 5: unresolvedBoth held, so the move is specific to the line: two checks remain open.
5. Rename: presentation
A reader might react to names rather than meaning. Presentation renames quill to heron, retry_after to wait_hint and backoff to pause, in one pass, in the evidence and the question alike.
$ uv run python examples/read.py examples/rung4.json
quill-upgrade-rung-4: 5 kept runs, read by google/gemma-4-26B-A4B-it-qat-q4_0-gguf@d1c082be
trial pr
baseline hold 25
repeat hold 25
ablation merge 25
sham-1 hold 25
sham-2 hold 25
presentation hold 25
run 1: unresolved
run 2: unresolved
run 3: unresolved
run 4: unresolved
run 5: unresolvedIt held. One check is left: whether something else would give the same answer without the line's connection to our code.
6. Cut the connection: bypass
The line matters only because our code calls the function it removes. The bypass cuts our call to retry_after() and keeps the line. A reader that reacts to the word "Breaking" would still hold, a hidden fallback; one that follows the connection would merge or say unclear.
$ uv run python examples/read.py \
examples/rung5.json --verdict 1
pr: carried
levels: baseline hold, repeat hold, ablation merge, sham-1 hold, sham-2 hold, presentation hold, bypass merge
evidence: ablation moved it hold -> merge; held under presentation, 2 shams
evidence: bypass moved it hold -> merge, so no hidden fallbackThe prediction was a fallback, at about 55%. It was wrong: Gemma merged on every call. Every carriage check has passed, so the finding is carried.
7. Can it move at all: the control
Carried does not need a control; cooled does. A reader that never moves would make every relation look cooled, so the control removes what any reader needs (here the line and our call together) to show that this one can move.
$ uv run python examples/read.py \
examples/rung6.json --verdict 1
pr: carried
levels: baseline hold, repeat hold, ablation merge, sham-1 hold, sham-2 hold, presentation hold, bypass merge, control merge
evidence: ablation moved it hold -> merge; held under presentation, 2 shams
evidence: bypass moved it hold -> merge, so no hidden fallback
evidence: the control moved it hold -> mergeThe next step of the ladder, Cooled, is where a control decides a verdict.
8. What "carried" says here
On this evidence, at this grain, for this model at this revision, the decision to hold rests on the changelog line, through our call to the function it removes: cut the line or the call and the decision changes; cut a neighbour or rename the names and it does not. It does not say that Gemma reviews upgrades well, or that it would on other evidence.
9. Read it fresh, and write your own
With eGemma on this machine and its key in EGEMMA_API_KEY or EGEMMA_API_KEY_FILE, these commands check the frame, plan it, run it fresh and read the verdict:
uv run gadfly check-frame examples/upgrade.json
uv run gadfly plan examples/upgrade.json
uv run gadfly run examples/upgrade.json \
--battery midwife.batteries.egemma:EGemma \
--calls 40 --out my-run.jsonl
uv run gadfly report examples/upgrade.json my-run.jsonlcheck-frame checks the frame's shape and then the parser, and plan shows the trials and the call count; neither makes a call.
To test your own decision, copy upgrade.json, replace the situation, body, anchor, question and labels, and run check-frame until it is accepted. Then write down what you expect each trial to give, before the first call. Write a frame has every key, and what each refusal means.
Next
The examples climb on from here, one capability a step, starting with Cooled, where a control decides the verdict.