Skip to content

Run your own ​

IMPORTANT

This step

  • Teaches: the method every example followed, and which trial answers which question.
  • Frame: yours.
  • After: Each anchor alone.

Which trial answers which question ​

You want to know whether...DeclareStep that shows it
the decision rests on this evidence at allablation (always there)Your first frame
the answer is stable before comparing itrepeatYour first frame, Quorum
any cut would move it, or only this oneshamYour first frame
the names matter rather than the meaningpresentationYour first frame
where a line sits mattersplacementMisaligned
it rests on the evidence through the connection you claimbypass, with allowed.bypassBypass
the reader can move at all, so that cooled means somethingcontrolCooled, A frame built wrong
it rests somewhere else toocumulativeCumulative
it moves the way the promise saysallowed.ablationMisaligned
several cases of one promise read alikeseveral inputsSeveral inputs, Paths
a finer cut tells moreanother frame at another grainGrain

The method ​

  1. Write the frame. Start from the example closest to your decision, and run uv run gadfly check-frame my-frame.json until it is accepted. Define every label in the instruction.
  2. Write the prediction first. For each trial, the answer you expect and how sure you are, and the verdict that follows. Write it down before the first call, as examples/PREDICTIONS.md was, and keep it beside the frame.
  3. Plan. gadfly plan prints the trials and the calls, and costs nothing.
  4. Run, keep, report. Keep every record; the records are the evidence, and a report reads them without a model.
  5. Set the result against the prediction. A wrong prediction stays in.
  6. Run it more than once when the result matters, and read each run as one draw.
  7. Never edit a frame after its runs. A frame that asked the wrong question gets a new version beside it, with the reason, and the old one keeps its runs.

With eGemma on this machine and its key in EGEMMA_API_KEY or EGEMMA_API_KEY_FILE, these commands plan a kept frame's 40 calls, run it fresh and read the verdict:

bash
uv run gadfly plan examples/09-sync-sms.json
uv run gadfly run examples/09-sync-sms.json \
  --battery midwife.batteries.egemma:EGemma \
  --calls 40 --out my-run.jsonl
uv run gadfly report examples/09-sync-sms.json my-run.jsonl

This reads a kept frame fresh, without adding to its runs. Each kept run was made by examples/climb.py, which runs a frame against eGemma and keeps the run beside the others.

A reader other than the one recorded (another model, another revision, another temperature) is a new experiment: write its predictions down first. Batteries covers other readers, and Write a frame every field.