Run your own
IMPORTANT
This step
- Teaches: the method every example followed, and which trial answers which question.
- Frame: yours.
- After: Each anchor alone.
Which trial answers which question
| You want to know whether... | Declare | Step that shows it |
|---|---|---|
| the decision rests on this evidence at all | ablation (always there) | Your first frame |
| the answer is stable before comparing it | repeat | Your first frame, Quorum |
| any cut would move it, or only this one | sham | Your first frame |
| the names matter rather than the meaning | presentation | Your first frame |
| where a line sits matters | placement | Misaligned |
| it rests on the evidence through the connection you claim | bypass, with allowed.bypass | Bypass |
| the reader can move at all, so that cooled means something | control | Cooled, A frame built wrong |
| it rests somewhere else too | cumulative | Cumulative |
| it moves the way the promise says | allowed.ablation | Misaligned |
| several cases of one promise read alike | several inputs | Several inputs, Paths |
| a finer cut tells more | another frame at another grain | Grain |
The method
- Write the frame. Start from the example closest to your decision, and run
uv run gadfly check-frame my-frame.jsonuntil it is accepted. Define every label in the instruction. - Write the prediction first. For each trial, the answer you expect and how sure you are, and the verdict that follows. Write it down before the first call, as
examples/PREDICTIONS.mdwas, and keep it beside the frame. - Plan.
gadfly planprints the trials and the calls, and costs nothing. - Run, keep, report. Keep every record; the records are the evidence, and a report reads them without a model.
- Set the result against the prediction. A wrong prediction stays in.
- Run it more than once when the result matters, and read each run as one draw.
- Never edit a frame after its runs. A frame that asked the wrong question gets a new version beside it, with the reason, and the old one keeps its runs.
With eGemma on this machine and its key in EGEMMA_API_KEY or EGEMMA_API_KEY_FILE, these commands plan a kept frame's 40 calls, run it fresh and read the verdict:
bash
uv run gadfly plan examples/09-sync-sms.json
uv run gadfly run examples/09-sync-sms.json \
--battery midwife.batteries.egemma:EGemma \
--calls 40 --out my-run.jsonl
uv run gadfly report examples/09-sync-sms.json my-run.jsonlThis reads a kept frame fresh, without adding to its runs. Each kept run was made by examples/climb.py, which runs a frame against eGemma and keeps the run beside the others.
A reader other than the one recorded (another model, another revision, another temperature) is a new experiment: write its predictions down first. Batteries covers other readers, and Write a frame every field.