Gadfly

Python API ​

Each module's section below is written from its public docstrings by tools/doc_outputs.py, and a test keeps it current, so it reads the same here and on GitHub. For narrative guidance start with Your first frame or Reading a verdict.

  • midwife.pipeline: situate, plan, run, judge, overall, calls_needed.
  • midwife.frame: frame parsing and hashing.
  • midwife.errors: what Gadfly refuses, a class for each thing to fix, with its exit status.
  • midwife.types: Observation, Record, Budget, Battery.
  • midwife.verdict: categories, levels, evidence and obligations.
  • midwife.interventions: trial planning, prompts and presentation renaming.
  • midwife.batteries.*: the shipped loopback batteries, two critics, local-json and tendora.
  • midwife.cli: the gadfly command, with plan, run, report, check-frame and portfolio.
  • midwife.portfolio: several frames' verdicts counted, never combined.

midwife.pipeline ​

The pipeline: situate → plan → run → judge (→ overall, over the inputs).

python
frame = situate(json.load(file))
trials = plan(frame)
result = run(frame, battery, Budget(calls=60))
findings = judge(frame, result.records)
verdict = overall(findings)

overall names one category over all the inputs. run is the only stage that calls out. It never starts a call the budget cannot cover, stops between calls when the caller says so, and hands each record to on_record as soon as it exists, so the caller keeps what was spent even if the run stops early. Asked to, it skips the calls a settled trial no longer needs, stops an input once it reads unresolved whatever its other trials give, and makes several calls at once for a battery that takes them.

situate ​

Another name for parse in midwife.frame:

Check a frame and hash it. The frame keeps its own copy: changing the caller's dict afterwards cannot change what the hash names.

python
situate = midwife.frame.parse

plan ​

Another name for plan in midwife.interventions:

The trials of one input, in a fixed order.

python
plan = midwife.interventions.plan

overall ​

Another name for overall in midwife.verdict:

One category over the frame's inputs: outside scope when declared; otherwise coupled but misaligned if any input is; unresolved if any is, or if the relation carried on some inputs and cooled on others; carried or cooled only when every input is. Each input's evidence is kept; what every input leaves open is said once, the rest names its input.

python
overall = midwife.verdict.overall

RunResult ​

What a run made: its records, the calls it planned, why it stopped early (None when it reached the end of its plan), and how many planned calls it skipped because their trial had settled or their input could read nothing but unresolved.

python
@dataclass(frozen=True)
class RunResult

calls_needed ​

The calls a whole run makes: every trial of every input, quorum times each. A budget of this many calls covers it.

python
calls_needed(frame)

run ​

Every planned trial of every input, quorum times each, in a fixed order. only limits the run to some trial ids (a partial run); the verdict then reads what is there. kept finishes a run that stopped: the calls those records made are not made again (drop a record from it to make its call again), and the verdict reads the two runs together.

python
run(
    frame,
    battery,
    budget,
    *,
    cancelled=lambda: False,
    on_record=lambda record: None,
    only=None,
    kept=(),
    settle=False,
    stop_unresolved=False,
    parallel=1,
)

settle makes no more calls for a trial whose level can no longer change (settled); the verdict reads it as it would read every call. Under standing.stable the baseline and repeat always run whole, as their split is read across all their calls. stop_unresolved asks the DECISIVE trials first and asks an input nothing more once every answer its other trials could give reads unresolved (unneeded); the verdict reads the category a whole run would, and names the trials not asked. parallel makes up to that many calls at once, for a battery that says it takes them (a concurrent attribute that is true); records are handed on as each call ends, and a call already started is always kept: a first Ctrl-C starts no more calls, keeps those already out, then goes on as KeyboardInterrupt, and a second stops at once.

judge ​

One finding per input, read from the records alone: the Records of a run, or the dicts a caller kept of them. A record the plan and the frame's battery could not have made is refused.

python
judge(frame, records)

midwife.frame ​

The frame you declare: the situated relation (L) and its assessment frame (Γ).

A frame is plain JSON. It is read whole, checked, and hashed in a canonical form before any trial runs; the hash travels with every record, so a verdict always names the frame it was read under. Changing any coordinate, the grain included, gives another hash: another frame.

Span ​

One piece of the body at the frame's grain.

python
@dataclass(frozen=True)
class Span

Frame ​

A checked frame. It keeps the canonical JSON its hash is taken of, and every coordinate read from it is a fresh copy: nothing a battery or the caller does with what it reads can change what the hash names.

python
@dataclass(frozen=True)
class Frame

Frame has these properties:

  • raw: dict[str, Any]
  • name: str
  • inputs: tuple[dict[str, str], ...]
  • instruction: str
  • interventions: dict[str, Any]
  • allowed: dict[str, Any]
  • battery: dict[str, Any]
  • outside_scope: str | None

canonical ​

The one serialization a hash is taken of.

python
canonical(value)

digest ​

The SHA-256 of a value's canonical serialization.

python
digest(value)

schema ​

The frame's JSON Schema (draft 2020-12), for editors and authoring checks. It describes the shape; parse stays the authority on what only code can check (anchors in the body, trials that change something).

python
schema()

load ​

Read a frame from a JSON file, then check and hash it as parse does.

python
load(path)

parse ​

Check a frame and hash it. The frame keeps its own copy: changing the caller's dict afterwards cannot change what the hash names.

python
parse(raw)

midwife.errors ​

What Gadfly refuses, as Python names it: a class for what the caller must fix (PEP 8), its exit status an attribute of the class (PEP 352), a message that says what went wrong, and notes that say what to do (PEP 678).

python
UsageError     2  the command or the call: flags, budget, standard input
FrameError     4  the frame
RecordsError   5  records or a report that are not this frame's run
BatteryError   6  the battery: its module, its settings, its service

A file the command line cannot read or write is Python's own error (PEP 3151), exit 7. Each class is a ValueError, as every refusal was.

What a run did not do as asked, though it went on and kept its records, is a warning (PEP 230), raised with warnings.warn, which a caller may make an error with Python's own filters (-W error::midwife.errors.OpeningWarning):

python
OpeningWarning  1  calls came back with no answer (1 when every one did)
StoppedWarning  3  the run stopped before its plan's end

GadflyError ​

A refusal: the message says what went wrong, each note what to do.

python
class GadflyError(ValueError)

one_line ​

Whether a value prints on one line: no line break, nor any other control character a terminal could act on.

python
one_line(value)

is_plain ​

Whether a value prints as one plain name: no line break, and no colon a label would end at.

python
is_plain(value)

plain ​

A value as a line prints it: as it is when it is one plain name, else quoted as Python's repr, so it can neither end a label nor start a line.

python
plain(value)

quoted ​

Values as a message names them: each quoted, as Python's repr does.

python
quoted(values)

noted ​

Python's own error, with notes that say what to do: its constructor takes none.

python
noted(error, *notes)

UsageError ​

The command or the call asks for what cannot be done: a flag, a budget, standard input.

python
class UsageError(GadflyError)

FrameError ​

A frame that cannot be assessed as declared.

python
class FrameError(GadflyError)

RecordsError ​

Records or a report that are not what this frame's run made.

python
class RecordsError(GadflyError)

BatteryError ​

A battery that cannot be made, cannot serve this frame as set up, or cannot reach what it needs.

python
class BatteryError(GadflyError)

GadflyWarning ​

What a run did not do as asked, though it went on and kept its records.

python
class GadflyWarning(UserWarning)

OpeningWarning ​

Calls came back with no answer, as openings; exit 1 when every call did, as then nothing was read.

python
class OpeningWarning(GadflyWarning)

StoppedWarning ​

The run stopped before its plan's end, so its records are partial.

python
class StoppedWarning(GadflyWarning)

midwife.types ​

The types a pipeline passes from stage to stage, and the one a battery implements. Gadfly keeps no state: every value here is returned to the caller, who keeps it (a file, a database, a test fixture) as they see fit.

Observation ​

What a battery saw: one of the frame's labels, or a typed opening when there is no outcome to read (a refusal, an unreadable answer, a failure). served names the scorer that answered, as the service reports it (a model, its revision, the endpoint), so a record says who read.

python
@dataclass(frozen=True)
class Observation

Record ​

One scorer call: which frame, input, trial and run, what was read and what it cost. A verdict is recomputed from records alone.

python
@dataclass(frozen=True)
class Record
Record.as_dict ​

The record as a plain dict, as a caller keeps it (JSON Lines, say).

python
as_dict()

Battery ​

A scorer: given the frame and the exact text it must read, one observation. It never sees which trial it is serving. A battery that can serve several calls at once says so with a class attribute concurrent = True; without it, run makes one call at a time.

python
@runtime_checkable
class Battery(Protocol)
Battery.check ​

Refuse a frame this battery must not serve (raise ValueError).

python
check(frame)
Battery.observe ​

Read the prompt once and return what it observed, spending at most max_output_tokens (the budget's cap per call) on the answer.

python
observe(frame, prompt, max_output_tokens)

Budget ​

The most a run may spend: calls scorer calls, each answer capped at output_tokens_per_call, and, if tokens is set, that many output tokens in all. Because every call is capped, a call is never started that could pass the budget; a battery that reports more than the cap has broken that promise, and the run stops after its call.

python
@dataclass(frozen=True)
class Budget

midwife.verdict ​

The verdict, read from the trial records by fixed rules and nothing else.

For each input, each trial's observations are reduced to one level when the frame's quorum agrees, or to none. The rules, in this order (the order of the reference implementation, After the Poem's finding with rules: 2, in scripts/load_bearing.py, https://github.com/mirzahusadzic/after-the-poem):

  1. unresolved: a planned trial has no agreed level; with standing.stable, the baseline and repeat sessions do not all agree (a split level cannot show a one-level change); or repeat differs from baseline;
  2. coupled but misaligned: presentation moves it, or placement does (the structure, not the wording, drove it);
  3. coupled but misaligned: ablation moves it to an observation the frame does not allow (for ordered labels, the wrong way);
  4. cooled: ablation leaves it as it was, unless a control trial (spans whose removal must move any scorer that reads) left it too: then unresolved, as the scorer cannot tell;
  5. unresolved: a sham moves it (the change is not specific to the relation);
  6. carried: otherwise.

Outside scope only when the frame declares the exclusion; never inferred.

With eachAnchor, each anchor is also cut alone. Those trials are evidence only and never change a category: after the ablation moved it, they say whether each anchor carries it, either suffices, or one adds nothing alone; where the ablation left it, they name any anchor that moved it alone.

What the trials do not settle is recorded as an open obligation beside the category:

  • carried: no hidden fallback (a bypass trial), presentation invariance and specificity when their trials were not run, and materialization (whether the observation is the form the recipient receives, which the frame must say);
  • cooled: discrimination, unless a control moved it;
  • outside scope: the independent scope check, which only the receiving owner can make.

Traceability rests on the records: each names its frame, battery, body, prompt and what served it.

Finding ​

One input's verdict: its category, each trial's agreed level (None where the quorum did not agree), the evidence for the category, and the obligations it leaves open.

python
@dataclass
class Finding
Finding.as_dict ​

The finding as a plain dict.

python
as_dict()

Overall ​

The frame's one category over its test domain: every input's.

python
@dataclass
class Overall
Overall.as_dict ​

The verdict as a plain dict.

python
as_dict()

overall ​

One category over the frame's inputs: outside scope when declared; otherwise coupled but misaligned if any input is; unresolved if any is, or if the relation carried on some inputs and cooled on others; carried or cooled only when every input is. Each input's evidence is kept; what every input leaves open is said once, the rest names its input.

python
overall(findings)

level ​

The observation at least agree runs gave, if exactly one did.

python
level(observations, agree)

settled ​

Whether a trial's level can no longer change: whatever its remaining calls answer, level gives what it gives now. One label at agree that no other can still reach settles it; so does no label able to reach it.

python
settled(observations, quorum, agree, labels)

judge ​

One finding per input, from the records of one run of one frame. Every planned trial is read; one without records has no level, and one with fewer than quorum has a level only once it is settled, so a stopped run never reads a level its missing calls could change.

python
judge(frame, records)

unneeded ​

The trials input x need not be asked, given the answers so far: once every answer the rest could give reads unresolved, all those not asked to the end, but each anchor alone while a reading still gets as far as the ablation, where its evidence is read. Empty while an answer could read otherwise, or the answers are too many to read.

python
unneeded(frame, x, samples)

unreachable ​

What no answers can make this frame read, found before a call: carried or cooled, each with the checks open in every reading that would name it but for them (empty when none comes that near), or the exclusion when the frame is declared outside scope. The rules themselves read every answer each trial could give; None when the readings would be more than OUTCOMES for an input.

python
unreachable(frame)

check_records ​

Records are read only as the plan and the frame's battery made them: this frame, its battery, its inputs and trials, each trial's own body, runs within the quorum, each run once, and each a label of the frame or an opening. A record copied twice can never make a quorum. run reads the records it is given to keep by the same checks.

python
check_records(frame, records, bodies=None)

allowed_for ​

What a trial may produce (ablation; bypass, its designated opening; an inversion or input inversion, what it must reach): the labels listed, or, for ordered labels, those below (or above) the baseline.

python
allowed_for(frame, base, trial="ablation")

midwife.interventions ​

The admissible interventions Δ: each changes or removes the relation and keeps everything else as it was.

Every trial body is derived from the frame alone, so the plan is fixed, and recorded, before a single observation is made; the sham's draw is seeded by the frame's hash, never by the clock.

Trial ​

One intervention, applied: the body the scorer will read.

python
@dataclass(frozen=True)
class Trial

Trial has these properties:

  • text: str
  • body_hash: str

place ​

The spans with one moved to follow another; every text unchanged.

python
place(spans, span_id, after)

without ​

The spans, in their order, less those named.

python
without(spans, removed)

rename ​

Rename each word consistently, whole words only, in one pass: a word is renamed once (a swap stays a swap), and a phrase wins over a word inside it. A capital is kept, and so is a word written in capitals. A word may begin or end in a sign, such as 20% or C++; it is never part of a longer word, nor of a longer run of its edge's sign (C++ is not in C+++).

python
rename(text, mapping)

renamer ​

rename with its map built once: the table and the pattern are made here, and the function returned applies them to any text.

python
renamer(mapping)

sham_draws ​

Spans to remove instead of the anchors: as many as the anchors, of the same kinds (or the kind the sham's kind names for an anchor), none cited, none bypassed, none a declared fallback, and no span in two draws. A fallback is declared to carry the relation, so cutting it is no sham. Each is the closest in length still free; spans equally close are drawn among by the frame's own seed. A draw that cannot be matched is left out; the verdict says so.

python
sham_draws(frame, count)

trial_ids ​

The ids plan gives its trials, in its order, without building any trial's body: what a budget is counted from.

python
trial_ids(frame)

plan ​

The trials of one input, in a fixed order.

python
plan(frame)

prompt ​

What the scorer reads: the trial's body and one input, nothing else. Presentation renames the input's words too, so only the wording moves; an inversion changes the relation's sense in the body alone, and an input inversion what is asked alone.

python
prompt(frame, trial, x)

midwife.batteries.critic ​

What every critic battery shares: one safe way to call a loopback server, and one way to read a label from a JSON answer (the frame's observation map).

loopback ​

The URL, if it names this machine over plain http on a valid port; else ValueError, which never repeats a user or password the URL holds.

python
loopback(url, what)

opener ​

No proxy from the environment, no redirect: only the address named is called. Built per call, so no environment read at import can linger.

python
opener(*handlers)

opened ​

The response to a request, opened through opener(), its status line and headers in within timeout seconds of the call: a server still sending the answer's head then is cut off, and the call raises TimeoutError. The body is read_reply's to bound, and a refusal's read_error's, both by the same deadline.

python
opened(request, timeout)

Late ​

A reply still coming at its deadline, with the bytes that had arrived: kept beside the opening, never read as an answer.

python
class Late(TimeoutError)

Late has these properties:

  • text: str. What had arrived, as text, for the record's raw.

wait_at_most ​

Let the next read on the response's socket wait at most seconds.

python
wait_at_most(response, seconds)

read_reply ​

A reply as it comes, until it ends or passes the size cap; Late, with what had arrived, if it is still coming at the deadline. Each read waits only what is left of the deadline, so a stall never runs past it.

python
read_reply(response, deadline, clock)

read_error ​

Up to 4,096 bytes of a refusal's body, read by the deadline: what had arrived by then, or nothing. Each read waits only what is left.

python
read_error(error, deadline, clock)

check_output ​

Refuse a battery.output that is not {schema, label}, the label a JSON pointer into the answer.

python
check_output(battery)

output_schema ​

The schema the critic answers to and the JSON pointer to its label.

python
output_schema(frame)

pointed ​

The value at a JSON pointer (RFC 6901), or KeyError.

python
pointed(document, pointer)

read_label ​

The label the answer gives at the frame's pointer, or an opening.

python
read_label(text, frame)

label_at ​

The label at the frame's pointer in an answer already read as JSON.

python
label_at(answer, frame)

midwife.batteries.unified ​

The unified battery: an LLM critic through the unified model router.

The unified model router serves the Responses API on loopback and maps a model slug to the provider that serves it, with the router's own key: this battery sees only the router. A slug whose provider trains on its inputs, by the router's own list, is sent only when the frame names it exactly under trainsOnInputs.

One battery covers them all; calling a provider's own endpoint is a different battery (a native one). An unprefixed slug is OpenAI's, which the router reaches only with the caller's Codex sign-in: this battery refuses it.

The critic answers in JSON to a schema, by default {"label": <one of the frame's labels>}. A critic of your own (its output schema, as your project already has it) is adapted by saying where in its answer the label is:

json
"battery": {"kind": "unified", "model": "anthropic/claude-sonnet-5-5",
            "reasoning": "low",
            "output": {"schema": {...}, "label": "/verdict"}}

What this battery promises, whatever the frame says:

  • it talks only to a router on this machine (loopback);
  • it sends no tools: the router's web search runs on another provider, and a body would leave the model's own provider with it;
  • it refuses a model whose provider trains on its inputs unless the frame names that model under battery.trainsOnInputs, and sends every other call under the router's deny data policy (X-Codex-Router-Data-Policy: deny), so the router refuses a model whose provider trains and reaches only endpoints that keep no data; before the first call it checks that the router lists the policy at GET /health, and refuses to run if not;
  • it uses no proxy and follows no redirect, so the body and the token go to the loopback address named and nowhere else;
  • each call has a deadline (timeout seconds for the whole answer), the budget's cap on output tokens, and a cap on what it reads back;
  • each record names what served it: the model the response names, and the provider the router reports (X-Codex-Router-Report: provider, asked when the router lists it), or, for one pinned endpoint, that endpoint's tag; never a provider the router calls unknown;
  • endpoints pins the endpoints the router may use, in order and without fallbacks, through the router's X-Codex-Router-Endpoints, so one endpoint reads a quorum; the router must list endpoint pins at GET /health;
  • the router's token comes from the environment or the router's own env file and is never logged, returned or recorded.

consented ​

Whether the frame lets this model go where inputs train: its slug, exactly, in the list under trainsOnInputs (a string is no list).

python
consented(battery)

policy ​

The router's headers each call of this frame sends: the deny data policy, unless the frame names its model under trainsOnInputs.

python
policy(frame)

pinned ​

The router's header that pins this frame's endpoints, in order and without fallbacks; none when the frame names none.

python
pinned(frame)

capabilities ​

What the router lists under capabilities at its health address, which takes no token; empty when it lists nothing.

python
capabilities(url, timeout)

payload ​

The request body. No tools key, ever; the data policy and the endpoints go in the router's headers (policy, pinned).

python
payload(frame, text, max_output_tokens)

parse_stream ​

The answer's text, its usage, an opening if the call did not complete, and what served it, from the router's server-sent events: the model the response names, then the provider the router reports, or the tag of the one endpoint the call was pinned to, joined by " via ". A stream still coming at deadline (on clock) is cut there, and a connection that times out or fails mid-stream keeps what arrived.

python
parse_stream(lines, deadline=None, clock=time.monotonic)

Unified ​

battery: {"kind": "unified", "model": "anthropic/claude-sonnet-5-5"}

python
class Unified
Unified.check ​

Refuse, before any call, a frame this battery must not serve: another battery's, an unknown key, an OpenAI slug, tools, a model whose provider trains on its inputs without the frame's consent, or no router token.

python
check(frame)
Unified.observe ​

One call through the router: the prompt under the frame's instruction, answered to its schema; the label at the frame's pointer, or an opening that says why there is none.

python
observe(frame, prompt, max_output_tokens)

midwife.batteries.egemma ​

The eGemma battery: Gemma 4 on this machine, through eGemma's /judge.

eGemma reads the prompt under the frame's instruction, with no persona added, and a grammar holds its answer to the schema: by default {"label": <one of the frame's labels>}, or a critic's own schema with the label's place named, as for the unified battery. Nothing leaves this machine, and nothing is cached, so the quorum's runs are fresh samples; at temperature: 0 the battery is close to deterministic. With reasoning: true, Gemma thinks before every answer, and only the answer is held to the schema.

json
"battery": {"kind": "egemma", "temperature": 0.7}

What this battery promises, whatever the frame says:

  • it talks only to an eGemma on this machine (loopback), through no proxy and no redirect;
  • each call has a deadline (timeout seconds for the whole reply), the budget's cap on output tokens, and a cap on what it reads back; with reasoning, the cap is the thought's and the answer's together: the answer keeps ANSWER_TOKENS of it and the thought gets the rest, up to MAX_THOUGHT_TOKENS, and a cap with no room left for a thought is an opening;
  • it calls one at a time and paces itself to the judge budget eGemma publishes at GET /rate-limits (read once, before the first call); when eGemma is busy anyway (429 or 503) it waits as long as eGemma asks, at most a minute, and tries at most three more times before the call becomes an opening;
  • each record names the model and revision eGemma reports; a frame that pins them (model, revision) gets an opening from any other;
  • the key comes from the file named by EGEMMA_API_KEY_FILE, else from EGEMMA_API_KEY or WORKBENCH_API_KEY, and is never logged or recorded.

payload ​

The /judge request: the instruction, the text, the schema, the answer's cap, and the frame's temperature if it names one. With reasoning, the call's cap is split: the answer keeps ANSWER_TOKENS and the thought the rest.

python
payload(frame, text, max_output_tokens)

EGemma ​

battery: {"kind": "egemma"}

python
class EGemma
EGemma.read_budget ​

The judge budget eGemma publishes; without one, only the backoff.

python
read_budget()
EGemma.pace ​

Wait until one more call fits the budget's window, then count it.

python
pace()
EGemma.check ​

Refuse, before any call, a frame this battery must not serve: another battery's, an unknown key, a temperature outside 0 to 2, an empty pin, reasoning that is not true or false, or no key; then read the budget eGemma publishes.

python
check(frame)
EGemma.observe ​

One /judge call, paced to eGemma's budget and retried while it is busy: the label at the frame's pointer, or an opening that says why there is none.

python
observe(frame, prompt, max_output_tokens)

midwife.batteries.local_json ​

The local-json battery: any evaluator on this machine that answers JSON.

A local classifier, test oracle or retrieval checker that already reads an instruction and a text and answers to a schema needs no adapter of its own: name its address in the frame and this battery calls it with the same care the shipped critics take.

json
"battery": {"kind": "local-json", "url": "http://127.0.0.1:8765/judge",
            "timeout": 30, "output": {"schema": {...}, "label": "/verdict"}}

The evaluator receives {"instructions", "input", "schema", "max_tokens"} and answers {"answer", "text", "usage", "served"}: answer is the JSON the label is read from (at /label, or the frame's own pointer), served names what read it.

What this battery promises, whatever the frame says:

  • it talks only to the address named, on this machine (loopback), over plain http, through no proxy and no redirect;
  • it sends the instruction, the exact prompt, the schema and the answer's cap, and nothing else: no credentials, no tools;
  • each call has a deadline (timeout seconds for the whole reply, 30 by default) and a cap on what it reads back;
  • a reply it cannot read is an opening, never a guessed label, and each record names what the evaluator says served it.

payload ​

The request: the frame's instruction, the exact prompt, the schema the answer is held to, and the answer's cap.

python
payload(frame, text, max_output_tokens)

LocalJSON ​

battery: {"kind": "local-json", "url": "http://127.0.0.1:8765/judge"}

python
class LocalJSON
LocalJSON.check ​

Refuse, before any call, a frame this battery must not serve: another battery's, an unknown key, an address off this machine, a timeout that is not a positive number of seconds, or an output that does not point at its label.

python
check(frame)
LocalJSON.observe ​

One call to the evaluator: the label at the frame's pointer, or an opening that says why there is none.

python
observe(frame, prompt, max_output_tokens)

midwife.batteries.tendora ​

The tendora battery: Tendora's engine, asked again against a trial's statements.

Tendora's engine server (in its demo package, npm run serve -- --project P --port N) answers POST /ask with statements against exactly the statements given, in their order, and saves nothing. This battery splits a trial's prompt by its documented form (the statements one per line, a line --- between blank lines, then the input), asks the frame's saved question and case again against those statements, and reads the label from the engine's answer.

json
"battery": {"kind": "tendora", "url": "http://127.0.0.1:8791",
            "timeout": 60, "scope": "...", "offline": true}

Each input reads Case: … on its first line when there is a case, then Question: …, exactly as the saved check asked it. The engine finds the saved check by the question, the case and the scope, so all three must be the check's own. When a trial has turned what is asked (an input inversion, or a presentation that renames the input's words), the battery finds the declared input that turned into it and names it as the basis, the saved check to ask it against; it never learns the trial or the input's id.

The label, from the engine's answer:

  • supported, with one answer whose text is a label: that label; several answers or another text: another answer when the frame lists it;
  • open, conditional, ambiguous and conflict: the label of the same name when the frame lists it (a proposal one confirmation away is still open);
  • limited, needs-reading, a refusal, any status or answer the frame does not list, and a reply out of the engine's form (answers that are not objects with words and a true or false condition, or no served) are openings, never labels.

Before any call, check refuses what the battery could not ask: no open among the labels, an input not in the form above, a statement with a line break in any trial (the engine is sent one a line), a rename that turns an input out of the form, and two inputs turned into one question.

What this battery promises, whatever the frame says:

  • it talks only to the address named, on this machine (loopback), over plain http, through no proxy and no redirect, and sends no credentials;
  • it sends the question, the case, the scope, the statements, the offline flag and, for a turned input, the saved check it came from (basis), and nothing else; the trial's id is never among them;
  • each call has a deadline (timeout seconds for the whole reply, its head included, 60 by default) and a cap on what it reads back: a reply past the cap is an opening, and nothing of it is kept;
  • each record names the engine version, revision and packs that served it;
  • each record keeps the answer, its status and each answer's text and condition, on its first line, then the reply as it came but for its echo of the statements: a long reply is cut at the record's cap, and the answer line holds as many answers as fit in half of it, with more saying how many were left out, so it is always whole JSON.

asked ​

The question and the case an input holds, or None when it is not in the form Case: … (optional, first) then Question: …, each with words in it, and no carriage return.

python
asked(text)

request_body ​

What the engine is sent for one prompt, or None when the prompt is not in Gadfly's form: the question, the case and the scope, the statements in their order, and the offline flag.

python
request_body(frame, prompt)

turned ​

Each text a declared input turns into, through the frame's input inversion or presentation, and the declared inputs that turn into it. A text some declared input already is is left out: a swap asks that input's own saved check. Worked out once for a frame.

python
turned(frame)

malformed ​

Why a reply is not an answer in the engine's form, or None: a status in words, and answers, if any, a list of objects, each with words for its text and true or false for its condition, when it says one.

python
malformed(reply)

label_of ​

The label the engine's answer gives, or the opening that says why it gives none; a reply out of the engine's form gives none.

python
label_of(reply, labels)

answer_line ​

The answer alone: its status, and each answer's text and condition, as many as fit in half the record's cap, and how many more there were (more), so the line a record keeps first is always whole JSON.

python
answer_line(reply)

served_by ​

What a record names as having served the reply: the engine's version, the first 16 characters of its revision, and its packs.

python
served_by(reply)

Tendora ​

battery: {"kind": "tendora", "url": "http://127.0.0.1:8791"}

python
class Tendora
Tendora.check ​

Refuse, before any call, a frame this battery must not serve: another battery's, an unknown key, an address off this machine, a timeout that is not a positive number of seconds, a scope or offline flag of the wrong type, no open among the labels, a statement with a line break in any trial, an input not in the form Case: … then Question: …, an input inversion or presentation that turns an input out of that form, or two inputs turned into one question.

python
check(frame)
Tendora.observe ​

One ask of the engine: the label its answer gives, or an opening that says why there is none.

python
observe(frame, prompt, max_output_tokens)

midwife.cli ​

The command line: plan, run, report, check-frame, portfolio.

bash
uv run gadfly plan examples/upgrade.json
uv run gadfly run examples/upgrade.json \
    --battery midwife.batteries.egemma:EGemma \
    --calls 40 --out records.jsonl
uv run gadfly report examples/upgrade.json records.jsonl
uv run gadfly check-frame my-frame.json
uv run gadfly portfolio a.json b.json

It uses only the public pipeline. run needs an explicit call budget, writes each record as one JSON line the moment it exists, never writes over a file and makes none when it is refused, and says when it stopped early; report keeps a finding's evidence and its open obligations apart. A battery is named module:Class and is imported from the current directory.

load_frame ​

The frame in a JSON file, or on standard input for -, checked and hashed.

python
load_frame(path)

load_battery ​

An instance of the battery class module:Class.

python
load_battery(spec)

read_text ​

A file's text, or standard input's for -; text that is not UTF-8 is a refusal.

python
read_text(path, refusal)

load_json ​

text as JSON, or a refusal that names its file and where it broke; line is where text starts in the file, when it is one line.

python
load_json(path, text, refusal, line=None)

load_records ​

The records kept in a JSON Lines file, or on standard input for -.

python
load_records(path)

render_report ​

The person-readable report: the overall category, then each input's levels, evidence and open obligations, each on its own line.

python
render_report(frame, records)

command_plan ​

gadfly plan: the trials, what each changes, and the call count, without a call.

python
command_plan(args)

command_run ​

gadfly run: every planned call the budget covers, each record kept.

python
command_run(args)

command_report ​

gadfly report: the verdict read from a kept JSON Lines file.

python
command_report(args)

command_check_frame ​

gadfly check-frame: the frame's shape against the schema (when jsonschema is installed), then the parser's authoritative refusal.

python
command_check_frame(args)

command_portfolio ​

gadfly portfolio: several reports counted side by side.

python
command_portfolio(args)

parser ​

The command line's arguments.

python
parser()

main ​

Run one command. A refusal prints its class and message, then its notes, and exits with its class's status; with --json, the same as one object on standard error. Anything else is a bug: its traceback, and 70.

python
main(argv=None)

refused ​

Say a refusal as the last lines of a Python traceback, or as JSON, and give its exit status: a GadflyError's own, 7 for a file, 130 for Ctrl-C.

python
refused(error, as_json)

described ​

A refusal or a warning as JSON: its type, the exit status it gives, its message and notes, and its troubleshooting section.

python
described(problem, status, message="", anchor="")

Parser ​

The arguments; a mistake in them is a UsageError.

python
class Parser(argparse.ArgumentParser)
Parser.parse_args ​

The arguments; one no command takes is that command's mistake.

python
parse_args(args=None, namespace=None)
Parser.error ​

Refuse as a UsageError, in one clause, pointing at the help.

python
error(message)

midwife.portfolio ​

Several frames' verdicts side by side, without a verdict of their own.

overall reads one frame over its inputs. Different frames have different relations, promises and recipients, and there is no fixed rule that combines them, so a portfolio counts and groups: frames by category, open obligations two or more frames share, and frames whose inputs disagree. It never names a category for the portfolio.

Entry ​

One frame's overall verdict, as gadfly report --json writes it.

python
@dataclass(frozen=True)
class Entry

summarize ​

Counts by category, the frames in each, the obligations frames share, and the frames whose inputs reached different categories.

python
summarize(entries)

entry_from ​

An entry from a gadfly report --json document (or a bare overall).

python
entry_from(document)