Experiment-driven development¶
NiFreeze changes how it registers volumes only on the evidence of an experiment. Each experiment starts as a question registered in an issue before anything runs. The issue states the hypotheses, the data, the metric, the decision rule, and the code or default that each outcome would change. The tracking issue #614 lists every experiment, its status and its decision.
The experiments run on the synthetic phantoms (Synthetic reference data), where the truth
of every volume is known exactly.
Each runs as a datalad run record in the phantom dataset, from a script under
tools/registration/, so every result traces back to the command and the
NiFreeze commit that produced it, and can be run again.
A second script generates every figure, table and quoted command of the report, so
no number in a report is typed by hand.
A report names the comparison behind each claim, and says so when an explanation
has not been tested.
When a result calls for a change to the registered plan, the change is posted on the
issue before the run it affects, and the report keeps the registered result next to
the one added after the fact (post hoc).
The reports on registration accuracy (Registration accuracy) follow the same approach.