ARTI guides
Evaluate saved task results
Separate reference availability, measured metrics, population coverage and acceptance policy. Evaluate committed outputs without model replay.
Define independent evaluation views
An EvaluationSuiteConfig contains named views. A local scorer receives the original input, prediction, target and execution context and returns finite metrics plus one score record per eligible ID. The scorer owns the units and correctness rule.
| View requires | What it measures | When evidence is missing |
|---|---|---|
| reference | Prediction against explicit targets | unavailable; never inferred labels |
| output | Committed format, file validity or alignment | Preserve the actual output population |
| external | Separately produced protocol evidence | Retain declared external origin |
Score the committed journal
Supply the suite configuration and explicit reference records. The service verifies committed outputs and runs the installed scorer; it does not load a model or task factory. File scorers receive verified relative file context.
python -m arti recipe score-suite outputs/prediction --suite suite.json --references references.jsonl --output evaluation.jsonApply an explicit acceptance policy
Criteria declare minimum or maximum thresholds with task-owned units. No criteria means criteria_passed=None. Missing metrics are not_evaluable. A scored view reports measurement, not automatic correctness; complete_evidence and acceptance remain separate.
An optional installed aggregator can combine views with declared weights and its own revision. There is no default composite score or metric direction. Scorer failures stay visible while other views can finish.
Compare the same sample population
By default, compare_prediction_suites pairs the same committed IDs and exact original input content. Trial state and seeds remain explicit. Candidate-minus-baseline deltas require matching scored ID sets; the task determines direction and units.
python -m arti recipe compare-suites outputs/candidate --baseline outputs/baseline --suite suite.json --references references.jsonl --output comparison.jsonThis guide targets arti-fit 3.3.0 · Public source dcb321d6