Severity ordering and confidence-segment review bands

A real-data evaluation of the word model with severity as the primary queue ordering, development-only diagnostics, and a stricter rule for assigning the priority label. Every moderation action still requires human confirmation.

What changed

The primary ordering is severity: for each comment, take the maximum over labels of calibrated probability × that label's severity weight, then review the highest-scoring comment first. The priority_review label no longer guarantees earlier service. Band-first priority, probability and FIFO remain comparisons on the same admitted comments and arrival streams.

For each label, the priority threshold is the lowest declared score edge for which every segment above it meets the 95.00% empirical agreement target with at least 30 rows. These are score ranges, not statistical confidence intervals. Human-review thresholds and identity-term subgroup thresholds retain cumulative precision selection.

The rule is a project extension inspired by score-based triage in Thomas et al., Supporting Human Raters. The paper does not prescribe this all-segments rule or these bin edges. The edges were designed after earlier test diagnostics were inspected.

Disabled means that no label-specific threshold qualified; other labels or policy rules can still admit a comment. The table shows population thresholds. Subgroup thresholds are stored in the source record.

Band precision and coverage

Precision here means that at least one original Jigsaw label is positive, after rules and subgroup thresholds. It is not an independent measure of whether a comment deserves review. Cumulative and segment test arms use the same saved model scores and selection data; neither fits thresholds on test labels.

The 5-fold development diagnostic refits thresholds on the other folds before routing each held-out fold. It estimates sensitivity to threshold selection within the same distribution; it is not a formal bound on optimism or evidence of generalization to a new distribution.

Where high-risk comments go

High risk uses the saved policy's true-label harm proxy and the simulation threshold harm proxy >= 5.0. A larger high-risk share in a smaller band can coexist with fewer high-risk comments in that band. 1,801 test comments change bands between these two rules. The two rules admit exactly the same test comments. Since severity ignores the review band, changing only these bands cannot change its queue results.

Queue results

These rates are admitted review jobs per hour, not all incoming platform comments. Capacity is 120 jobs/hour with 4 reviewers, 2.0 minutes per job and an 8-hour shift. Cells show mean ± standard deviation across 5 paired simulation seeds.

At 180 review arrivals/hour, the paired mean difference, severity minus band-first priority, is 0.40 high-risk completions and 0.144 harm proxy per reviewer-hour. These are the current bands; the earlier v2 comparison is a different experiment.

Wait p90 includes jobs that started, including reviews still in progress. Never-started jobs are excluded from waiting-time quantiles and remain in the unfinished count. These seeds characterize simulated arrivals on this fixed corpus; they do not measure uncertainty over new corpora. Harm is a label-weighted proxy, not measured real-world harm.

Evidence and limits

Recorded GitHub real-evaluation run: success. The record includes the workflow jobs and artifact hashes. A successful run does not mean every diagnostic met a precision target or that fairness was established. Per-band test precision gates remain unset.

For priority-review false discoveries, the identity-term subgroup ratio has 95% interval [0.720, 1.294], against a limit of 1.250. Its recorded-metric status is inconclusive. FDR concerns false discoveries among flagged comments; it does not replace the separately recorded FPR on clean comments.

The public test set has been inspected repeatedly. This is a benchmark rerun and a descriptive comparison, not a fresh holdout. The segment rule concentrates the priority band; it does not establish an improvement in model recall or actual human review value.

Two clean retrains with the same data, split, configuration and recorded dependency versions differed in two admitted comment IDs. Those membership changes also change which comments the queue's random indices select. Compare orderings within this saved run; differences between retrains cannot be attributed to the ordering alone. The retraining check found different vocabulary selections at the feature cap; the exact platform cause remains unresolved.

Keep the word model and human confirmation. Jev evaluation remains unattempted following the reported registration limit. An independent human evaluation of review worthiness remains deferred and not started because of annotation workload. Original Jigsaw labels are retained.

Reproduce this step

# Download and collect the recorded CI run
gh run download 35857528118 --name evaluation-210d9857220b23d175d4aff5a54ab221ad86280b --dir reports/ci-35857528118
gh run view 35857528118 --json headSha,status,conclusion,url,jobs >reports/ci-35857528118-ci.json
PYTHONPATH=. python record/collect_priority.py reports/ci-35857528118 --ci-metadata reports/ci-35857528118-ci.json
python record/render.py
# Development diagnostics without scoring test rows
python scripts/run_pipeline.py --config configs/dev.yaml
# Rerun the benchmark protocol from the recorded code revision
python scripts/run_pipeline.py --config configs/baseline.yaml
REVIEW_ROUTER_EVAL_REPORT=reports/<run>/report.json pytest -q tests/test_gate.py
python scripts/render_results.py reports/<run>

Run 20260923T115815829404Z-human-review-baseline, commit 210d9857220b23d175d4aff5a54ab221ad86280b, clean working tree. The Step 6 source record contains policy, thresholds, development diagnostics, the matched-score test comparison and CI provenance. Steps 1 to 5 retain their saved results.