← Back to the article Benchmark

Meet Jev

Republikken, October 8, 2026

Snapshot of the Benchmark tab of the workshop demo app, taken on October 9, 2026. Raw results and code: github.com/hmatringe/workshops_material_20261008_republikken_meet_jev. Same prompt for every scorer, decision threshold 0.5.

Inbox triage: is this message unsolicited?

The dataset

easymediumhardtotal
spam585050158
not spam576050167

340 generated, 325 usable (ok 311, reject 15, review 14). Languages: da 68, en 204, fr 68. Written label-first by Claude Sonnet 5.5, then audited blind.

Blind auditors: disagreement with the label

auditoritemseasymediumhardmean confidence
claude_sonnet_5_5340 2.5%3.3%4.0% 0.92
gemma4_26b340 2.5%8.3%14.0% 1.00
claude_opus_5_5340 0.8%6.7%6.0% 0.91

A disagreement that grows with the difficulty says the tiers mean something. A confidence that never moves says the auditor has none.

Scorers

scoreritemsaccuracyF1BrierECEin 0.40-0.60acc. in zoneeasymediumhardmedian msp95 mstotal timecost USD
Jev 1.13 (TypeSafe)
inbox-jev-20261008-171530.csv		 325 91.7% 0.914 0.062 0.061 5.5% 55.6% 95.7%94.5%84.0% 29 50 0m11s 0.0054
Laya multilingual (local, 322M)
inbox-laya-20261009-085303.csv		 325 70.5% 0.713 0.228 0.172 8.0% 42.3% 76.5%70.0%64.0% 47 72 0m28s 0.0000
Claude Sonnet 5.5
inbox-sonnet-20261008-070650.csv		 325 98.2% 0.981 0.017 0.072 1.8% 33.3% 100.0%97.3%97.0% 1334 3146 9m13s 0.5505
Claude Opus 5.5
inbox-claude_opus_5_5-20261008-082905.csv		 325 98.5% 0.984 0.021 0.073 0.9% 66.7% 100.0%98.2%97.0% 2048 7483 15m07s 1.0915
Claude Fable 5.1
inbox-claude_fable_5_1-20261008-140741.csv		 325 97.8% 0.978 0.019 0.053 0.3% 100.0% 98.3%97.3%98.0% 1573 1949 9m06s 2.3072
Qwen3-4B, asked for its confidence
inbox-ollama_qwen3_4b-20261008-070621.csv		 325 86.2% 0.854 0.124 0.077 0.0% n/a 88.7%92.7%76.0% 1292 1881 7m25s 0.0000
Qwen3-4B, 25 lines
inbox-Qwen_Qwen3_4B-20261008-071941.csv		 325 81.8% 0.779 0.170 0.176 0.3% 0.0% 87.8%88.2%68.0% 1146 5307 10m08s 0.0000
Qwen3-0.6B, 25 lines
inbox-local-20261008-073432.csv		 325 53.8% 0.107 0.369 0.327 5.2% 70.6% 53.9%56.4%51.0% 272 2090 3m24s 0.0000

Stability, Claude Sonnet 5.5: 5 passes on 30 items, mean SD of p 0.0084, decision flipped on 1 items (3.3%).

Jev 1.13 (TypeSafe) ECE 0.061, accuracy 91.7%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 5% on average, spam in 1% of cases, 127 items 1% says 13% on average, spam in 0% of cases, 14 items 0% says 24% on average, spam in 46% of cases, 13 items 46% says 32% on average, spam in 43% of cases, 7 items 43% says 44% on average, spam in 57% of cases, 7 items 57% says 56% on average, spam in 60% of cases, 10 items 60% says 64% on average, spam in 82% of cases, 11 items 82% says 75% on average, spam in 85% of cases, 20 items 85% says 85% on average, spam in 88% of cases, 26 items 88% says 96% on average, spam in 99% of cases, 90 items 99% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 127 items, 126 right 1 wrong 127 (1) 14 items, 14 right 14 13 items, 7 right 6 wrong 13 (6) 7 items, 4 right 3 wrong 7 (3) 7 items, 3 right 4 wrong 7 (4) 10 items, 6 right 4 wrong 10 (4) 11 items, 9 right 2 wrong 11 (2) 20 items, 17 right 3 wrong 20 (3) 26 items, 23 right 3 wrong 26 (3) 90 items, 89 right 1 wrong 90 (1) how many items the scorer placed in each bracket, in red the wrong decisions (27 in all)
Laya multilingual (local, 322M) ECE 0.172, accuracy 70.5%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 3% on average, spam in 25% of cases, 71 items 25% says 14% on average, spam in 24% of cases, 34 items 24% says 25% on average, spam in 28% of cases, 18 items 28% says 34% on average, spam in 22% of cases, 18 items 22% says 43% on average, spam in 50% of cases, 8 items 50% says 56% on average, spam in 39% of cases, 18 items 39% says 65% on average, spam in 57% of cases, 14 items 57% says 75% on average, spam in 56% of cases, 16 items 56% says 86% on average, spam in 67% of cases, 27 items 67% says 98% on average, spam in 76% of cases, 101 items 76% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 71 items, 53 right 18 wrong 71 (18) 34 items, 26 right 8 wrong 34 (8) 18 items, 13 right 5 wrong 18 (5) 18 items, 14 right 4 wrong 18 (4) 8 items, 4 right 4 wrong 8 (4) 18 items, 7 right 11 wrong 18 (11) 14 items, 8 right 6 wrong 14 (6) 16 items, 9 right 7 wrong 16 (7) 27 items, 18 right 9 wrong 27 (9) 101 items, 77 right 24 wrong 101 (24) how many items the scorer placed in each bracket, in red the wrong decisions (96 in all)
Claude Sonnet 5.5 ECE 0.072, accuracy 98.2%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 3% on average, spam in 0% of cases, 118 items 0% says 14% on average, spam in 0% of cases, 21 items 0% says 22% on average, spam in 0% of cases, 16 items 0% says 33% on average, spam in 0% of cases, 6 items 0% says 43% on average, spam in 50% of cases, 2 items 50% says 50% on average, spam in 0% of cases, 3 items 0% says 63% on average, spam in 80% of cases, 5 items 80% says 72% on average, spam in 50% of cases, 2 items 50% says 84% on average, spam in 100% of cases, 22 items 100% says 96% on average, spam in 100% of cases, 130 items 100% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 118 items, 118 right 118 21 items, 21 right 21 16 items, 16 right 16 6 items, 6 right 6 2 items, 1 right 1 wrong 2 (1) 3 items, 0 right 3 wrong 3 (3) 5 items, 4 right 1 wrong 5 (1) 2 items, 1 right 1 wrong 2 (1) 22 items, 22 right 22 130 items, 130 right 130 how many items the scorer placed in each bracket, in red the wrong decisions (6 in all)
Claude Opus 5.5 ECE 0.073, accuracy 98.5%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 4% on average, spam in 0% of cases, 116 items 0% says 14% on average, spam in 0% of cases, 33 items 0% says 22% on average, spam in 0% of cases, 10 items 0% says 33% on average, spam in 33% of cases, 9 items 33% no item no item says 63% on average, spam in 89% of cases, 9 items 89% says 73% on average, spam in 92% of cases, 12 items 92% says 85% on average, spam in 100% of cases, 20 items 100% says 96% on average, spam in 100% of cases, 116 items 100% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 116 items, 116 right 116 33 items, 33 right 33 10 items, 10 right 10 9 items, 6 right 3 wrong 9 (3) 0 items, 0 right 0 0 items, 0 right 0 9 items, 8 right 1 wrong 9 (1) 12 items, 11 right 1 wrong 12 (1) 20 items, 20 right 20 116 items, 116 right 116 how many items the scorer placed in each bracket, in red the wrong decisions (5 in all)
Claude Fable 5.1 ECE 0.053, accuracy 97.8%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 3% on average, spam in 0% of cases, 117 items 0% says 13% on average, spam in 0% of cases, 34 items 0% says 21% on average, spam in 0% of cases, 9 items 0% says 33% on average, spam in 67% of cases, 3 items 67% says 40% on average, spam in 0% of cases, 1 items 0% no item says 65% on average, spam in 50% of cases, 2 items 50% says 73% on average, spam in 67% of cases, 6 items 67% says 84% on average, spam in 89% of cases, 19 items 89% says 97% on average, spam in 100% of cases, 134 items 100% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 117 items, 117 right 117 34 items, 34 right 34 9 items, 9 right 9 3 items, 1 right 2 wrong 3 (2) 1 items, 1 right 1 0 items, 0 right 0 2 items, 1 right 1 wrong 2 (1) 6 items, 4 right 2 wrong 6 (2) 19 items, 17 right 2 wrong 19 (2) 134 items, 134 right 134 how many items the scorer placed in each bracket, in red the wrong decisions (7 in all)
Qwen3-4B, asked for its confidence ECE 0.077, accuracy 86.2%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 5% on average, spam in 15% of cases, 174 items 15% no item no item no item no item no item no item no item says 85% on average, spam in 77% of cases, 35 items 77% says 95% on average, spam in 91% of cases, 116 items 91% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 174 items, 148 right 26 wrong 174 (26) 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 35 items, 27 right 8 wrong 35 (8) 116 items, 105 right 11 wrong 116 (11) how many items the scorer placed in each bracket, in red the wrong decisions (45 in all)
Qwen3-4B, 25 lines ECE 0.176, accuracy 81.8%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 0% on average, spam in 22% of cases, 207 items 22% says 15% on average, spam in 75% of cases, 4 items 75% says 24% on average, spam in 100% of cases, 3 items 100% says 38% on average, spam in 100% of cases, 2 items 100% no item says 56% on average, spam in 0% of cases, 1 items 0% says 68% on average, spam in 100% of cases, 2 items 100% says 75% on average, spam in 100% of cases, 2 items 100% says 84% on average, spam in 100% of cases, 2 items 100% says 100% on average, spam in 96% of cases, 102 items 96% how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 207 items, 161 right 46 wrong 207 (46) 4 items, 1 right 3 wrong 4 (3) 3 items, 0 right 3 wrong 3 (3) 2 items, 0 right 2 wrong 2 (2) 0 items, 0 right 0 1 items, 0 right 1 wrong 1 (1) 2 items, 2 right 2 2 items, 2 right 2 2 items, 2 right 2 102 items, 98 right 4 wrong 102 (4) how many items the scorer placed in each bracket, in red the wrong decisions (59 in all)
Qwen3-0.6B, 25 lines ECE 0.327, accuracy 53.8%, 325 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 5% on average, spam in 51% of cases, 148 items 51% says 15% on average, spam in 45% of cases, 78 items 45% says 24% on average, spam in 45% of cases, 49 items 45% says 34% on average, spam in 42% of cases, 31 items 42% says 44% on average, spam in 44% of cases, 9 items 44% says 54% on average, spam in 88% of cases, 8 items 88% no item says 75% on average, spam in 100% of cases, 2 items 100% no item no item how often it really was spam 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of spam, by bracket of 10 points items 148 items, 73 right 75 wrong 148 (75) 78 items, 43 right 35 wrong 78 (35) 49 items, 27 right 22 wrong 49 (22) 31 items, 18 right 13 wrong 31 (13) 9 items, 5 right 4 wrong 9 (4) 8 items, 7 right 1 wrong 8 (1) 0 items, 0 right 0 2 items, 2 right 2 0 items, 0 right 0 0 items, 0 right 0 how many items the scorer placed in each bracket, in red the wrong decisions (150 in all)

Trips: is this a business trip?

The dataset

easymediumhardtotal
business505850158
personal595650165

350 generated, 323 usable (ok 305, reject 27, review 18). Languages: da 70, en 210, fr 70. Written label-first by Claude Sonnet 5.5, then audited blind.

Blind auditors: disagreement with the label

auditoritemseasymediumhardmean confidence
gemma4_26b350 7.5%10.8%8.0% 0.96
claude_sonnet_5_5350 1.7%5.4%8.0% 0.86
claude_opus_5_5350 1.7%3.1%6.0% 0.85

A disagreement that grows with the difficulty says the tiers mean something. A confidence that never moves says the auditor has none.

Scorers

scoreritemsaccuracyF1BrierECEin 0.40-0.60acc. in zoneeasymediumhardmedian msp95 mstotal timecost USD
Jev 1.13 (TypeSafe)
trips-jev-20261008-171547.csv		 323 94.7% 0.948 0.047 0.096 8.0% 65.4% 97.2%95.6%91.0% 28 63 0m10s 0.0050
Laya multilingual (local, 322M)
trips-laya-20261009-085930.csv		 323 57.0% 0.215 0.323 0.347 5.6% 44.4% 61.5%54.4%55.0% 35 40 0m22s 0.0000
Claude Sonnet 5.5
trips-sonnet-20261008-071603.csv		 323 96.6% 0.966 0.029 0.102 3.7% 33.3% 98.2%97.4%94.0% 1400 3152 9m52s 0.5348
Claude Opus 5.5
trips-claude_opus_5_5-20261008-084653.csv		 323 97.2% 0.972 0.032 0.108 5.3% 58.8% 99.1%98.2%94.0% 2006 8405 14m56s 0.9509
Claude Fable 5.1
trips-claude_fable_5_1-20261009-094541.csv		 323 97.2% 0.972 0.031 0.092 4.6% 60.0% 98.2%97.4%96.0% 1575 2505 9m18s 1.9210
Qwen3-4B, asked for its confidence
trips-ollama_qwen3_4b-20261008-071346.csv		 323 65.9% 0.742 0.288 0.296 0.0% n/a 66.1%65.8%66.0% 1062 1136 5m54s 0.0000
Qwen3-4B, 25 lines
trips-Qwen_Qwen3_4B-20261008-072949.csv		 323 81.1% 0.838 0.186 0.190 0.3% 100.0% 83.5%81.6%78.0% 856 863 4m42s 0.0000
Qwen3-0.6B, 25 lines
trips-local-20261008-073756.csv		 323 49.2% 0.658 0.278 0.185 13.9% 35.6% 45.9%51.8%50.0% 149 153 0m51s 0.0000
Jev 1.13 (TypeSafe) ECE 0.096, accuracy 94.7%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 4% on average, business in 0% of cases, 90 items 0% says 14% on average, business in 0% of cases, 20 items 0% says 23% on average, business in 0% of cases, 27 items 0% says 35% on average, business in 25% of cases, 8 items 25% says 44% on average, business in 18% of cases, 11 items 18% says 54% on average, business in 53% of cases, 15 items 53% says 66% on average, business in 50% of cases, 12 items 50% says 75% on average, business in 100% of cases, 9 items 100% says 85% on average, business in 100% of cases, 33 items 100% says 94% on average, business in 100% of cases, 98 items 100% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 90 items, 90 right 90 20 items, 20 right 20 27 items, 27 right 27 8 items, 6 right 2 wrong 8 (2) 11 items, 9 right 2 wrong 11 (2) 15 items, 8 right 7 wrong 15 (7) 12 items, 6 right 6 wrong 12 (6) 9 items, 9 right 9 33 items, 33 right 33 98 items, 98 right 98 how many items the scorer placed in each bracket, in red the wrong decisions (17 in all)
Laya multilingual (local, 322M) ECE 0.347, accuracy 57.0%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 3% on average, business in 32% of cases, 180 items 32% says 14% on average, business in 57% of cases, 60 items 57% says 24% on average, business in 57% of cases, 28 items 57% says 34% on average, business in 81% of cases, 26 items 81% says 45% on average, business in 100% of cases, 10 items 100% says 53% on average, business in 100% of cases, 8 items 100% says 65% on average, business in 100% of cases, 6 items 100% says 74% on average, business in 100% of cases, 4 items 100% says 83% on average, business in 100% of cases, 1 items 100% no item how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 180 items, 122 right 58 wrong 180 (58) 60 items, 26 right 34 wrong 60 (34) 28 items, 12 right 16 wrong 28 (16) 26 items, 5 right 21 wrong 26 (21) 10 items, 0 right 10 wrong 10 (10) 8 items, 8 right 8 6 items, 6 right 6 4 items, 4 right 4 1 items, 1 right 1 0 items, 0 right 0 how many items the scorer placed in each bracket, in red the wrong decisions (139 in all)
Claude Sonnet 5.5 ECE 0.102, accuracy 96.6%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 4% on average, business in 0% of cases, 92 items 0% says 12% on average, business in 0% of cases, 33 items 0% says 22% on average, business in 0% of cases, 23 items 0% says 33% on average, business in 0% of cases, 4 items 0% says 42% on average, business in 33% of cases, 6 items 33% says 54% on average, business in 0% of cases, 6 items 0% says 65% on average, business in 70% of cases, 10 items 70% says 74% on average, business in 100% of cases, 7 items 100% says 84% on average, business in 100% of cases, 39 items 100% says 94% on average, business in 100% of cases, 103 items 100% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 92 items, 92 right 92 33 items, 33 right 33 23 items, 23 right 23 4 items, 4 right 4 6 items, 4 right 2 wrong 6 (2) 6 items, 0 right 6 wrong 6 (6) 10 items, 7 right 3 wrong 10 (3) 7 items, 7 right 7 39 items, 39 right 39 103 items, 103 right 103 how many items the scorer placed in each bracket, in red the wrong decisions (11 in all)
Claude Opus 5.5 ECE 0.108, accuracy 97.2%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 4% on average, business in 0% of cases, 83 items 0% says 12% on average, business in 0% of cases, 43 items 0% says 21% on average, business in 0% of cases, 14 items 0% says 33% on average, business in 13% of cases, 15 items 13% says 44% on average, business in 22% of cases, 9 items 22% says 55% on average, business in 20% of cases, 5 items 20% says 65% on average, business in 86% of cases, 7 items 86% says 73% on average, business in 100% of cases, 11 items 100% says 85% on average, business in 100% of cases, 44 items 100% says 94% on average, business in 100% of cases, 92 items 100% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 83 items, 83 right 83 43 items, 43 right 43 14 items, 14 right 14 15 items, 13 right 2 wrong 15 (2) 9 items, 7 right 2 wrong 9 (2) 5 items, 1 right 4 wrong 5 (4) 7 items, 6 right 1 wrong 7 (1) 11 items, 11 right 11 44 items, 44 right 44 92 items, 92 right 92 how many items the scorer placed in each bracket, in red the wrong decisions (9 in all)
Claude Fable 5.1 ECE 0.092, accuracy 97.2%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 3% on average, business in 0% of cases, 92 items 0% says 12% on average, business in 0% of cases, 39 items 0% says 21% on average, business in 0% of cases, 10 items 0% says 32% on average, business in 0% of cases, 9 items 0% says 42% on average, business in 25% of cases, 12 items 25% says 56% on average, business in 0% of cases, 2 items 0% says 62% on average, business in 67% of cases, 12 items 67% says 73% on average, business in 100% of cases, 11 items 100% says 85% on average, business in 100% of cases, 24 items 100% says 94% on average, business in 100% of cases, 112 items 100% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 92 items, 92 right 92 39 items, 39 right 39 10 items, 10 right 10 9 items, 9 right 9 12 items, 9 right 3 wrong 12 (3) 2 items, 0 right 2 wrong 2 (2) 12 items, 8 right 4 wrong 12 (4) 11 items, 11 right 11 24 items, 24 right 24 112 items, 112 right 112 how many items the scorer placed in each bracket, in red the wrong decisions (9 in all)
Qwen3-4B, asked for its confidence ECE 0.296, accuracy 65.9%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 5% on average, business in 0% of cases, 51 items 0% says 15% on average, business in 0% of cases, 4 items 0% no item no item no item no item no item no item says 85% on average, business in 5% of cases, 41 items 5% says 95% on average, business in 69% of cases, 227 items 69% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 51 items, 51 right 51 4 items, 4 right 4 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 41 items, 2 right 39 wrong 41 (39) 227 items, 156 right 71 wrong 227 (71) how many items the scorer placed in each bracket, in red the wrong decisions (110 in all)
Qwen3-4B, 25 lines ECE 0.190, accuracy 81.1%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% says 0% on average, business in 0% of cases, 101 items 0% says 13% on average, business in 0% of cases, 2 items 0% no item no item says 44% on average, business in 0% of cases, 1 items 0% no item no item says 73% on average, business in 0% of cases, 1 items 0% says 85% on average, business in 0% of cases, 1 items 0% says 100% on average, business in 73% of cases, 217 items 73% how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 101 items, 101 right 101 2 items, 2 right 2 0 items, 0 right 0 0 items, 0 right 0 1 items, 1 right 1 0 items, 0 right 0 0 items, 0 right 0 1 items, 0 right 1 wrong 1 (1) 1 items, 0 right 1 wrong 1 (1) 217 items, 158 right 59 wrong 217 (59) how many items the scorer placed in each bracket, in red the wrong decisions (61 in all)
Qwen3-0.6B, 25 lines ECE 0.185, accuracy 49.2%, 323 items. Dashed line: perfect calibration. human review band 25% 50% 75% 100% 0% no item no item no item no item says 44% on average, business in 0% of cases, 1 items 0% says 55% on average, business in 34% of cases, 44 items 34% says 66% on average, business in 47% of cases, 163 items 47% says 74% on average, business in 56% of cases, 107 items 56% says 82% on average, business in 75% of cases, 8 items 75% no item how often it really was business 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% what the scorer said: probability of business, by bracket of 10 points items 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 0 items, 0 right 0 1 items, 1 right 1 44 items, 15 right 29 wrong 44 (29) 163 items, 77 right 86 wrong 163 (86) 107 items, 60 right 47 wrong 107 (47) 8 items, 6 right 2 wrong 8 (2) 0 items, 0 right 0 how many items the scorer placed in each bracket, in red the wrong decisions (164 in all)

Reading the columns: accuracy is being right at 0.5. Brier is the mean squared error of the probability. ECE is the average gap between stated probability and observed frequency over 10 bins, 0 is perfect. The 0.40 to 0.60 zone is where a human-review band would sit: a calibrated scorer is near 50% right there and sends its doubts there; an overconfident one is never there. Reliability diagrams, top panel: for the items where the scorer said 30 to 40%, the bar is how many really were; on the diagonal, the probability is honest. Bottom panel: how many items the scorer put in each bin, so you see whether it ever hesitates.