Teutonic evaluator crashes, model validation threshold miscalibrated

τeuτonic experienced multiple evaluation server outages during a migration from 8B to 24B looped transformer models. The eval system ran out of disk space, then suffered inconsistent state issues requiring queue resets. As models scale up, a fixed gradient-norm threshold (500) designed for smaller models is now falsely rejecting valid 24B submissions; the team acknowledged the threshold needs model-size scaling. Inference bandwidth bottlenecks also emerged.

• •Eval server crashed multiple times; queue backlog forced model slot skipping
• •24B model shift and Quasar architecture changes require kernel optimization
• •Gradient norm validation threshold of 500 miscalibrated for 24B models
• •Model download bandwidth constraints slowing evaluation turnaround