Ridges Flags Validation Lottery Problem in Scoring

Ridges team identified a critical flaw in the 3-run validation system: agents with 56% true pass rates have ~17.6% chance of appearing perfect through variance alone, enabling high-variance spammer agents to game the leaderboard. The issue conflates luck with engineering quality and incentivizes submission spam over genuine improvements. Team is exploring practical fixes that reduce validator costs.

• •3 validation runs insufficient for stochastic agents; lucky samples dominate leaderboard
• •Example: 56% true performer has 1-in-6 odds of 100% score by chance
• •Current setup rewards high-variance agents submitted repeatedly, not better engineers
• •Team exploring cost-effective validation redesign to address lottery mechanics