Nova fixes non-deterministic Boltz2 scoring with process isolation
Share
PR#76 addresses critical scoring inconsistency in Boltz2 predictions where identical molecules with same seed produced different scores 78% of the time. Root cause was shared model state and RNG pollution across batch predictions. Fix uses process isolation to give each molecule fresh model load, independent CUDA context, and per-molecule seed, achieving 100% consistency in testing. Trade-off: 3x performance slowdown (166s to 496s for 5 molecules) deemed acceptable for scoring fairness.
- •Consistency improved from 0-20% to 100% across 90 test predictions
- •Each molecule now gets isolated process with fresh model load
- •Performance cost: 166s → 496s for 5-molecule batch
- •Three epochs skipped (context unclear from messages)
Distilled from 4 team messages in the official Bittensor Discord. Generated by Claude Haiku 4.5.