Pretraining Failure Modes: MoE Causality Bugs and FP16 Precision Errors

Two MoE routing strategies: token routing and expert choice

Token routing lets tokens pick their top-k experts but can cause unbalanced expert load. Expert-choice routing has experts select preferred tokens, enforcing balance — but it is only usable during training because it requires seeing future tokens to make routing decisions.

Expert-choice routing breaks causality, creating a train/deploy mismatch

During training, which expert token N is assigned to depends on later tokens. At deployment, those later tokens don't exist yet. The model is therefore trained on information it cannot access in production — a fundamental mismatch that can silently degrade real-world performance.

Causality violation via expert-choice routing reportedly explains Llama 4's underwhelming results

This is a second-hand rumor, not a confirmed finding. If accurate, it would mean a subtle routing design choice — not model scale or data quality — was the primary driver of the model underperforming expectations.

Token dropping in expert routing can also break causality

When an expert drops a token it was supposed to process because a later, better-matched token arrives, the earlier token is ignored based on information that didn't exist when it was processed. This is a distinct causality violation from expert-choice routing.

Token-dropping causality violations reportedly affected Gemini 2 Pro training

This is a second-hand claim to be treated as rumor. It illustrates that causality bugs in MoE systems can manifest in multiple distinct ways, and that multiple frontier labs may have encountered related issues independently.

FP16 precision gaps at large accumulator values cause ~10x gradient calculation errors

At FP16 accumulator values above 1024, the gaps between representable numbers span multiple whole integers. Repeatedly adding small gradient values into a large accumulator can round each addition to zero. This causes gradient calculations to be off by ~10x — a precision error in the computed values, not a 10x slowdown in training speed.

Original GPT-4 training reportedly suffered this FP16 all-reduce precision bug

The bug reportedly occurred in the all-reduce collectives — the distributed step where gradient values are summed across GPUs. Using FP16 at that stage introduced the precision error. This is a second-hand, unverified claim.

Source argues new, bespoke training failure modes keep emerging at each new scale

This is an attributed opinion, not established fact. The source's position is that there is no finite checklist of failure modes — each new scale regime introduces novel numeric and systems issues, contrasting with a more optimistic view that the problem space is enumerable and solvable once.

Same source is bearish on AI fully automating GPU kernel writing anytime soon

The source characterizes GPU kernel authoring as closer to an AGI-complete problem — requiring the kind of general reasoning and systems intuition that current AI cannot reliably provide. This is an opinion, not a consensus technical position.