Skip to main content
OnTheRiceSEE THE WORLD EARLY.
Search the world…
OnTheRice’s Emerging DataTechnology

A correct repeat run can hide AI errors after memory changes

By OnTheRice · Sources and limitations · Report a correction

Two reports show different AI calculation errors linked to memory or saved state. On one 32 GB M1 Pro, PyTorch MPS reported wrong training gradients in 74 of 1,831 passes on version 2.14.1 and 179 of 767 on 2.13.0. A separate vLLM Qwen3-1.7B test gave token 16,448 instead of 271 with a cold cache, but matched after warming it. The proposed fix passed the author’s reproduction and CPU tests, while 13 broader tests failed and the proposal remained open. These are two bugs, not an industry failure-rate trend.

Current priceNot available
24-hour changeNot available

Source confidenceNot assessed

Published
Check timeNot applicable

What’s changing?

PyTorch concerns old data used after a GPU buffer stops. The separate vLLM report concerns startup order, empty caches and copied worker state. Warm runs can miss state-dependent errors.

Key points

  • The PyTorch MPS issue reports author-run wrong-gradient counts of 74/1,831 passes on torch 2.14.1 and 179/767 on 2.13.0, with zero exceptions, under four-process memory pressure on one 32 GB M1 Pro. The separate vLLM issue reports a cold-cache 16-token Qwen3-1.7B/FA2 prefill on RTX PRO 6000 Blackwell returning token 16,448 vs reference 271; warm cache matched. H200/FA3 matched 17/17 in a separate configuration. Open PR #59941 reports patched cold in-process/default-multiprocess repros and CPU tests pass, while 13 broader gated-model/shared-memory tests fail; PR remains unmerged.

What could change this view?

  • Two separate author-run, configuration-specific bugs, not an industry failure-rate estimate. PyTorch MPS reproducer used one 32 GB M1 Pro. vLLM error was reported for Qwen3-1.7B, FlashAttention2 and RTX PRO 6000 Blackwell; the 17/17 H200/FA3 result is a separate setup. Proposed vLLM fix is open; wider test pair still has 13 gated-model/shared-memory failures.

Bottom line

State-dependent starts and memory pressure can hide silent incorrect outputs; regression coverage should include cold/warm cache and concurrent/forked paths.

This is based on the information recorded with this signal.

Terms in simple English

These explanations help you read this signal. The saved forecast, prices, deadline and outcome rules stay the same.

Asset or investible
Something that can be bought or sold, such as a share, bond, fund, currency or cryptocurrency.
GPU or CPU
A GPU runs many calculations at once, often for AI. A CPU is the main general-purpose computer processor.
Cache
A stored copy used to avoid repeating work. An old cached copy can be out of date.
How likely is this call? · Three frameworks

This signal has no investible bullish or bearish call. A market-direction percentage does not apply.

Read the shared method · Read the public probability method

Evidence behind this view

Checks whether separate pieces of evidence support the same view.

Recorded connection

PyTorch MPS can consume stale buffers after a Metal command-buffer abort without raising; vLLM issue #59940 attributes a separate wrong-prefill path to import-order/cold-cache state and forked workers replaying captured attention arguments.

We have not established a separate count of independent evidence clusters for this card. Several reports may rely on the same event or data provider.

What past price moves show

Compares today’s price moves with recorded past patterns.

This card has no recorded investible call. A price-pattern comparison is not applicable unless a specific asset is added with evidence.

Evidence against this view

Shows evidence and risks that could challenge this view.

Recorded opposing evidence and risks

Two separate author-run, configuration-specific bugs, not an industry failure-rate estimate. PyTorch MPS reproducer used one 32 GB M1 Pro. vLLM error was reported for Qwen3-1.7B, FlashAttention2 and RTX PRO 6000 Blackwell; the 17/17 H200/FA3 result is a separate setup. Proposed vLLM fix is open; wider test pair still has 13 gated-model/shared-memory failures.

Why the direction changed

Explains recorded changes from expecting a rise to expecting a fall, or the reverse.

No documented bullish-to-bearish or bearish-to-bullish change is available for this card. We keep the original call and its time unchanged.

What people say vs the facts

Compares recorded public discussion with checked market facts.

No separate Reddit or public-discussion comparison with verified market data is recorded for this card. We cannot treat social attention as proof of a market move.

How past forecasts performed

Shows recorded forecast results and what they can teach us.

This card has no recorded investible forecast to score.

No tested mapping from evidence combinations to future success is recorded for this card. Confidence in sources and past outcomes do not establish a calibrated probability.

RETAINEDHistorical record
Sources, original wording and price checks

Published

First published through Sites 05 Oct 2026, 01:54:01 SGT

No explicit investible BULLISH or BEARISH call on this card.

What changed

Recorded public history

A warm run can't establish correctness when cache or memory state changes. On a 32 GB M1 Pro, the PyTorch MPS report found wrong gradients…

This is the first recorded public view. Further public updates will appear here.

  1. A clean warm run can hide state-dependent inference errors

    A warm run can't establish correctness when cache or memory state changes. On a 32 GB M1 Pro, the PyTorch MPS report found wrong gradients in 74/1,831 passes on 2.14.1 and 179/767 on 2.13.0; a vLLM cold-cache Qwen3-1.7B/FA2 repro returned token 16,448 versus reference 271, while warm cache matched. The PR author reports the proposed fix passing its repro and CPU tests; 13 broader tests still fail, and PR remains open. H200/FA3 matched separately. These are two bugs, not a failure-rate trend. Test cold/warm and concurrent/fork paths; next replay both fixes.

    First published by OTR: 5 Oct, 01:54 SGTPublished by OTR: 5 Oct 2026, 01:54 SGT
    [MPS] Command buffers aborted by the driver are silently ignored: wrong gradients · Issue #199680 ↗[Bug] VLLM_BATCH_INVARIANT=1 produces wrong tokens on a cold model-info cache · Issue #59940 ↗Proposed fix for cold-cache batch-invariant attention · PR #59941 (open, unmerged) ↗

Why the view changed

This is the first recorded view. Future source-backed updates will appear here.

Evidence, memory & review 3 linked sources

Directional takeaway: N/A: these configuration-specific software reports do not support an asset-specific listed-security conclusion.

OTR reasoning: PyTorch MPS can consume stale buffers after a Metal command-buffer abort without raising; vLLM issue #59940 attributes a separate wrong-prefill path to import-order/cold-cache state and forked workers replaying captured attention arguments.

Source-grounded facts (paraphrased)

  • Time not supplied: The PyTorch MPS issue reports author-run wrong-gradient counts of 74/1,831 passes on torch 2.14.1 and 179/767 on 2.13.0, with zero exceptions, under four-process memory pressure on one 32 GB M1 Pro. The separate vLLM issue reports a cold-cache 16-token Qwen3-1.7B/FA2 prefill on RTX PRO 6000 Blackwell returning token 16,448 vs reference 271; warm cache matched. H200/FA3 matched 17/17 in a separate configuration. Open PR #59941 reports patched cold in-process/default-multiprocess repros and CPU tests pass, while 13 broader gated-model/shared-memory tests fail; PR remains unmerged.

Source trail

Review status: SUPPORTED WITH QUALIFICATIONS

This evidence panel does not assign a score or probability. Any forecast estimate is shown separately with its own source and limitations.