A conference audited by machines, and what broke
Edition of 2026-08-14
Four things landed yesterday, and each one changes what someone can do today.
Machines audited a conference, and a quarter of it did not hold
Hugging Face ran an open challenge through July: point your own coding agent at a paper accepted to the year's largest machine learning conference, and try to reproduce it claim by claim. The scale explains why this was worth doing at all — the conference received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, a jump the post attributes partly to agents making experiments faster to run and write up. Peer review did not double alongside it, and one accepted spotlight paper carried a reviewer's admission that he had not checked the proofs carefully. In the challenge, 1,221 people took part and launched 2,962 cloud jobs. Aggregated per paper, 51% of those examined had at least one claim independently confirmed — 1,103 papers, and 3,978 individual claims confirmed by real experiments. But 23%, or 496 papers, had at least one claim falsified or contested, and 242 papers received opposite verdicts from independent teams checking the same claim. The named failures are concrete: the paper whose proofs went unchecked has a guarantee that degrades as the problem grows; a theorem about attention holds for 224 steps and then breaks, which explains why everyone who "verified" it had stopped checking too early; another paper's theory analyses one loss function while the code that produced its results computes a different one. Two arXiv corrections are in flight. The post is equally clear that some claimed falsifications were themselves wrong, including one built on comparing per-item against per-batch timings. Every logbook, verdict, artefact and agent trace was published, so the audit is itself auditable — which is the part worth borrowing. Hugging Face
The workhorse model, at half price until January
Google shipped a new version of its mid-tier Flash model three weeks after the last one, aimed at coding and at software that runs on its own for minutes at a time. The introductory price is $0.75/1M input tokens and $3.75/1M output tokens, running to the end of the year; Artificial Analysis, which measures models independently and benchmarked this one ahead of release, notes the standard rate is double, at $7.50 per 1M output tokens. Diary that for January rather than budgeting on today's number. Two findings from the independent measurement are more useful than Google's own benchmark wall: cost per task fell 30% against the previous generation at the discounted price, and the model produces roughly 340 output tokens per second, close to 3x the output speed of the two rivals it names, which works out at 1.7 minutes per task on average. Google also pushed it into its consumer agent product for paying subscribers in over 160 countries on launch day, which is a distribution decision rather than a developer preview. This is not the smartest model available; it is the cheap fast one that got cheaper and faster, and that is the claim the independent numbers support. Google DeepMind · Artificial Analysis
A songwriter that runs on a gaming PC
MiniMax published the weights of a music generation model, so anyone can download it and run it privately. Given lyrics and a written description of style, instruments and production, it returns a complete song with sung vocals of up to 5 minutes of music, in 32 khz stereo audio. Structure is explicit: the lyrics carry section tags for intro, verse, chorus, bridge and outro, and the card documents the model holding theme, rhythm and vocal identity across the full length. The hardware note is what makes this consequential — the card documents running it on 8 gb video cards by streaming the model layer by layer, and in about 22 gb with automatic offloading. That is a gaming machine, not a datacentre, and the lyrics never leave it. The limits are stated on the same page: it needs an NVIDIA card, it cannot stream output as it generates, the prompt is capped at 5,000 tokens, and the tempo, key and structure you ask for may not come back exactly. No independent evaluation of the output quality exists yet, so nothing above is a quality claim. MiniMax model card
Speed you cannot buy yet
Cerebras and OpenAI launched a new service tier in OpenAI's developer interface: the same top model, running on Cerebras hardware, at a stated speed of up to 750 output tokens per second. The mechanism is physical rather than algorithmic — Cerebras packs 44 gb of sram onto a single wafer-sized chip, so the weights stay on the chip instead of being fetched from separate memory for every token generated. As a demonstration, Cerebras ran an exam of 2,500 questions through both systems, finishing in 11 hours and 11 minutes against 78 hours for a rival model, and claims comparable accuracy. Two caveats do most of the work here. Every figure is Cerebras's, produced by Cerebras on its own hardware against a rival it configured itself, and Cerebras sells the chips; no independent measurement of this tier exists. And it is a limited preview for a selected set of customers, so the practical answer to "can I use this" is no, not yet. Treat the existence of the tier as the fact and the speed as a vendor claim until somebody neutral measures it. Cerebras
Signal faible
DeepSeek, whose flagship went generally available yesterday, has now published that model's weights under a permissive licence — the day after the commercial launch. The card documents three deliberation settings and a comparison table that is DeepSeek's own; the parameter count being quoted everywhere is not stated in the card text, so it does not appear here. Publishing a flagship's weights within a day of selling access to it is the pattern to watch, not the table. The card is at huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813.
Watch January, when the discounted price on Google's workhorse model is scheduled to double back to its standard rate.