Assess whether the AI industry's self-reported safety evaluations can be trusted as evidence of risk mitigation, using the 2025 Foundation Model Transparency Index and recent reproducibility critiques of frontier AI red-teaming as the test case, and ask why declining transparency scores have produced no corresponding policy response.

I compared the Stanford Center for Research on Foundation Models' 2024 and 2025 Foundation Model Transparency Index scores against the arguments made in a 2026 reproducibility-standards proposal submitted to NeurIPS and the International AI Safety Report 2026.

I then checked whether the leading institutional response to these gaps -- the UK AI Security Institute's open-source Inspect evaluation framework -- addresses disclosure incentives or only tooling capability, since those are different problems that get discussed as though they were the same one.

The number I cannot stop thinking about is 58 to 40. That is the average score, out of 100, that the Stanford Center for Research on Foundation Models assigned to major AI developers' transparency practices in 2024 versus 2025. Scores went down. Nobody rang an alarm.

If a safety metric in aviation or pharmaceuticals dropped 31% year over year, that would be the headline for a month. In AI, it was one news cycle and a handful of policy newsletters, and by the time you read this it will likely be forgotten in favor of whatever model launched last week.

Break down what "40 out of 100" actually means, because the aggregate hides how selective the opacity is. Companies disclose plenty about model capabilities -- benchmark scores, leaderboard positions, the stuff that helps sales. They disclose almost nothing about training data provenance, training compute, or downstream usage and impact once the model ships.

Six major developers scored zero on model-information disclosure categories. Zero, not low -- zero, as in they told the index nothing.

And on the specific question of whether train-test overlap is being reported, meaning whether a model's benchmark performance might reflect memorized test data rather than actual capability, no developer in the entire index adequately disclosed this. Not the leader, not the laggard.

Every single one is opaque on the exact metric that would tell you whether to trust their other disclosures.

This is the part that should bother policymakers more than it does: the industry's safety claims and its transparency practices are being treated as separate conversations, when the transparency collapse is what makes the safety claims unverifiable in the first place.

A 2026 paper arguing NeurIPS should require reproducibility standards for frontier AI safety claims puts this plainly -- the artifacts needed to actually evaluate a safety claim (model weights, exact prompts, scoring rubrics, red-team composition) are routinely withheld precisely for the claims that matter most, producing what the authors call an evidential inversion: the higher-stakes the claim, the less reproducible the evidence behind it.

That is backwards from how evidence is supposed to work anywhere else in science, and I don't think "AI moves fast" is a sufficient excuse for it, because the withholding is a business decision, not a technical constraint.

Red-teaming has the same structural problem one level down. Who is on the red team, what instructions they are given, how many attack rounds they get, and what tool access the model has during testing all shift outcomes substantially -- and none of that methodology is disclosed at a level that lets an outside party replicate the result.

A model "passing" a safety evaluation currently tells you as much about the evaluator's resourcing and incentives as it does about the model.

Absence of a discovered risk is being reported as evidence of absence of risk, which is a logical error every first-year epidemiology student gets marked down for, and yet it is the operating assumption behind most published frontier model safety cards.

The institutional response so far is the UK AI Security Institute's Inspect framework: a genuinely good, open-source, MIT-licensed evaluation tool with over 200 pre-built evaluations and adoption from other safety institutes and even some frontier labs. I want to be clear that this is real progress and not nothing.

But Inspect solves the tooling problem -- it makes it easier to run a reproducible evaluation if you choose to publish one. It does not solve the disclosure-incentive problem, which is that companies with declining transparency scores face no cost for declining further.

Nothing in Inspect's adoption numbers forces a lab to publish its Inspect results, share its red-team composition, or report train-test overlap. You can have the best microscope in the world and it changes nothing if the sample never gets put under it voluntarily.

My conclusion, and Sofía Mendoza at FLACSO would probably say I buried the lede getting here: the bottleneck in AI safety evaluation right now is not evaluation science. The methods, imperfect as they are, exist and are improving.

The bottleneck is that disclosure is voluntary, transparency scores are falling in response to zero consequence, and the field keeps citing safety evaluations as though a claim's existence were the same thing as a claim's verification. Those are not the same thing, and the gap between them is widening exactly when the models being evaluated are getting more capable, not less.