Test whether the commonly cited claim that problem-solving courts (adult drug courts, mental health courts) meaningfully reduce recidivism holds up once completion-based survivorship bias, quasi-experimental design limits, and inconsistent outcome definitions are accounted for. Compare reported effect sizes against national baseline recidivism rates to judge practical significance, not just statistical significance.

Reviewed NIJ's Multi-Site Adult Drug Court Evaluation (MADCE, 2005-2010, 23 courts/8 states, quasi-experimental), GAO's two major program reviews (GAO-05-219 and its screening of drug court evaluations for methodological soundness), Bureau of Justice Statistics' 2018 nine-year recidivism follow-up of prisoners released in 2005 as a population baseline, and published meta-analytic effect sizes for mental health court recidivism outcomes.

Cross-checked completion-rate data against reported 'completer vs. comparison' framings to identify survivorship bias.

Let's start with the number everyone quotes: drug court participants get rearrested at 40% versus 53% for the comparison group, per NIJ's Multi-Site Adult Drug Court Evaluation (MADCE), the biggest quasi-experimental study we have — 23 courts, eight states, 1,156 participants against 625 comparison-site enrollees, tracked 2005 to 2010.

That's a real 13-point gap and I'm not going to pretend it isn't. What I will pretend to be mildly offended about is how rarely anyone mentions that MADCE is quasi-experimental, not randomized — comparison sites were chosen because they didn't run drug courts, not because participants were randomly assigned to them.

Isabel Vargas at El Colegio de México pointed this out to me while I was drafting this and getting overconfident: 'quasi-experimental' in this literature is doing an enormous amount of load-bearing work for a field that likes to cite its own effect sizes as if they came from randomized trials.

Then there's GAO-05-219, which screened a much larger pool of drug court evaluations and kept only 27 that met minimum methodological standards, covering 39 programs — meaning a large share of the literature GAO reviewed didn't clear the bar at all. Of the ones that did, re-arrest rates for completers ran 12 to 58 percentage points below comparison groups.

That's a massive spread, and 'completers' is the tell: national drug court completion rates run somewhere between 40 and 60%, meaning studies comparing graduates to non-participants are quietly comparing survivors to everyone else. The people who wash out — often the highest-risk, highest-need participants — get dropped from the 'success' side of the ledger.

That's not fraud, it's selection bias wearing a lab coat, and it inflates the apparent effect size in exactly the direction that makes for a good press release.

Now put this against baseline reality: BJS's 2018 update tracking people released from state prison in 2005 found 83% rearrested within nine years, 68% within three. That's the counterfactual world drug courts are competing against.

A 13-point reduction against an 83%-baseline recidivism population is not nothing — arguably it's more impressive than the raw percentage suggests, because you're carving a chunk out of a population that mostly reoffends no matter what happens to it.

But it also means anyone claiming drug courts are 'the answer' to mass incarceration is innumerate; you're moving the needle on a subset of a subset, not rewriting the base rate.

Mental health courts are the weaker sibling here and the numbers show it plainly. 20 relative to standard case processing — depending which studies get pooled, the field can't even agree on the direction of the effect, let alone its magnitude.

The same body of work finds essentially no effect, sometimes a negative one, on the mental illness these courts are nominally designed to treat. Eighteen percent — that's the estimated share of crimes among justice-involved people with serious mental illness that are actually motivated by the illness.

Eighty-two percent of the target population's offending has nothing to do with what the court is structurally designed to fix, which should worry anyone who treats 'diversion' as a synonym for 'treatment.'

My complaint isn't that problem-solving courts don't work. Some clearly reduce recidivism, especially adult drug courts running lower-risk dockets with strong judicial supervision.

My complaint is that the field keeps citing pooled effect sizes as though heterogeneity across sites, populations, and completion criteria doesn't exist, when NIJ's own numbers, GAO's own numbers, and BJS's own baseline all point to the same honest answer: it depends enormously on which court, which docket, and who stays enrolled long enough to be counted.

Report that, and the story is less triumphant. It's also true.