AI Has a Lean, a Blind Spot, and Tin Ears

Most AI Models Lean Left on Economics — Measurably

Ask an AI whether rent control reduces housing supply — the answer may reveal the model's politics more than the economics.

A research team extended a benchmark called EconCausal, which contains over 10,000 cause-and-effect relationships drawn from peer-reviewed economics journals. They separated out 1,056 cases where two ideological frameworks — call them 'let markets work' and 'government should intervene' — predict opposite outcomes. Then they tested 20 state-of-the-art AI models, including models you likely use, on those contested cases. Think of it like a bathroom scale that reads correctly for most people but consistently gives a lower number to people wearing a specific brand of shoe. The bias isn't dramatic, it isn't random — it's systematic. That's what they found. Eighteen of the 20 models were more accurate when the correct economic answer aligned with the pro-intervention view. The accuracy gap ranged from 9.7 to 15.1 percentage points, depending on the model. When models got the answer wrong, they disproportionately guessed in the intervention-friendly direction. Why does this matter? These models are being embedded in tools that advise on policy, summarize economic research, and answer questions from students and journalists. A model with a detectable lean isn't broken — it can still do useful work — but it's a measuring tape that's slightly off on certain measurements. The catch: the paper identifies the bias but doesn't fully explain where it comes from. Training data is the obvious suspect, but that's unconfirmed. And the study measures accuracy on empirically verified causal relationships, which is not the same as measuring political neutrality in open-ended conversations. One-shot prompting — giving the model a hint about both frameworks before asking — didn't fix the bias. So it's not easily patched with better instructions.

The AI Judges Grading Other AIs Are Missing Half the Mistakes

We're now using AI to grade AI — and the graders are failing to spot errors in more than half of cases in some tests.

Here's a problem that's gotten less attention than it deserves. As AI systems get better at generating images and text, we need reliable ways to judge whether their outputs are actually good. The solution the field landed on: use other AI systems as judges. Feed the output to a powerful vision-language model — an AI that can see images and read text — and ask it to score the quality. A team built a benchmark called FOCUS with over 4,000 test cases to stress-test these AI judges. Their method: take a correct, high-quality output and deliberately degrade it in specific, measurable ways — swap an object in the wrong position, add something the original image didn't contain, introduce a spatial contradiction. Then ask the AI judge to catch the problem. Think of hiring a food critic who can eloquently describe why a dish is bad in their written notes but still circles four stars on the rating card. That's almost literally what they found. In the most striking failure pattern, evaluator AIs would correctly mention an error in their written explanation — then give the flawed output a passing score anyway. In some categories of error, judges failed to flag problems in more than 50% of cases. Visual tasks — judging whether an AI-generated image matches a text description — were worse than text tasks. Asking judges to do pairwise comparison ('which of these two is better?') was more reliable than asking for a single score. The catch: this paper tests four evaluator models in controlled conditions. Real deployment is messier. But if the judges are unreliable, every AI benchmark that uses AI judging inherits that unreliability. We may be measuring progress with a broken ruler.

AI Scores Four Times Worse Than Humans on Audio Trivia

Play a foghorn, a jazz riff, or a bird call for the best AI audio system — then ask a question about it. The AI will probably fail.

We've been impressed by AI systems that can transcribe speech, identify songs, and describe what they hear. So it's worth asking: can they actually reason about audio the way a person does? Not just label sounds, but answer a question that requires genuine comprehension of what they just heard? The researchers behind AUDITA assembled 9,690 audio trivia questions built around real sound clips — not speech, but environmental sounds, music, animal calls, and similar material. Average clip length: about 37 seconds. They tested both humans and a range of state-of-the-art AI audio models on the same questions. Imagine giving a listening comprehension test to a class where the students can only hear the sounds through a wall — muffled, indirect, reconstructed from vibrations rather than direct sound. That's roughly where AI audio understanding is right now. Human participants averaged 32% accuracy on these questions — which sounds low, but reflects how hard the questions are (experts in specific categories reached 87%). AI models averaged below 9%. The paper also makes a methodological point worth understanding. Most existing audio benchmarks have a hidden backdoor: you can score decently by reading the question text, checking metadata, or recognizing common sound labels — without genuinely processing the audio. AUDITA is specifically designed to close those shortcuts. When you do that, AI performance collapses. The catch: 32% human accuracy is still modest, and the study doesn't tell us exactly which bottleneck is killing AI performance — whether it's the audio processing itself, the reasoning on top of it, or something about how questions were framed. That matters for figuring out how to fix it.

Here is what I think today's three stories are collectively telling you. We've been measuring AI progress with tools that are themselves unreliable. AI judges miss more than half the mistakes. AI models carry ideological leans we haven't fully mapped. AI audio systems score four times worse than humans on tasks we thought were within reach. None of this means the technology is fake — it means the measurement layer is younger than the capability layer, and that gap is starting to show. This matters because AI is being embedded in decision-support systems, research pipelines, and content evaluation tools right now, at scale. If the yardstick is bent, everything measured with it is off. The researchers publishing these papers are doing the unglamorous but necessary work of finding the cracks before they become load-bearing failures. That's the real progress happening today — not more capability, but more honest accounting of what's actually there.

The ideological bias paper doesn't suggest a fix, which means the next move is someone's replication or a counter-study using different benchmark methodology — watch for that in the next few months. More immediately: the AI judging problem is directly relevant to every major AI leaderboard in use today. If a lab or benchmark consortium announces changes to their evaluation methodology this spring, this line of research is likely part of why. The open question I'd most want answered: are the AI judges that are failing to catch errors also the ones certifying which models are 'better'? If so, our entire ranking system is built on a wobbling foundation.

• Wikipedia: Hallucination in artificial intelligence
• Wikipedia: AI alignment — overview of the field
• Wikipedia: Item response theory — how test difficulty and skill are measured