AI Keeps Passing Tests It Should Be Failing

AI Finds Security Bugs — But Often for the Wrong Reasons

Imagine getting full marks on a maths exam where every line of working you showed was wrong.

Finding security holes in a large codebase is exhausting, painstaking work — the kind where you're hunting for a single unlocked window across an entire city block of software. A team building the DREA system decided to split that job between two AI agents: one does the high-level thinking, forming a hypothesis about where a vulnerability might be hiding, and a second, cheaper agent goes through the code doing the actual legwork. Together they improved detection rates from roughly 19–26% up to 30–42% across three different AI models, on a benchmark called RepoPairBench — 100 real Python security vulnerabilities drawn from public records. That's a genuine improvement. There's also a striking cost benefit: because the expensive reasoning model only handles strategy and the lightweight agent handles grunt work, the overall bill dropped by a factor of 16 to 48. So far, so good. Here's the catch that deserves its own sentence. Between 26 and 55 percent of what DREA flagged as correct were what the researchers call 'Lucky Hits' — the system named the right vulnerable file, but the reasoning it produced to justify that answer was wrong. Like a student who circled the right answer on a multiple-choice test but couldn't explain why. The team at the paper's lab identifies security reasoning quality as the shared bottleneck for all current AI models on this kind of task. The benchmark itself is small — 100 examples, no statistical significance tests — so treat these numbers as directional, not definitive.

AI Can Describe a Room but Can't Track What Changes Over Hours

Your phone can describe a photo of a kitchen in impressive detail — ask it to track what changed over four hours of footage and it's basically guessing.

There is a real difference between understanding a single snapshot and understanding a scene as it unfolds over time. A team of researchers built a benchmark called VSI-SUPER-WILD — 442 real-world videos, some running over four hours, across eight scene categories like kitchens, offices, and outdoor spaces. They extracted 6,980 human-verified questions and put thirteen mainstream vision-language AI models through their paces. The models had to track three things at once: where the camera-holder is in the space, where specific objects are, and how the overall layout of the environment changes. The results were blunt. Across the board, the thirteen models performed near chance level on tasks that required tracking the world over time. Think of it like a football commentator who can describe any single frame of the game with impressive accuracy — but ask them to recall where every player was twenty minutes ago, and they're making it up. Performance dropped off consistently as the videos got longer, which means errors accumulate rather than correct themselves. The researchers identified four specific failure patterns: spatial collapse (losing track of where things are in 3D), semantic shortcuts (guessing from context rather than watching), insufficient update (not registering that things have moved), and instance confusion (mixing up which object is which). No training was done here — this is purely testing models as they exist today. The open question is whether any training approach can actually fix this, or whether the architecture itself needs rethinking.

AI Speech Judges Are Copying the Label Instead of Tasting the Wine

An AI hired to score how natural your voice sounds might not be listening to you at all.

In speech technology, AI models are increasingly used as judges — automated systems that score things like how natural a synthesised voice sounds, or whether a speaker's emotion comes through clearly. It saves time and money over human evaluation. A research team decided to audit six of these AI judges across four speech qualities — emotion, naturalness, language identification, and speaker similarity — using publicly available audio datasets including RAVDESS and BVCC. The trick they played was elegant and a little alarming. In one test setup, they handed the AI judge deliberately wrong specialist labels alongside the audio — telling it, in writing, that a recording was angry when it was actually calm — and watched what happened. Five out of six judges accepted the wrong label and scored accordingly, dropping their emotion accuracy to 0.10 or below. A score of 0.10 is roughly what you'd get by random chance across emotion categories. Think of it like a wine critic who ignores the glass entirely and just reads the description on the back of the bottle. The presence of the audio barely mattered. There's a nuance worth noting: when wrong language labels were injected, judges were more likely to push back — because language is directly audible in a way that emotion is not. So the shortcut is capability-dependent, not universal. The practical implication is uncomfortable. If these AI judges are gaming the evaluation protocol rather than analysing actual audio, then the leaderboards they produce for speech systems may not be measuring what we think they are. Six judges is a small sample, and the finding needs replication across more systems and tasks.

All three stories today are circling the same problem from different angles. DREA's 'Lucky Hits' tell you that a correct output doesn't guarantee correct reasoning. The spatial benchmark tells you that models which look capable on static images fall apart when time is added. The audio judge audit tells you that models can ace an evaluation without actually doing the thing the evaluation was designed to measure. Put those together and you get a picture of AI systems that are very good at fitting the surface shape of a test — and much harder to trust on the underlying task. This isn't a reason to dismiss the progress: a 30–42% detection rate on security vulnerabilities is still real and useful. But it does mean we need evaluation methods that can distinguish genuine reasoning from sophisticated pattern-matching. Right now, in too many cases, we can't tell the difference. That's the work to watch.

The audio judge finding is going to land awkwardly at any upcoming speech synthesis competition that relies on automated scoring — and there are several major ones each year. Watch for reactions from the TTS community specifically. On the security side, the 'Lucky Hits' phenomenon deserves a proper replication study on a larger benchmark before anyone builds a production tool around it. The open question I'd most want answered: can you design a test that forces a model to show its reasoning before it commits to an answer — and does that change the Lucky Hit rate?

• Wikipedia: Hallucination in artificial intelligence
• Wikipedia: Common Vulnerabilities and Exposures
• DREA paper on arXiv