AI Agents Need Reality Checks, Not Just More Thinking

Giving AI Agents a Running Checklist Doubles Their Success Rate

What if the reason AI assistants forget what they were doing is that they never had a proper to-do list to begin with?

Imagine you're assembling flat-pack furniture. You could try to hold every step in your head — or you could keep a physical checklist where you tick off each confirmed step, note what's still uncertain, and flag where you got confused. The second approach is obviously better. You'd think AI agents would already work this way. Mostly, they don't. A team building StructAgent at the paper's affiliated lab gave AI agents exactly that: a structured, running record of task state — what's been verified, what's still open, what the agent *thinks* it did versus what it can prove it did. They then tested it on OSWorld-Verified, a benchmark where an AI has to complete real computer tasks (booking, file management, multi-step web workflows) from start to finish. The results are hard to dismiss. A mid-sized model called Qwen3.5-9B went from completing 27% of tasks successfully to 46.9%. The larger Qwen3.5-27B jumped from 31.6% to 62.2%. Their best result — using a model called MiniMax-M3 — hit 78.9%, which the authors claim is the best openly available result on this benchmark. The catch: OSWorld-Verified is a benchmark, not your actual computer. Tasks are curated and defined in advance. Real-world tasks are messier, more ambiguous, and often don't have a clean definition of 'done'. We also don't have full access to the ablation analysis, so we can't be sure which part of the framework drives the biggest gains. Still — going from 27% to 47% with the same underlying model, just by keeping better notes? That's a real signal worth paying attention to.

Video AI Gets the Right Answer for the Wrong Reasons — And Nobody Notices

What if an AI aced your video quiz without actually watching the video?

Picture a student who gets 80% on a multiple-choice test by pattern-matching to the phrasing of the questions — without reading the passage. They score well. They understood nothing. A team of researchers built a benchmark called ST-Evidence to test whether AI systems that answer questions about videos actually *see* what they claim to see. The task they designed, called E-VQA (Evidence-backed Video Question Answering), doesn't just ask for an answer — it asks the model to point to the exact frames and regions in the video that support the answer, using dense tracked segmentation masks (think: a moving, shape-hugging highlight reel of the relevant objects). What they found is stark. Most open-source video AI models perform near random-chance when asked to select which part of the video backs up their answer — even when their actual answer is correct. High accuracy on the question. Near-zero accuracy on the evidence. The two don't correlate. To be fair, this isn't unique to cheap models. Even large systems like Qwen3-VL, OpenAI's o3, and Gemini 2.5 Pro struggled on the spatial evidence task. The team fine-tuned a 7B model called UniPixel on their new training data and got +27 points on temporal grounding and +14 points on spatial grounding — which shows the capability can be taught, not just measured. The catch: spatial grounding of this kind is genuinely hard to annotate, and the benchmark is new, so we don't yet know how well these scores generalise to real-world video tasks. But the core finding — that QA accuracy and actual visual understanding are decoupled — is important. It means a lot of AI video benchmarks may be measuring the wrong thing.

Letting AI Check Its Work Against Reality Beats Letting It Think Longer

More thinking time isn't always the answer — sometimes you just need to run the code and look at what breaks.

You've probably heard that AI models can get better answers if you give them more 'thinking time' — longer internal reasoning chains, or sampling many possible answers and picking the best. Both approaches help. Both also have a ceiling. This paper argues there's a third approach — and it works differently. Think of the difference between a plumber who estimates where a pipe leaks by reasoning about it, versus one who turns the water on and watches. The second approach doesn't require more thinking. It requires grounded feedback from reality. The team tested three strategies on hard coding tasks: reasoning-only (think longer), best-of-N (try many times, pick the best), and what they call interaction scaling — a loop where a 'proposer' writes code and a 'reviewer' runs it and reports what actually broke. Reasoning and sampling both plateau. The interaction approach reached 100% pass rate on hard coding benchmarks with zero variation across runs. They also tested this on visual layout tasks — making slides, figures, and web pages. Here the finding is almost funny: using a vision AI to judge whether a layout looks right made things *worse*. Using a tool that actually measured pixel geometry made things better. The VLM judge rated 14 out of 15 visibly broken figures as 'perfect'. The catch: this works best in settings where you can actually run code or measure pixels — domains with a real, checkable ground truth. For open-ended creative tasks or research questions, there's no equivalent feedback signal, and the authors themselves say interaction scaling offers minimal gains there. Small sample sizes in the visual experiments are also worth noting.

Read these three papers together and a clear pattern emerges. StructAgent found that AI agents double their success rate when they track verified facts rather than assumed ones. The E-VQA team found that video models answer questions correctly while being completely blind to the actual visual evidence. The interaction scaling paper found that running code and measuring pixels beats reasoning harder — and that AI judges lie confidently about broken outputs. All three are pointing at the same gap: the distance between what an AI *says* it knows and what it can actually *verify*. That gap is significant. It means a lot of apparent progress in AI benchmarks may be measuring fluency rather than understanding. The path forward doesn't seem to be bigger models thinking longer — it looks like tighter feedback loops, better ways to track what's actually been confirmed, and more willingness to run the test rather than predict the outcome. That's a more boring engineering story than 'intelligence emerges from scale.' It also might be the right one.

The ST-Evidence benchmark is brand new — I'd watch to see how frontier models perform on it over the next few weeks as teams run evaluations. On StructAgent, the 78.9% OSWorld-Verified number will attract replication attempts quickly; the real test is whether the gains hold on tasks not in the benchmark. The open question I'd most want answered: can interaction scaling work in domains without clean executable feedback — and if not, what does that tell us about the limits of AI in open-ended reasoning?

• Wikipedia: Metacognition (background on why 'knowing what you know' matters)
• Quanta Magazine: The hard problem of AI hallucination
• Wikipedia: OSWorld benchmark overview