When AI Helps Doctors, Misses Tumors, and Fools Our Brains

Two AI chatbots were tested on real cancer patients — results are complicated

Could a chatbot read a patient's chart and correctly stage their cancer three times out of four?

A team at Zonguldak Bulent Ecevit University in Turkey fed clinical data from 180 real head and neck cancer patients — the kind of cancer that affects your throat, tongue, jaw, and sinuses — into both ChatGPT-4o and Gemini 1.5 Pro. Each AI was asked to do two things: stage the cancer (rank how far it has spread using the AJCC staging rulebook, which is the industry-standard system oncologists worldwide follow) and then suggest a treatment plan. The AI answers were checked against both that rulebook and a panel of human cancer specialists. Think of cancer staging like sorting parcels by zip code. Most of the time the addresses are clear. Both AIs got the right zip code roughly 75% of the time — three patients out of four. For a general-purpose chatbot, that's not nothing. The two models were statistically tied on staging. But treatment planning is more like actually routing the delivery truck, and here Gemini pulled ahead: 78.9% accuracy versus ChatGPT's 71.7%. That gap held up under statistical testing, so it's not random noise. The catch — and it's a real one — is that both models failed on roughly one patient in four. ChatGPT's errors weren't random either: it specifically stumbled on anatomically complex regions like the oropharynx (the back wall of your throat) and paranasal sinuses (hollow spaces around your nose). Those are exactly the spots where a wrong stage can mean the difference between radiation and surgery. No one is handing either model a prescription pad. But in clinics where access to specialists is limited, a 75% floor is a real starting point — not a destination.

When AI is too reliable, humans quietly stop paying attention

What if the most dangerous version of AI isn't the one that fails constantly, but the one that almost never does?

This paper — an experimental study that hasn't yet received a DOI, so treat it as work in progress — set up a simulated task where people had to identify phishing emails, those fake messages designed to steal your passwords or money, with the help of an AI assistant. The AI was tuned to perform very well. That's the trap. The researchers wanted to test something psychologists call automation complacency — the gradual erosion of your own vigilance when a tool is consistently right. Think of it like a passenger who stops glancing at the road once the GPS has given perfect directions for a thousand trips. The skill of reading the road quietly atrophies. The finding is that sustained exposure to a highly accurate AI shifts how people think. You move from what psychologists call System 2 processing — slow, deliberate, effortful checking — toward System 1, which is fast and heuristic: 'the AI flagged it, so it's fine.' The trouble is that when the AI eventually makes a mistake (and it will), the human watching it has by then lost the sharpness to catch the error. This is the paradox the title names: reliability, pushed far enough, becomes a liability. The honest caveat is that I don't have the full paper in front of me — the experimental numbers weren't available in the data I received, so I can't tell you the effect size. The concept itself is well-documented in prior safety research on aviation and nuclear plant operators. Whether this study confirms it cleanly for AI-assisted email tasks specifically is something we'll need the published version to settle. Keep this one in your 'watch' pile.

A philosopher argues AI systems cannot actually reason — and explains why

Knowing every recipe ever written is not the same as understanding why heat turns raw dough into bread.

Published in Frontiers in Artificial Intelligence, this paper is philosophy rather than engineering — no data, no training runs, no benchmark scores. The author's argument is worth understanding anyway, because it cuts at a question everyone who uses ChatGPT eventually asks: is it actually thinking, or is something else going on? The core claim, rooted in philosopher Robert Brandom's theory of meaning, is that genuine reasoning is not pattern-matching. It requires what the author calls reason relations — the ability to understand *why* one statement follows from another, not just that it tends to follow in a large corpus of text. A student who has memorised every past exam answer can score brilliantly on familiar questions. Put a novel problem in front of them and the machinery breaks, because there is no underlying structure to transfer — only stored outputs. The author argues that pure neural systems like today's large language models are, in principle, doing the second thing: very sophisticated pattern-matching over vast text. They are not following chains of inference; they are reproducing the shape of chains of inference. Pure symbolic systems — old-school logic engines — fare no better, because humans themselves cannot fully formalise the messiness of natural language into clean rules. The cautiously hopeful conclusion is that neuro-symbolic systems, hybrids that combine learned patterns with explicit logical structure, might be the path to something closer to genuine reasoning. The catch is large: this is a philosophical argument, not an experiment. It cannot be falsified by a benchmark score. Plenty of researchers would contest the definition of 'reasoning' the author uses. Think of it as a sharp map of what the debate is really about, not a measurement of where we are.

Put these three stories side by side and a single thread runs through them. Story one shows AI performing well enough on a high-stakes medical task to be genuinely useful — but failing in exactly the anatomically awkward corners where a human expert earns their salary. Story two shows that once AI gets reliable enough, the human in the loop starts to switch off — which means the failure mode is no longer just 'AI is wrong' but 'AI is right so often that nobody catches it when it's wrong.' Story three asks whether the whole enterprise is missing something structural: that even a highly accurate language model is doing something categorically different from the deliberate, justified inference we call reasoning. Taken together, this is not a picture of AI being overhyped or underpowered. It is a picture of a technology that is genuinely capable enough to be deployed in serious settings, but whose failure modes are subtle, intermittent, and partly created by its own success. That is a harder problem to solve than simple inaccuracy.

The cancer staging paper will likely attract follow-up studies at larger multi-institution cohorts — the 180-patient single-centre design is a known limitation the authors flagged, and replication in diverse hospital settings is the obvious next step. On the automation complacency front, watch for this paper to pick up a DOI and full data release; the experimental results will either sharpen or soften the warning considerably. The open question I'd most want answered: does brief, structured re-training of human reviewers restore vigilance, or does complacency return as soon as the AI resumes performing well?

• Wikipedia: TNM staging system — what the letters actually mean
• Wikipedia: Automation complacency — the broader research history
• Wikipedia: Neuro-symbolic AI — the hybrid approach the reasoning paper recommends