292

Papers

10/10

Roadblocks Active

3

Connections

This benchmark tests AI phone agents on 133 tasks inside a realistic iOS environment built around a single persistent user identity — contacts, calendar, messages, health data — across 26 custom apps. Frontier models average 52% overall but fall to 37% on tasks requiring coordination across multiple apps, and giving models access to structured accessibility data (XML) boosts scores by up to 26 points, revealing that vision-only agents lack the structural grounding needed for real personal-assistant work. The result matters because it is one of the first benchmarks to test cross-app reasoning and long-term user context together, exposing a gap that simpler single-app benchmarks have been hiding.

██████████ 0.9 agent-tool-use Preprint

ReadSaveConnections

Real computer tasks require fluent movement between graphical interfaces, the command line, and code — but existing benchmarks test these separately. WeaveBench assembles 114 tasks sourced from real user requests that mandate all three, and the best frontier model achieves only 41.2% pass rate. Critically, judging agents on final outcomes alone substantially overestimates performance; the benchmark's trajectory-aware evaluator, which checks intermediate steps and detects fabricated evidence, is a methodological contribution that could recalibrate how capable current agents actually are.

██████████ 0.9 agent-tool-use Preprint

ReadSaveConnections

A 32B vision-language model was first trained by imitating trajectories from frontier proprietary models, then fine-tuned with reinforcement learning on real Alibaba Cloud console tasks, reaching 63.5% success — within 1.8 percentage points of proprietary frontier models at 92% lower inference cost. The two-stage recipe (imitation to build a foundation, RL to develop autonomous judgment) is the mechanistic key, and the open-sourced benchmark on real cloud tasks provides a concrete measure of web-agent capability that is harder to game than synthetic environments. This is a meaningful demonstration that smaller open models can approach frontier performance on specialized professional workflows through structured training.

██████████ 0.9 agent-tool-use Preprint

ReadSaveConnections

Across 760 tasks spanning household navigation, driving, and social collaboration in eight different simulators, GPT-4o achieves only 17.4% task success and open-source models do worse, all constrained to first-person camera views with no privileged map data. Large performance differences across domains show that spatial reasoning skills do not transfer — a model that navigates a house poorly does not do so for a consistent, fixable reason. This matters for robotics and embodied AI, where the assumption has been that multimodal models already have basic spatial competence; this benchmark quantifies how far that is from reality.

██████████ 0.8 multimodal-understanding Preprint

ReadSaveConnections

When a single AI reasons through a visual question in one chain of thought, it tends to lock in wrong perceptual interpretations early and hallucinate. This paper trains three role-specialized agents (a main reasoner, a worker that gathers evidence, and a summarizer) that share a single underlying model but operate in parallel with distinct objectives — improving results on counting, referring expression, and hallucination benchmarks over single-trajectory baselines. The important point is that the improvement comes from role-specific training (reward signals tied to token segments), not just running the same model multiple times, suggesting that how reasoning is structured at training time shapes visual reliability.

██████████ 0.8 hallucination-grounding Preprint

ReadSaveConnections

Conversations between people are full of pronouns, implicit references, and context that only makes sense across multiple sessions — and current LLM agents fail substantially at retaining and using this kind of information. H2HMem tests agents across nine task types covering memory recall, reasoning over remembered facts, and applying memories to new situations, using synthetic multi-session dialogues that mirror real dyadic and group conversations. The benchmark identifies discourse phenomena like anaphora and deixis as specific failure points, giving researchers concrete targets for improving conversational memory architectures.

██████████ 0.8 long-context Preprint

ReadSaveConnections

Real-time task guidance — detecting that someone is about to make a mistake and intervening before it happens — turns out to be far harder for video AI models than simply answering questions about recorded footage. Current state-of-the-art video LLMs fail substantially on a new benchmark of egocentric cooking videos where participants deliberately make errors, but fine-tuning on a synthetic dataset of counterfactual intervention examples (normal cooking re-annotated with what a corrective instructor would say) measurably improves performance, especially for smaller edge-deployable models. This is relevant because real-time AI assistance in physical tasks — cooking, surgery, manufacturing — requires this reactive capability, and the paper shows it requires specific training data, not just larger models.

██████████ 0.8 multimodal-understanding Preprint

ReadSaveConnections

A small subset of attention heads inside robot-controlling vision-language models already reliably identifies which object the robot intends to approach — information that was previously assumed to require external state estimation. By reading these attention patterns and feeding them into a control barrier function, the system creates a real-time safety filter that prevents collisions without any additional training or auxiliary models, matching an oracle that has perfect simulator state. The significance is architectural: it demonstrates that safety-relevant world knowledge is already encoded in VLA model internals, meaning interpretability methods can unlock safety tools essentially for free.

██████████ 0.8 embodied-ai Preprint

ReadSaveConnections

Clinical AI systems that hallucinate — confidently recommending the wrong treatment — are especially dangerous, and this preprint proposes structuring medical reasoning through a knowledge graph that enforces hierarchical clinical logic before generating recommendations. Uncertainty-aware transformer components flag predictions the system is less confident about, adding a layer of transparency for clinicians. Note this is an unreviewed preprint with limited methodological detail, so the claims should be treated cautiously; the approach is architecturally sound but the validation evidence is not yet independently assessable.

██████████ 0.8 hallucination-grounding Peer-reviewed

Read

Code fails in three distinct ways — wrong tokens, wrong algorithmic structure, and wrong runtime behavior — and standard uncertainty methods borrowed from natural language processing only capture the first. This paper defines three complementary uncertainty axes (token entropy, pseudocode consistency, behavioral consistency across test executions) and shows that combining all three improves a key detection metric (AUROC) from 0.696 to 0.776 across five code language models. For AI coding tools used in production, knowing when to flag a suggestion as unreliable is as important as generating correct code, and this gives a practical multi-axis framework for doing so.

██████████ 0.8 alignment-safety Preprint

ReadSaveConnections