278

Papers

11/11

Roadblocks Active

0

Connections

ROSE is a controlled benchmark of 7,560 task instances that holds visual scenes fixed while varying whether a model must count objects or act on that count — revealing a drop of up to 44.5 percentage points between the two tasks, while humans score 98.8% on both. The gap is model-dependent and persists even when the same model correctly answers the counting version of the scene, ruling out perceptual failure as the cause. This exposes a structural weakness in current multimodal models: they can see but struggle to reason-to-act, which matters enormously for any real-world deployment where perception must drive decisions.

██████████ 0.9 multimodal-understanding Preprint

ReadSaveConnections

NRT-Bench deploys a five-role LLM operator team managing an abstract nuclear plant simulator and shows that adaptive multi-turn adversarial attacks cause 8.7–12.1% of sessions to result in loss of critical safety functions across four frontier models. A key finding is that model vulnerabilities are nearly disjoint — no single attack defeats all four models, and identical defense guardrails can both increase and decrease attack success depending on the model. This means safety properties of LLM agent teams cannot be assumed to transfer between models, and composite defenses must be tuned per-model rather than applied generically.

██████████ 0.9 alignment-safety Preprint

ReadSaveConnections

Using a carefully constructed benchmark of 834 manually curated Linux kernel vulnerable-patched pairs with strict temporal decontamination, this paper finds that 84% of nominally contaminated training samples carry no usable memorization signal, and that fine-tuning shifts output thresholds without changing underlying decision policies — a phenomenon the authors call 'calibration without comprehension.' Directional failure modes are stable and model-dependent, with directional failure indices ranging from -85.5 to +94.8 percentage points across backbones. This matters broadly: it suggests that fine-tuning LLMs on domain-specific security tasks may produce misleadingly confident models without genuine reasoning improvement, a risk applicable well beyond vulnerability detection.

██████████ 0.9 data-quality-curation Preprint

ReadSaveConnections

ENPIRE demonstrates that coding agents can autonomously improve robotic manipulation policies to 99% success on dexterous tasks by running closed-loop reset-execute-verify-refine cycles on real robot hardware without human intervention, using fleets of eight bimanual robot stations and frontier LLM coding agents in parallel. The core insight is that automating the environment-management layer (resetting scenes, verifying outcomes) is the bottleneck that previously required constant human presence, not the policy learning itself. The scalability claim — more parallel agents accelerate improvement — has direct implications for how AI research and robotic deployment pipelines may be structured going forward, though the reliance on proprietary APIs and custom hardware limits immediate reproducibility.

██████████ 0.8 embodied-ai Preprint

ReadSaveConnections

S-Agent augments vision-language models with a hierarchy of spatial tools — 2D object grounding, metric depth estimation, and 3D reconstruction — and a dual-memory system that accumulates spatio-temporal evidence across multi-view images and video rather than reasoning from isolated frames. Evaluated zero-shot on frontier models (Qwen3, GPT-5.4, Gemini 3) and in a fine-tuned setting across four spatial reasoning benchmarks, it shows that treating spatial reasoning as evidence accumulation rather than single-frame question answering is the key architectural choice. This is relevant to any AI system that must reason about 3D environments from 2D sensor streams, including robotics, augmented reality, and autonomous navigation.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

TimeProVe reduces VLM inference calls by 75% and total inference cost by 93% compared to dense video processing baselines, while improving accuracy by 7.3% over the strongest baseline on a new long-video QA benchmark for activities of daily living. It achieves this by using a lightweight action detector and small LLM to propose candidate evidence windows first, then invoking the expensive VLM only for targeted verification clips. The propose-then-verify pattern is a practically significant recipe for deploying powerful but costly vision-language models on long video tasks where frame-dense processing is economically prohibitive.

██████████ 0.8 efficiency-scaling Preprint

ReadSaveConnections

Tri-Info extracts three information-theoretic signals from vision-language-action model rollouts — action diversity, temporal consistency, and state-transition coupling — and uses these to predict task failure, achieving 83% accuracy on real-world robotic tasks where prior failure detectors fall to chance performance. The approach is training-free and model-agnostic because it operates on action distributions rather than learned features, making it portable across VLA architectures. Being able to predict robot failures before or during execution, without task-specific training, is a practical prerequisite for safely deploying VLA models outside controlled lab settings.

██████████ 0.8 interpretability Preprint

ReadSaveConnections

Phoenix resolves 75% of GitHub issues on a 24-instance SWE-bench Lite slice with zero pass-to-pass regressions, and achieves 100% correctness preservation across 42 real issues in a pilot deployment, using six specialized agents (Planner, Reproducer, Coder, Tester, Failure Analyst, Pull Request) coordinated via a label-based GitHub webhook state machine. Seven layered safety controls — including WAF filtering, token expiry, and permission boundaries — are built into the architecture to prevent deployment failures. The result matters because it demonstrates that safety and task performance in software engineering agents are not in direct tension when safety is designed into the coordination layer rather than bolted on afterward.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

MACR addresses the problem of an LLM having conflicting information in its training (parametric knowledge) versus a retrieved document (contextual knowledge), using modified semantic entropy to measure model confidence and then routing to a three-agent reasoning pipeline that explicitly identifies and resolves the conflict rather than defaulting to one source. The paper reports outperforming prior baselines on multiple benchmarks, though specific numbers and datasets are not visible in the available text, and no code is released, making claims difficult to verify independently. The mechanism is notable because most RAG and retrieval systems handle knowledge conflict implicitly through prompt design rather than explicit conflict detection, which introduces silent failure modes.

██████████ 0.7 hallucination-grounding Preprint

ReadSaveConnections

ScholarQuest constructs a benchmark of queries derived from 1,000+ computer science taxonomy topics across four intent types (method-oriented, setting-anchored, comparison-based, scope-controlled) and finds that even the best agentic search system achieves only 0.314 Recall@100, leaving most relevant papers unfound despite outperforming single-shot retrieval. The benchmark uses a shared million-scale arXiv backend (ScholarBase) to enforce identical retrieval conditions across systems, which is a methodological strength that makes comparisons meaningful. Low recall ceilings matter for AI research workflows and literature review tools that increasingly rely on LLM agents to find relevant work.

██████████ 0.7 reasoning-reliability Preprint

ReadSaveConnections