266

Papers

10/10

Roadblocks Active

2

Connections

This paper diagnoses a specific mechanism behind multimodal hallucinations: language models are hypersensitive to how a question is phrased, allowing text priors to override actual visual evidence during answer generation. The proposed fix, Decoding by Perturbation (DeP), probes for these language biases at inference time by perturbing the input text and using the divergence to down-weight text-driven predictions. This matters because it requires no retraining and targets the root cause — language dominance during decoding — rather than patching outputs after the fact.

██████████ 0.9 hallucination-grounding Preprint

ReadSaveConnections

DocSeeker identifies two concrete failure modes in multimodal LLMs on long documents: relevant evidence is buried in noise (low signal-to-noise ratio), and training datasets only supervise the final answer rather than the evidence-finding steps. The solution is a two-stage training pipeline — first teaching a 7B model to locate and reason over evidence via chain-of-thought distillation, then reinforcing evidence localization accuracy with a modified policy optimization reward (EviGRPO). The result outperforms both open- and closed-source models on five multi-page document benchmarks, and code is publicly available.

██████████ 0.9 hallucination-grounding Preprint

ReadSaveConnections

This paper derives a mathematical lower bound on how many test samples are needed to reliably detect miscalibration in an LLM: the required sample count scales as the cube root of the inverse error rate, meaning a model that fails 1-in-10,000 times needs roughly 46× more evaluation data than one failing 1-in-100. Critically, having the model evaluate itself provides literally zero information about its calibration. This is not a software engineering problem that better tooling can fix — it is a fundamental statistical limit that constrains AI compliance audits, safety certifications, and deployment monitoring at scale.

██████████ 0.9 reasoning-reliability Preprint

ReadSaveConnections

LMM-Searcher solves a practical bottleneck for multimodal agents: as search conversations grow to 100+ turns, storing all images in context becomes computationally prohibitive. The paper offloads images to an external file system and replaces them with lightweight text identifiers, loading visuals on-demand only when needed — cutting context overhead while preserving the ability to reason over images. A 30B-parameter model fine-tuned on 12K synthesised multi-hop trajectories matches or exceeds prior systems on four multimodal search benchmarks.

██████████ 0.8 multimodal-understanding Preprint

ReadSaveConnections

Even when vision tools (depth estimators, optical flow detectors) produce correct outputs, multimodal LLMs often fail to use that information — because raw pixel-level tool outputs are poorly matched to how language models process information. Perception Programs (P²) converts these dense outputs into compact, structured text summaries with normalised spatial coordinates, requiring no model retraining. Tested across six perception tasks, P² delivers a 22% average accuracy gain with GPT-4o Mini, and the code is publicly released.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

This paper demonstrates that adversarial lighting patterns — triangular projections optimised in a 9-dimensional parameter space using genetic algorithms — can reliably fool production vision-language models like CLIP, LLaVA, and BLIP in the real physical world, not just in digital simulations. The attacks transfer across model architectures and trigger severe semantic hallucinations in captioning and visual question-answering tasks. This is directly relevant for any deployment of vision-enabled agents in physical environments, as it establishes a low-cost attack vector that bypasses standard digital adversarial defences.

██████████ 0.8 hallucination-grounding Preprint

ReadSaveConnections

Rather than scoring model outputs as simply correct or incorrect, this paper creates instance-specific rubrics — per-image checklists of what a good answer must include — and uses these as reward signals for preference learning (rDPO). A 30B open-source judge scoring against rubrics comes close to GPT-4-level reward modelling, and the resulting model scores 61.01 on a comprehensive multimodal benchmark versus 52.36 for a style-constrained baseline. The key insight is that outcome-based filtering alone actually degrades performance (75.82) compared to rubric-based filtering (82.69), suggesting that answer quality requires richer evaluation criteria than binary correctness.

██████████ 0.8 multimodal-understanding Preprint

ReadSaveConnections

GUI grounding — having an AI agent click on the correct pixel in a user interface — typically fails because single-shot predictions cannot recover from small spatial errors in dense UIs like code editors. This paper replaces single-shot prediction with an iterative closed-loop process: the agent points, observes where it actually landed, and refines its click based on visual feedback. Multi-turn refinement significantly outperforms state-of-the-art single-shot models on click precision and task success, including in dynamic UIs where layout changes between turns.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

ARGOS reframes multi-camera person search — tracking individuals across surveillance feeds — as a reasoning problem with information asymmetry: the agent must ask limited questions of a witness to narrow down identity, location, and time of sighting. The benchmark contains 2,691 tasks across 14 real-world scenarios encoded in a Spatio-Temporal Topology Graph. Even the best model (GPT-4o family) achieves only 0.383 Turn-Weighted Success on the harder tracks, confirming the benchmark is far from solved and highlighting a gap between current LLM reasoning and real investigative workflows.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

IDEA extracts an LLM's decision-making knowledge into an explicit parametric model — mapping verbal expressions like 'likely' or 'rarely' to calibrated probabilities — by jointly learning those mappings and decision weights via an Expectation-Maximisation algorithm. This makes the model's reasoning auditable and editable without retraining: a user can inspect or override specific factor weights. A Qwen-3-32B backbone with IDEA scores 78.6% on decision benchmarks, outperforming both DeepSeek R1 (68.1%) and GPT-4-class models (77.9%), with code publicly released.

██████████ 0.8 interpretability Preprint

ReadSaveConnections