285

Papers

11/11

Roadblocks Active

0

Connections

This paper proposes four distinct measurement axes — factual precision, reasoning coherence, compliance reconstruction, and calibrated abstention — for evaluating how enterprise AI agents perform on long-horizon regulated decisions like loan qualification and insurance claims. Testing six memory architectures reveals that retrieval-based memory degrades factual precision while structured 'schema-anchored' architectures carry a scaffolding overhead, and simpler summarization baselines remain surprisingly competitive across most axes. This matters because regulated enterprise deployments need evaluation frameworks that identify the specific failure mode a regulator would care about, not just average accuracy.

██████████ 0.9 alignment-safety Preprint

ReadSaveConnections

This paper tests whether large language models subtly distort logical problems when formalizing them into Lean 4 proof code — a form of 'cheating' that would make proofs succeed by misrepresenting the original question. Across 303 logic problems, single-pass generation shows no systematic gaming, but a two-stage pipeline reveals qualitatively different failure modes: GPT-5 fabricates axioms during proof generation while DeepSeek-R1 mistranslates premises at the formalization stage. This matters because it shows that formal verification tools can diagnose where specifically a model's reasoning goes wrong, which is more actionable than knowing it fails.

██████████ 0.9 reasoning-reliability Preprint

ReadSaveConnections

This paper benchmarks whether leading multimodal models (Qwen, Gemma, Gemini families) can translate hazard awareness into safe behavior during simulated household tasks across 30 kitchen environments with six hazard categories. Models score high on static hazard recognition in QA settings but achieve near-zero success at actually avoiding those hazards during active embodied planning — a sharp alignment gap between knowing and doing. For anyone building AI agents that interact with physical or semi-physical environments, this gap means recognition-based safety evaluations are insufficient measures of real-world safety.

██████████ 0.9 alignment-safety Preprint

ReadSaveConnections

This paper tackles visual semantic arithmetic — inferring analogical relationships directly from images rather than text — which is harder than text analogy because models must extract the relevant relational concept from noisy visual context. The authors fine-tune vision-language models with reinforcement learning (GRPO) using a soft reward on a newly constructed benchmark (IRPD), outperforming supervised fine-tuning and prior embedding-based approaches. The result advances cross-modal relational reasoning, a capability needed for AI systems that must understand relationships between real-world objects rather than words.

██████████ 0.9 multimodal-understanding Preprint

ReadSaveConnections

This paper introduces CLVG-Bench, a 1,000+ item benchmark testing whether video generation models can reason across text, image, audio, and video inputs to produce logically constrained outputs. Even the best-performing model (Seedance 2.0) achieves below 25% on logically grounded generation tasks and approximately 0% on interactive generation tasks. The results put a concrete number on the gap between current video AI and the multimodal reasoning needed for real interactive applications, and the proposed evaluator (AVE) offers a scalable way to track progress without large annotation costs.

██████████ 0.9 multimodal-understanding Preprint

ReadSaveConnections

This paper benchmarks how well state-of-the-art LLMs can convert multi-turn natural language instructions into executable automation workflows on commercial platforms, using 27 production-level tasks across six domains. Models capture high-level intent but fail on structural correctness and stability, especially when requirements change mid-conversation; an agentic error-mitigation framework adds only about 5 percentage points to the resolve rate. This quantifies a practical bottleneck: the gap between an LLM understanding what you want and reliably producing working automation code is still large enough to block real enterprise deployment.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

This paper systematically evaluates 27 open-source models under 10 billion parameters across three deployment modes — plain prompting, single-agent with tools, and multi-agent collaboration — on 20 financial benchmarks. Single-agent tool use delivers the best performance-per-cost ratio; multi-agent setups add substantial overhead with limited accuracy gains. For teams considering small models to reduce inference cost, this provides concrete evidence that a single agent with a calculator, wiki search, and web search often matches multi-agent complexity at a fraction of the cost.

██████████ 0.8 agent-tool-use Preprint

ReadSaveConnections

This paper evaluates whether vision-language models can zero-shot predict what actions a non-humanoid robot (e.g., a wheeled arm rather than a humanoid) can usefully perform on objects across household, food, environmental, and construction domains. VLMs show a systematic conservative bias: low false positives but high false negatives, meaning they miss many valid robot-object interactions — especially for novel tools and unconventional manipulations. This conservative bias matters because AI-driven robot planners trained on human-centric priors will systematically under-utilize non-humanoid robot capabilities in the field.

██████████ 0.8 embodied-ai Preprint

ReadSaveConnections

This paper argues that measuring how similar an LLM's medical answer is to a reference answer (semantic similarity) is a poor proxy for actual medical correctness, and introduces a component-wise scoring framework (VB-Score) that checks factual entity accuracy separately. Three tested LLMs appear semantically close to correct answers but fail severely on entity-level factual checks, with health equity implications for under-served populations who may rely on AI health advice. The finding is a direct challenge to using embedding-based metrics as a certification threshold for medical AI deployments.

██████████ 0.8 hallucination-grounding Preprint

ReadSaveConnections

This paper addresses legal issue identification — determining which legal principles are actually relevant to a case — where GPT-4o achieves only 62% precision despite generating many candidate issues. A neuro-symbolic approach (LePREC) combines LLM-generated analytical factors with structured statistical reasoning over factor-issue correlations, outperforming end-to-end neural baselines by 30–40 percentage points. The result shows that injecting structured legal reasoning into the pipeline — rather than relying on LLM judgment alone — substantially improves precision in a domain where errors have real professional consequences.

██████████ 0.8 reasoning-reliability Preprint

ReadSaveConnections