Research Reality Cards

WebAnything achieves higher fidelity metrics and improved task success rates compared to human-assisted methods by automating the reconstruction of web environments.

Constructed 244 website artifacts supporting 12,984 executed tasks.

Interaction coverage is constrained by captured content and replay capabilities.

OpenFrontiers provides a comprehensive resource linking a corpus of 12,151 open research problems with a benchmark of 200 problems for evaluating LLMs in scientific rediscovery.

Closed-book rediscovery was rare, with at most 7 of 40 resolved problems successfully rediscovered.

The evaluation protocol's reliance on closed-book conditions may limit reproducibility of results.

Localized posterior calibration significantly reduces temperature and zonal-velocity errors in ocean models compared to deterministic and global-posterior calibration.

Localization reduces aggregate predictive errors relative to the global posterior.

The method's effectiveness may vary based on the context distribution and the specific application, which could affect reproducibility.

ThinkJev allows for a significant reduction in decision-making error rates by enabling a model to think before answering, achieving a realized error of 3.9% at a 5% target error rate.

Qwen3.5-27B answers 24.5% of test decisions with a realized error of 3.9% using the new system.

The method's effectiveness is dependent on the statistical test guarantees, which may not generalize across all decision types.

LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% compared to existing methods.

Achieved a reduction in training time by 21.2-44.8%.

The paper does not specify the authors or provide detailed experimental setups, which may hinder reproducibility.

GeoUG achieves high accuracy in spatial reasoning tasks while minimizing generation calls, demonstrating the effectiveness of integrating geometric evidence.

GeoUG achieves circular accuracies of 97.9%, 90.8%, and 94.2% on CV-Bench-3D, 3DCueBench, and DA-2K.

The paper does not specify the authors or provide detailed experimental setups, which may hinder reproducibility.

The study demonstrates that vision-language models can non-deterministically transform images to reflect sensitive attributes like race and sexual orientation, often reinforcing stereotypes.

Generated images are statistically distinguishable and accentuate deep-seated stereotypes.

The findings may not be reproducible across all VLMs or datasets, raising concerns about generalizability.

Localized steering in CohortFlow reduces mean correction latency by 64.8% and model calls by 38.1% while maintaining requirement-level accuracy.

Achieved 77.0% requirement micro-F1 with localized steering compared to 76.4% for full regeneration.

The statistical significance of the performance improvement is not established (p=0.493).

Skill adaptation significantly improves the performance of weaker models by reformulating existing skills to better match their capabilities.

The adaptation improved three weak executors by 5.1-14.1 points over the original skills.

The study's findings may not be universally applicable to all models, as benefits vary significantly across different model capabilities.

The proposed data generation strategies significantly improve classification accuracy by addressing data insufficiency and class imbalance.

Substantial improvement in classification accuracy observed with the EfficientNetV2B0 architecture.

The reproducibility of results may be limited by the specific data generation techniques and their implementation.

OmniBrain-0 achieves strong cross-domain task performance with lower inference and execution overhead compared to its closed-source teacher.

Demonstrated efficient transfer and consolidation of embodied capabilities into a compact unified agent.

The paper does not provide detailed information on the reproducibility of the training framework and the specific implementations of the models.

The paper establishes a framework that emphasizes the need for human understanding and direction in scientific research, even as AI capabilities advance.

The framework includes a two-part test for delegating research tasks to AI, focusing on human understanding and human direction.

The framework's applicability may vary across different scientific fields and may not address all ethical concerns related to AI use in research.

The study establishes a model that quantifies time saved by AI assistance and the associated defect costs, demonstrating that over-trust in AI can lead to significant time savings but at a higher cost in terms of defects.

AI helped on somewhere between 3% and 82% of the 246 issues analyzed, with a slowdown of 19% to 24% depending on the estimand.

The results are based on a specific dataset and may not generalize across different contexts or applications.

Any3R achieves consistent accuracy in 3D reconstruction across diverse camera types without the need for lens-specific retraining.

Outperforms thirteen central cameras across three datasets in accuracy.

The method's performance may vary with extreme camera types not included in the benchmark.

Instruction tuning significantly alters the number-choice heuristics in language models, leading to a preference for odd numbers and avoidance of round numbers.

15 out of 18 evaluated model pairs showed that instruction tuning made the distribution of chosen numbers less uniform.

The study does not provide specific author names or detailed methodology, which may limit reproducibility.

MedJev achieves accuracies approaching GPT-4 performance on medical benchmarks while maintaining low computational costs and efficient local calibration.

MedJev achieves an AUROC of 0.926 for In Hospital Mortality predictions.

Patient level predictions exhibit substantial miscalibration, particularly in In Hospital Mortality.

ObjectTwin establishes a measurable interface for asset correspondence, revealing that generated assets lose 0.302-0.497 MaskIoU when compared to scan-derived geometry.

Visible, anchored generated assets lose 0.302-0.497 MaskIoU under the replacement of object representation.

The evaluated conservative pose refinement does not systematically recover the observed discrepancies.

Lessons derived from chance failures can lead to suboptimal tool selection in subsequent decisions, with a significant increase in the likelihood of choosing a less reliable tool.

After a chance failure of the better tool, the lesson from that failure led to a 11.6% selection rate of the worse tool in subsequent runs.

The study's findings may not be generalizable due to the synthetic nature of the worlds and the specific conditions under which the lessons were evaluated.

PAI-KNN achieves competitive performance in data stream classification under dynamic class imbalance and concept drift without relying on memory modification.

PAI-KNN performs particularly well in scenarios involving dynamic class imbalance and concept drift, evaluated on multiple synthetic data streams.

The paper does not specify authors or detailed experimental setups, which may hinder reproducibility.

ReCoPII-Ko corpus achieves 88.42% agreement on KEEP labels, significantly outperforming SPY-style augmentation which only achieved 26.27%.

Classifiers trained on ReCoPII-Ko over-mask less than those trained on SPY-style augmentation.

The framework relies on specific entity relations and may not generalize well across all domains or languages.

The study demonstrates that a task-conditioned utility score is a better predictor of visual input effectiveness for downstream tasks compared to conventional metrics.

The proposed MLLM-based embodied IQA agent framework shows that task-conditioned utility scores outperform generic MLLM-based IQA scores in predicting task success.

The study's findings may be limited by the specific context of the VisDrone evidence study, which may not generalize to all embodied tasks.

Dynamic evaluation allows softmax-attention models to reduce next-token loss by 5-8% over a document without training, outperforming frozen models with full context.

Masked-diffusion models show an improvement of 17-20% in next-token loss when gradient updates are applied.

The adaptation observed is specific to style and vocabulary rather than retrieval, which may limit generalizability.

FinLLM-KG improves sentiment predictions by incorporating company-level knowledge from SEC 10-K filings, leading to better portfolio performance compared to state-of-the-art methods.

FinLLM-KG consistently improves portfolio performance over the state-of-the-art baseline.

The model operates as a black box for existing LLMs, which may limit broader applicability and understanding.

The Trinity framework improves mean performance by 34 points across 28 training benchmarks and 26 points across 18 unseen benchmarks through a capability-driven approach.

Evolved Harnesses improve mean performance by 34 points across 28 training benchmarks.

The complexity of constructing heterogeneous, verifiable environments may hinder reproducibility.

Standardizing features for an SVM significantly improved macro recall from 0.51 to 0.78, demonstrating that preprocessing has a larger impact than model architecture.

An SVM with standardized features achieved a macro recall of 0.78.

The main limitation is the confounding effect of recording sources on label assignment, which may affect reproducibility.

The Poly-Ripple framework allows multiple clients to collaboratively estimate indoor radio maps without requiring model updates, outperforming existing ICL baselines.

Poly-Ripple consistently outperforms representative ICL baselines in extensive experiments.

The method relies on private demonstrations from clients, which may limit reproducibility across different environments.

The paper identifies a significant shift towards deep learning and generative models for automated defect recognition in NDI.

The review synthesizes a broad range of literature, illustrating the transition from manual to automated inspection techniques.

The scarcity of standardized open-access datasets and the 'black box' nature of AI models hinder widespread adoption.

The proposed dual-memory framework outperforms single-paradigm baselines by 4.67% in multimodal tasks, effectively bridging the gap between episodic and latent memory.

State-of-the-art performance with a +4.67% improvement over vanilla Episodic memory.

The framework's reliance on a training-free routing mechanism may limit reproducibility across different tasks.

The study demonstrates that OCR quality cannot be fully characterized by aggregate transcription metrics alone and emphasizes the need for entity preservation evaluation.

Qwen 3.6 achieves the highest entity coverage across all tested tolerances.

The paper does not specify the authors, which may affect reproducibility and validation of the findings.

The study provides mechanisms that ensure truthful declaration of context mixtures and unbiased audits, which are crucial for long-lived AI deployments under regulatory frameworks.

A proper-scoring penalty on the declared mixture bounds the excess failure of a best-responding deployer by O(λ^{-1/2}) in the penalty weight.

The study does not evaluate any language model or production system, which may limit the applicability of the findings.

Saga effectively connects shared patient states to predict future clinical events and generate multimodal images, achieving a one-year progression AUROC of 0.882.

Event F1 score of 0.672 in free rollouts, outperforming a longitudinal Transformer by 0.119.

The model's performance may vary significantly when patient conditioning is removed, indicating potential issues with generalizability.

The introduction of the ICF Laser Pulse Shape Design System (LPDS) allows for direct mapping of desired outcomes to optimized laser pulse shapes, significantly improving the design process.

The framework includes a multi-objective loss function and mechanisms for fine-grained control over pulse characteristics.

The transfer of the framework to real experimental shots is limited, which may affect reproducibility.

LeWorldModel's action sensitivity varies significantly across perturbation scales, with a notable reduction in Gram discrepancy when increasing perturbation scale.

Increasing perturbation tenfold reduces median normalized Gram discrepancy by roughly 70% on Reacher and 47% on Cube.

LeWM's errors are higher at smaller scales, indicating potential limitations in its performance and reproducibility.

EKT-Router achieves the highest mean score of 61.4 across 6 embodied QA benchmarks, outperforming other systems.

Achieves a mean score of 61.4 on embodied QA benchmarks and 74.7 on general vision-language benchmarks.

The paper does not specify the authors or provide detailed implementation information, which may hinder reproducibility.

The introduction of IRON-Meme provides a comprehensive human-annotated dataset that reveals the interconnectedness of identities, roles, and templates in political memes.

Evaluation of nine vision-language models shows that frontier-scale and reasoning models achieve the highest performance.

Accuracy on social identities and narrative roles decreases for less frequent labels, which may affect generalizability.

Human reviewers can reduce algorithmic false positives, but their performance declines at stricter security thresholds.

801 human reviewers were tested across three security configurations using the Difficult Facial Image Comparison Test (DFICT).

The study's findings may vary based on the specific algorithms used and the security configurations tested.

ESMs outperform GPTs on most downstream tasks and demonstrate promising scaling behaviors with increased model size and training data.

ESMs were evaluated across 26 downstream tasks, showing superior performance compared to GPTs.

The main limitation is the potential complexity in reproducing the energy-guided pre-training process.

The model achieves higher accuracy than leading models with similar memory footprints and speed across various benchmarks for agentic capabilities.

Achieves a TPS of 84 on RTX Spark, comparable to models with much lower active parameter counts.

The paper does not specify authors or detailed experimental setups, which may hinder reproducibility.

Boa Compressor achieves performance that matches or surpasses state-of-the-art offline compression methods while being significantly faster, reducing end-to-end latency.

Boa scales to compression ratios as aggressive as 16x and operates in linear time.

The paper does not specify the authors or provide detailed reproducibility metrics.

LLMs show little loss aversion and diminishing sensitivity compared to humans, but exhibit human-like probability distortion.

Pooled loss aversion $BB$ is 1.26, significantly lower than the strongest human estimate of 1.96.

The findings may vary significantly based on specific model families and fitting methods.

The proposed Differentiable Diffusion Conditioning (D2C) method transforms a frozen conditional diffusion model into an end-to-end gradient-based optimizer, achieving superior results in real-world tasks.

Achieved the strongest results on two real-world tasks and the best average rank across four Design-Bench tasks.

The method relies on a fixed dataset and may face challenges in generalizing to new test-time conditions.

The study demonstrates that multigraph neural networks are vulnerable to learned structural attacks, and adversarial training can effectively improve their robustness.

Adversarial training consistently improves robustness against learned structural attacks, as shown in experiments on synthetic and realistic AML datasets.

The main limitation is the potential complexity in reproducing the adversarial training framework due to the specific optimization methods used.

The paper introduces a constrained-transformation framework that allows for the governance of academic revisions by linking evidence and judgment to an authorized claim boundary.

The Academic Writing Filter screens candidate revisions to ensure they adhere to the defined admissible space.

The framework is not a universal definition or classifier of good academic writing, which may limit its applicability across different contexts.

The embedding-based subset selection approach using a greedy k-center algorithm improves ASR fine-tuning effectiveness, particularly with smaller training subsets.

k-center selection improves performance over random sampling, especially at smaller subset sizes.

The approach has only been evaluated on Hebrew and Danish, limiting generalizability to other languages or domains.

The paper emphasizes the integration of AI into mechanical engineering practices while ensuring that scientific rigor and safety are not compromised.

The discussion includes principles for responsible AI adoption in ME, focusing on physical consistency and rigorous validation.

The paper highlights educational and ethical challenges that may hinder the reproducibility of AI applications in engineering.

The integrated framework significantly improves assignment compliance and opportunity balance in territory carving for enterprise GTM planning.

Optimizer-generated scenarios improved assignment compliance while maintaining comparable seller-level opportunity distribution.

The manual nature of the previous processes may limit the generalizability of the findings to other contexts.

Phronix enables agents to solve recurring tasks on the first attempt in 93-95% of cases, significantly improving reliability and reducing solution non-determinism.

Phronix generates 7.6-30.8 durable capabilities per task without user instruction.

The effectiveness of Phronix may depend on the specific implementation of the predictor-validator-corrector loop and the quality of the initial harness.

The second approach using pre-trained BERT embeddings fine-tuned for the PCL task achieved an F1 score of 0.528 over the development set.

F1 score of 0.528

The paper does not specify the dataset size or diversity, which may affect reproducibility.

AudioLift produces 3D talking avatars that can capture extreme facial expressions and maintain strong multi-view consistency without relying on parametric face models.

Achieves improved geometric completeness and strong multi-view consistency across a full 360° viewing range.

The method's reliance on generated multi-view videos may pose challenges in reproducibility and generalization to diverse datasets.

SyncVid successfully synchronizes the diffusion process in video generation, resulting in improved perceptual quality and cross-view consistency.

The method achieves enhanced coherence by lifting a monocular video into a canonical 4D representation and applying a cross-view-consistent whitening procedure.

The main limitation is the reliance on a specific architectural adaptation (LoRA) without detailed reproducibility guidelines.

DirCast outperforms both rule-based and zero-shot directors, and its commentary is significantly better than that of existing streaming commentators.

DirCast's commentary wins 29.2% of comparisons against GPT-4o under the Proact-VL protocol.

The dataset is limited to 27 professional Dota 2 matches, which may affect generalizability.

The study demonstrates that adopting a multi-valid learning framework and graph-based reasoning improves accuracy, stability, and robustness in structured generation tasks.

Experiments show consistent improvements in accuracy and stability under varying levels of ambiguity.

The paper does not specify authors or detailed experimental setups, which may hinder reproducibility.

The research demonstrates that combining articulatory, facial, acoustic-prosodic, and turn-taking measures improves the prediction of perceived conversation success.

Utilized three spontaneous dyadic-interaction datasets to achieve a more comprehensive understanding of conversation success.

The study may face challenges in generalizing findings across diverse populations due to the specific datasets used.

PnP* is the first practical solver that guarantees efficient computation and returns a certifiable globally optimal solution for the Maximum Likelihood Estimation of Perspective-n-Point.

PnP* returns a provably correct camera pose at over 10 Hz.

The paper does not specify limitations or reproducibility concerns.

The SCI effectively measures belief organization in language models, achieving a 79.5% recovery rate of keyed beliefs by clinically trained raters.

Validation studies show belief adapters are target-specific on every backbone with a +17.7 points improvement.

The study's reliance on specific clinical raters may limit generalizability and reproducibility across different contexts.

Researchers should validate the exact difference they plan to report in LLM evaluations, not just the overall scorer accuracy.

In a balanced audit, overall F1 was 0.916, but prompt-specific errors ranged from 2.6% to 33.3%.

The uncertainty in prompt-specific estimates and the potential for misrepresentation of performance due to audit composition.

The study reveals a significant discrepancy between benchmark-style exact-match scoring and human-calibrated evaluations, indicating that current evaluation methods may not accurately reflect LLM tutoring effectiveness.

The measured Remediation Success Rate was 42.1% according to benchmark scoring, while human-calibrated evaluations reported 81.3%, showing a 39.2 percentage-point spread.

The study's results are based on simulated dialogues rather than real student interactions, raising concerns about the applicability of findings to actual learning scenarios.

PaperFormalizer successfully automates the formalization of research papers while keeping the author in control of the decision-making process regarding the interpretation of statements.

The pipeline produces an annotated and revised version of the paper, along with Lean code linked to each formalized statement.

The effectiveness of the tool relies heavily on the author's input and understanding of their own statements, which may vary.

The framework improves physical stability of generated structures from 45.3% to 72.4% while achieving a silhouette IoU of 0.970.

Best-of-10 selection improves physical stability from 45.3% to 72.4%.

The abstract does not specify limitations or reproducibility concerns.

Self-Direction improves overall success rates of agents by 8.9 to 17.0 percentage points while reducing repeated mistakes across future encounters.

The method achieves an improvement in success rates by 8.9 to 17.0 percentage points.

The paper does not specify the authors, which may limit reproducibility and verification of results.