GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

Not provided in the abstract

multimodalalignmentoptimal transport

2611.00319

Builder Relevance

70%

2h ago

Abstract

The paper proposes a method to improve the grounding of words to visual regions in vision-language models by introducing a referential alignment mechanism using optimal transport.

Reality Card

Core Claim

The addition of a referential alignment loss to a next-token captioning model improved word-to-picture accuracy from 32.4% to 48.7% under high non-reference rates.

Method / Result

Achieved a word-to-picture accuracy of 48.7% and localization IoU of 0.38 after adding the alignment loss.

Limitations

The model's performance is still weaker compared to a contrastive model trained on the same data, indicating potential limitations in generalizability.

Paper to code

Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.

No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.

← Back to all papers