An Oct. 9 arXiv preprint finds that, in a controlled caption-only test, CLIP’s pooled text feature retained about one-fifth of a word-count baseline’s above-chance signal for spatial relations such as “on” or “under.” The relation word remained easy to identify at its own token position, but much of that signal faded when the caption was compressed into one feature. The experiment changed captions, not images, so it does not test how a model recognizes a changed scene.
The paper, by Shubh Garg of Thapar Institute of Engineering and Technology and Chen Shani of Tel Aviv University, asks whether a model’s text summary preserves the relation between two objects or mainly retains the words in the sentence. Such summaries are used for downstream text-image similarity and classification.
To test that, the authors built CASP-VL from Visual Genome scene-graph annotations. They changed one concept in a caption, for example “a cup is on a table” to “a cup is beneath a table,” while holding the original image fixed. The benchmark contains 1,491 spatial-relation edits and 2,400 color edits. It is a controlled text-encoder test, not a set of images with altered layouts.
The authors call their EICS metric a way to test what survives in CLIP's text summary. They train a separate classifier to identify relations from summaries of original captions, then test whether it can name the changed relation in an edited caption it has not seen before. The score subtracts chance, 1 in 16 for this test, so CLIP's +0.070 means seven percentage points above chance, not 7% accuracy. Word counts scored +0.372 on the same scale, making CLIP's above-chance signal about one-fifth as large. Across five splits that kept images separate between training and testing, color scores were +0.312 for CLIP and +0.332 for word counts.
The relation word itself was easy to identify from the representation at its own position: the paper reports an EICS score of +0.930 there. When the model compressed the whole caption into one vector from its final token, the score was +0.070; averaging the token vectors scored +0.093. Identifying a relation word at its own position does not show that a model preserves who is related to whom.
This does not contradict CLIP’s 69.3% result on a separate two-choice spatial-relation split of SugarCrepe. That score measures which caption the model prefers; EICS probes what information remains in a text representation. A model can perform above chance on an output benchmark while still losing much of a relation in the feature used downstream.
The same broad pattern appeared in the paper’s tests of larger CLIP, SigLIP and LLaVA models, but it covers only a small set of systems and probes text states. It does not test image-encoder or joint representations, and the captions and interventions come from Visual Genome annotations. The paper links a code-and-data repository, but that link returned a 404 during reporting. The results are an unreviewed arXiv v1, not an independent replication.
Sources
- Full paper, arXiv v1
- Code and data repository linked by the paper; returned 404 during reporting
- Illustrative photo: Ssu, “Cup of coffee on café table,” Wikimedia Commons, licensed under CC BY-SA 4.0.
