Researchers have developed an inference-time defense for CLIP, the model family that connects images with text, to find and repair image regions that may carry a hidden backdoor trigger. Their method, CLIPGuard, tests which parts of an image most alter CLIP’s visual representation, then evaluates repaired versions before classification.
A backdoor can be planted by adding a small trigger to some training images so that a model sends later images containing the same pattern to an attacker-chosen class. CLIPGuard is designed for a black-box setting: the defense does not need access to CLIP’s weights, gradients, logits or training data.
The system masks cells in an image grid and measures how each mask shifts CLIP’s image embedding, its numerical representation of the picture. It uses Segment Anything to isolate sensitive regions, fills in each candidate region (inpainting it), and compares the repaired candidates against text descriptions of classes. CLIPGuard passes the selected repaired candidate to CLIP.
In tests with a poisoned CLIP ViT-B/16 on STL-10 and Imagenette, a 10-class subset of ImageNet, CLIPGuard reduced average attack success across four conventional attacks from 98.41% to 3.22% on STL-10 and from 98.07% to 4.18% on Imagenette. Clean accuracy also fell: from 90.04% to 83.95% and from 58.26% to 54.69%, respectively. Attack success rate is the share of triggered test images classified into the attacker’s target class.
The paper reports a speed–protection tradeoff on Imagenette: a 4×4 masking grid took 0.54 seconds per image and left 10.87% attack success, while an 8×8 grid took 0.95 seconds and left 3.14%. Those timings came from the authors’ single RTX 4090 test system. The evaluation covers one CLIP encoder and benchmark datasets; it does not establish performance on deployed systems or independently reproduced results. The authors have released the implementation.