level: technical
pre-trained vision-language models can classify images without training by comparing images to text descriptions. these descriptions usually come from prompts like 'a photo of a [class]'. the similarity score between an image and a prompt depends heavily on the prompt wording. to reduce this sensitivity, researchers often combine scores from multiple prompts using a fixed set of weights. however, current methods use the same weights for all classes, which ignores that some prompts fit certain classes better than others. for example, 'an aerial view of' works well for 'airport' but poorly for 'apple'.
a new approach called class-aware zero-shot prompt reweighting (carprt) solves this by learning different weights for each class. it captures how relevant each prompt is to a specific class. the method works with black-box models, meaning it only needs the model's output scores and not its internal details. carprt optimizes the weights using a small set of labeled examples, then applies them to new images. experiments show it outperforms existing prompt ensembling techniques on several standard benchmarks.
the technique is simple to implement and adds little computational cost. it can be used with any pre-trained vision-language model without retraining. by tailoring prompt weights to each class, carprt makes zero-shot classification more reliable and accurate. this is especially useful when deploying models in new domains where collecting large labeled datasets is impractical. the code is available for others to use and build upon.
why it matters: it makes zero-shot image classification more accurate without extra training, useful for ai systems that must adapt to new tasks quickly.