Kuan-Chieh Wang, James Zou, Serena Yeung, Jeff Haochen, Yuhui Zhang, Shih-Cheng Huang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by leveraging the rich information in large-scale image-caption datasets. Our work highlights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language. The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation. Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behaviors, without requiring any visual data. Through a combination of theoretical explanation and empirical verification, we present conditions under which classifiers trained on embeddings from one modality can be equivalently applied to embeddings from another modality. On a range of image datasets with known error slices, we demonstrate that our method can effectively identify the error slices and influential attributes, and can further use language to rectify failure modes of the classifier. Text Embedding Image Embedding Classifier Waterbird or Landbird? Discover Error Slices Identify Influential Attributes Rectify Model Misbehaviors Training Western gull in lake.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2302.04269")
get_code_for_paper("2302.04269")
have("2302.04269")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2302.04269.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2302.04269)
Connect an agent — have() is free.