Zihui Xue, Hang Zhao, Zhengqi Gao, Sucheng Ren
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data. Since existing datasets and well-trained models are primarily unimodal, the modality gap between a unimodal network and unlabeled multimodal data poses an interesting problem: how to transfer a pre-trained unimodal network to perform the same task with extra unlabeled multimodal data? In this work, we propose multimodal knowledge expansion (MKE), a knowledge distillation-based framework to effectively utilize multimodal data without requiring labels. Opposite to traditional knowledge distillation, where the student is designed to be lightweight and inferior to the teacher, we observe that a multimodal student model consistently rectifies pseudo labels and generalizes better than its teacher. Extensive experiments on four tasks and different modalities verify this finding. Furthermore, we connect the mechanism of MKE to semi-supervised learning and offer both empirical and theoretical explanations to understand the expansion capability of a multimodal student.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2103.14431")
get_code_for_paper("2103.14431")
have("2103.14431")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2103.14431.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2103.14431)
Connect an agent — have() is free.