Karsten Roth, Oriol Vinyals, Zeynep Akata, Olivier Hénaff, Lukas Thede, A Koepke
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation -a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock training guidance, auxiliary gains and knowledge fusion from any model repository without restrictions on model & problem specifics -including from weaker, lower-performance models. This work provides a first, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a general extension via data partitioning for successful transfer between nearly all pretrained modelswhich can also be done unsupervised. Finally, we assess both the scalability and impact of model properties on successful model-agnostic knowledge transfer.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.17653")
get_code_for_paper("2310.17653")
have("2310.17653")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2310.17653.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2310.17653)
Connect an agent — have() is free.