SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2407.16245 · ACL · 2024

Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning

Dietrich Klakow, Marius Mosbach, Pin-Jie Lin, Miaoran Zhang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Identifying beneficial tasks to transfer from is a critical step toward successful intermediatetask transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights. 1 1 We release the code publicly at https://github.com/udslsv/intermediate-task-selection. source embedding Task embedding avg 0.621 QNLI CB

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2407.16245")
get_code_for_paper("2407.16245")
have("2407.16245")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2407.16245.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2407.16245.svg)](https://syntology.ai/paper/2407.16245)

Connect an agent — have() is free.