John Wu, Yonatan Belinkov, Fahim Dalvi, Nadir Durrani, Hassan Sajjad, James Glass
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks. 1 * Equal contribution 1 The code is available at https://github.com/ johnmwu/contextual-corr-analysis.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2005.01172")
get_code_for_paper("2005.01172")
have("2005.01172")
Connect an agent — have() is free.