Yizhi Li, Chenghao Xiao, Hudson Chenghua, Lin Noura, Al Moubayed, Thomas
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models. In this work, we question the length generalizability of CLbased models, i.e., their vulnerability towards length-induced semantic shift. We verify not only that length vulnerability is a significant yet overlooked research gap, but we can devise unsupervised CL methods solely depending on the semantic signal provided by document length. We first derive the theoretical foundations underlying length attacks, showing that elongating a document would intensify the high intra-document similarity that is already brought by CL. Moreover, we found that isotropy promised by CL is highly dependent on the length range of text exposed in training. Inspired by these findings, we introduce a simple yet universal document representation learning framework, LA(SER) 3 : length-agnostic self-reference for semantically robust sentence representation learning, achieving state-of-theart unsupervised performance on the standard information retrieval benchmark. Our code is publicly available.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.16193")
get_code_for_paper("2310.16193")
have("2310.16193")
Connect an agent — have() is free.