Huy Vu, H Schwartz, Matthew Matero, Adithya Ganesan, Aravind Ravula
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effectively leverage transformers. Here, we provide a systematic study on the role of dimension reduction methods (principal components analysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a limited amount of data pose a significant difficulty which can be overcome with a pre-trained dimension reduction regime. RoBERTa consistently achieves top performance in humanlevel tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results comparable to the best performance with just 1 12 of the embedding dimensions.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2105.03484")
get_code_for_paper("2105.03484")
have("2105.03484")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2105.03484.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2105.03484)
Connect an agent — have() is free.