Ivan Srba, Maria Bielikova, Branislav Pecher ♠ † ‡
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
While learning with limited labelled data can effectively deal with a lack of labels, it is also sensitive to the effects of uncontrolled randomness introduced by so-called randomness factors (i.e., non-deterministic decisions such as choice or order of samples). We propose and formalise a method to systematically investigate the effects of individual randomness factors while taking the interactions (dependence) between them into consideration. To this end, our method mitigates the effects of other factors while observing how the performance varies across multiple runs. Applying our method to multiple randomness factors across in-context learning and fine-tuning approaches on 7 representative text classification tasks and metalearning on 3 tasks, we show that: 1) disregarding interactions between randomness factors in existing works led to inconsistent findings due to incorrect attribution of the effects of randomness factors, such as disproving the consistent sensitivity of in-context learning to sample order even with random sample selection; and 2) besides mutual interactions, the effects of randomness factors, especially sample order, are also dependent on more systematic choices unexplored in existing works, such as number of classes, samples per class or choice of prompt format.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.12817")
get_code_for_paper("2402.12817")
have("2402.12817")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2402.12817.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2402.12817)
Connect an agent — have() is free.