Ian Covert, James Zou, Tatsunori Hashimoto, Wenlong Ji
We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| iancovert/data-scaling | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_shapley_value | Ran | iancovert/data-scaling/data_scaling/valuation.py code served (permissive licence) · get_code("855e6fe53ca5d583") |
| generate_metrics | Ran | iancovert/data-scaling/data_scaling/utils.py code served (permissive licence) · get_code("a267ca43deea84da") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent works have shown that machine learning models improve at a predictable rate with the total amount of training data, leading to scaling laws that describe the relationship between error and dataset size. These scaling laws can help design a model's training dataset, but they typically take an aggregate view of the data by only considering the dataset's size. We introduce a new perspective by investigating scaling behavior for the value of individual data points: we find that a data point's contribution to model's performance shrinks predictably with the size of the dataset in a log-linear manner. Interestingly, there is significant variability in the scaling exponent among different data points, indicating that certain points are more valuable in small datasets while others are relatively more useful as a part of large datasets. We provide learning theory to support our scaling law, and we observe empirically that it holds across diverse model classes. We further propose a maximum likelihood estimator and an amortized estimator to efficiently learn the individualized scaling behaviors from a small number of noisy observations per data point. Using our estimators, we provide insights into factors that influence the scaling behavior of different data points. Finally, we demonstrate applications of the individualized scaling laws to data valuation and data subset selection. Overall, our work represents a first step towards understanding and utilizing scaling properties for the value of individual data points.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.20456")
get_code_for_paper("2405.20456")
have("2405.20456")
Connect an agent — have() is free.