We lifted 8 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| eboix/theory-of-model-distillation | canonical | 8 of 8 |
| Function | Status | Where it lives |
|---|---|---|
| batch_linear_probe_adam | Ran | eboix/theory-of-model-distillation/probing_utils.py pointer only (licence: NONE) · get_code("b62beacb52e18332") |
| batch_logistic_probe_adam | Ran | eboix/theory-of-model-distillation/probing_utils.py pointer only (licence: NONE) · get_code("20fe419b3cd8dfbf") |
| comp_and | Ran | eboix/theory-of-model-distillation/decision_tree_utils.py pointer only (licence: NONE) · get_code("01704e6e4419e521") |
| comp_and_list | Ran | eboix/theory-of-model-distillation/decision_tree_utils.py pointer only (licence: NONE) · get_code("55004cdca28a206f") |
| get_random_data_unif_binary | Ran | eboix/theory-of-model-distillation/train_utils.py pointer only (licence: NONE) · get_code("fcae99f48ee56066") |
| literals_to_tup | Ran | eboix/theory-of-model-distillation/decision_tree_utils.py pointer only (licence: NONE) · get_code("39d9c79e201d3259") |
| test_class | Ran | eboix/theory-of-model-distillation/train_utils.py pointer only (licence: NONE) · get_code("59983924736c5a8a") |
| validation_split | Ran | eboix/theory-of-model-distillation/train_utils.py pointer only (licence: NONE) · get_code("51b5eb19649eb81d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications, basic questions about the extent to which models can be distilled, and the runtime and amount of data needed to distill, remain largely open. To study these questions, we initiate a general theory of distillation, defining PAC-distillation in an analogous way to PAC-learning [Val84]. As applications of this theory: (1) we propose new algorithms to extract the knowledge stored in the trained weights of neural networks -- we show how to efficiently distill neural networks into succinct, explicit decision tree representations when possible by using the ``linear representation hypothesis''; and (2) we prove that distillation can be much cheaper than learning from scratch, and make progress on characterizing its complexity.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.09053")
get_code_for_paper("2403.09053")
have("2403.09053")
Connect an agent — have() is free.