We lifted 9 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| shavit-lab/sparse-expansion | canonical | 4 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| find_layers | Ran | shavit-lab/sparse-expansion/utils/modelutils.py code served (permissive licence) · get_code("a9e7f2cdf016b88b") |
| maybe_0th_element | Ran | shavit-lab/sparse-expansion/llama_finetune.py code served (permissive licence) · get_code("1e547e8d369f2be4") |
| quantize | Ran | shavit-lab/sparse-expansion/utils/quant.py code served (permissive licence) · get_code("50ceff9d34d96d60") |
| sparsity_type | Ran | shavit-lab/sparse-expansion/utils/parser.py code served (permissive licence) · get_code("1c09038e6654a6f8") |
| get_llama | Not yet run | shavit-lab/sparse-expansion/utils/modelutils.py code served (permissive licence) · get_code("c6dd03f3342d1724") |
| get_ptb | Not yet run | shavit-lab/sparse-expansion/utils/datautils.py code served (permissive licence) · get_code("a7cbf4fc49c71e03") |
| get_pythia | Not yet run | shavit-lab/sparse-expansion/utils/modelutils.py code served (permissive licence) · get_code("34a49223d77496fe") |
| get_tokenizer | Not yet run | shavit-lab/sparse-expansion/utils/datautils.py code served (permissive licence) · get_code("1e2490fb310d6a27") |
| get_wikitext2 | Not yet run | shavit-lab/sparse-expansion/utils/datautils.py code served (permissive licence) · get_code("380c081883a25dfd") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Disentangling polysemantic neurons is at the core of many current approaches to interpretability of large language models. Here we attempt to study how disentanglement can be used to understand performance, particularly under weight sparsity, a leading post-training optimization technique. We suggest a novel measure for estimating neuronal entanglement: the Wasserstein distance of a neuron's output distribution to a Gaussian. Moreover, we show the existence of a small number of highly entangled "Wasserstein Neurons" in each linear layer of an LLM, characterized by their highly non-Gaussian output distributions, their role in mapping similar inputs to dissimilar outputs, and their significant impact on model accuracy. To study these phenomena, we propose a new experimental framework for disentangling polysemantic neurons. Our framework separates each layer's inputs to create a mixture of experts where each neuron's output is computed by a mixture of neurons of lower Wasserstein distance, each better at maintaining accuracy when sparsified without retraining. We provide strong evidence that this is because the mixture of sparse experts is effectively disentangling the input-output relationship of individual neurons, in particular the difficult Wasserstein neurons.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.15756")
get_code_for_paper("2405.15756")
have("2405.15756")
Connect an agent — have() is free.