We lifted 4 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| artemyk/ibcurve | canonical | 1 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| one_hot | Ran | artemyk/ibcurve/rundata.py code served (permissive licence) · get_code("e222c7d8f6af3765") |
| GMM_entropy | Not yet run | artemyk/ibcurve/entropy.py code served (permissive licence) · get_code("7871484af5235933") |
| GMM_negative_LLH | Not yet run | artemyk/ibcurve/entropy.py code served (permissive licence) · get_code("a83047e65c528b73") |
| Gaussian_entropy | Not yet run | artemyk/ibcurve/entropy.py code served (permissive licence) · get_code("7bc934ace5894a34") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Information bottleneck (IB) is a method for extracting information from one random variable $X$ that is relevant for predicting another random variable $Y$. To do so, IB identifies an intermediate "bottleneck" variable $T$ that has low mutual information $I(X;T)$ and high mutual information $I(Y;T)$. The "IB curve" characterizes the set of bottleneck variables that achieve maximal $I(Y;T)$ for a given $I(X;T)$, and is typically explored by maximizing the "IB Lagrangian", $I(Y;T) - βI(X;T)$. In some cases, $Y$ is a deterministic function of $X$, including many classification problems in supervised learning where the output class $Y$ is a deterministic function of the input $X$. We demonstrate three caveats when using IB in any situation where $Y$ is a deterministic function of $X$: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of $β$; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when $Y$ is a small perturbation away from being a deterministic function of $X$, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1808.07593")
get_code_for_paper("1808.07593")
have("1808.07593")
Connect an agent — have() is free.