Noga Zaslavsky, Naftali Tishby
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1503.02406")
get_code_for_paper("1503.02406")
have("1503.02406")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/1503.02406.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/1503.02406)
Connect an agent — have() is free.