SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2103.07668 · AAAI · 2021

Robust Model Compression Using Deep Hypotheses

Omri Armstrong, Ran Gilad-Bachrach

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Machine Learning models should ideally be compact and robust. Compactness provides efficiency and comprehensibility whereas robustness provides resilience. Both topics have been studied in recent years but in isolation. Here we present a robust model compression scheme which is independent of model types: it can compress ensembles, neural networks and other types of models into diverse types of small models. The main building block is the notion of depth derived from robust statistics. Originally, depth was introduced as a measure of the centrality of a point in a sample such that the median is the deepest point. This concept was extended to classification functions which makes it possible to define the depth of a hypothesis and the median hypothesis. Algorithms have been suggested to approximate the median but they have been limited to binary classification. In this study, we present a new algorithm, the Multiclass Empirical Median Optimization (MEMO) algorithm that finds a deep hypothesis in multi-class tasks, and prove its correctness. This leads to our Compact Robust Estimated Median Belief Optimization (CREMBO) algorithm for robust model compression. We demonstrate the success of this algorithm empirically by compressing neural networks and random forests into small decision trees, which are interpretable models, and show that they are more accurate and robust than other comparable methods. In addition, our empirical study shows that our method outperforms Knowledge Distillation on DNN to DNN compression. Large models, such as Deep Neural Networks (DNNs) and ensembles achieve high accuracy on diverse problems (Caruana, Karampatziakis, and Yessenalina 2008; Usmanim 2018). However, their size presents a challenge in many cases because of their resource requirements and their incomprehensibility (Lage et al. 2019). The lack of interpretability of large machine learning models is a limitation especially for applications that require critical decision making such as medical diagnostics (Kononenko 2001;Caruana et al. 2015) and hiring decisions (Hamilton 2018). Small models are efficient in terms of computational cost and memory footprint while also being more interpretable. When small models are required, it has been shown that

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2103.07668")
get_code_for_paper("2103.07668")
have("2103.07668")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2103.07668.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2103.07668.svg)](https://syntology.ai/paper/2103.07668)

Connect an agent — have() is free.