We lifted 9 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| pytorch/opacus | canonical | 0 of 2 |
| facebookresearch/pytorch-dp | pwc_unofficial | 4 of 4 |
| woodyx218/private_vision | extension | 0 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| dtype_safe | Ran | facebookresearch/pytorch-dp/opacus/data_loader.py code served (permissive licence) · get_code("0b1c417eba15b30c") |
| get_layer_set | Ran | facebookresearch/pytorch-dp/benchmarks/utils.py code served (permissive licence) · get_code("38d3968927d62f28") |
| reset_peak_memory_stats | Ran | facebookresearch/pytorch-dp/benchmarks/utils.py code served (permissive licence) · get_code("e49fd334c05bf4ec") |
| shape_safe | Ran | facebookresearch/pytorch-dp/opacus/data_loader.py code served (permissive licence) · get_code("556ad0ee1120c05e") |
| apply_permutation | Not yet run | pytorch/opacus/opacus/layers/dp_rnn.py code served (permissive licence) · get_code("46783ac8b5216877") |
| filter_out_old_keys | Not yet run | pytorch/opacus/opacus/layers/param_rename.py code served (permissive licence) · get_code("a20422f8abbda65c") |
| get_sigma_from_gdp | Not yet run | woodyx218/private_vision/private_vision/privacy_engine.py code served (permissive licence) · get_code("fc285748768188e6") |
| get_sigma_from_rdp | Not yet run | woodyx218/private_vision/private_vision/privacy_engine.py code served (permissive licence) · get_code("4cb8d0becf117730") |
| get_sigma_from_rdp_cks | Not yet run | woodyx218/private_vision/private_vision/privacy_engine.py code served (permissive licence) · get_code("5d0f89a9f572d13f") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead. We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization; and (3) fine-tuning objectives which are aligned with the pretraining procedure. With the above, we obtain NLP models that outperform state-of-the-art DP-trained models under the same privacy budget and strong non-private baselines -- by directly fine-tuning pretrained models with DP optimization on moderately-sized corpora. To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model. The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead. Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models doesn't tend to suffer from dimension-dependent performance degradation. Code to reproduce results can be found at https://github.com/lxuechen/private-transformers.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2110.05679")
get_code_for_paper("2110.05679")
have("2110.05679")
Connect an agent — have() is free.