H Mcmahan, Peter Kairouz, Christopher Choquette-Choo, Zheng Xu, Galen Andrew, Yanxiang Zhang, Jesse Rosenstock, Yuanbo Zhang Google
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We train language models (LMs) with federated learning (FL) and differential privacy (DP) in the Google Keyboard (Gboard). We apply the DP-Follow-the-Regularized-Leader (DP-FTRL) (Kairouz et al., 2021b) algorithm to achieve meaningfully formal DP guarantees without requiring uniform sampling of client devices. To provide favorable privacy-utility trade-offs, we introduce a new client participation criterion and discuss the implication of its configuration in large scale systems. We show how quantile-based clip estimation (Andrew et al., 2021) can be combined with DP-FTRL to adaptively choose the clip norm during training or reduce the hyperparameter tuning in preparation for training. With the help of pretraining on public data, we train and deploy more than twenty Gboard LMs that achieve high utility and ρ-zCDP privacy guarantees with ρ ∈ (0.2, 2), with two models additionally trained with secure aggregation (Bonawitz et al., 2017). We are happy to announce that all the next word prediction neural network LMs in Gboard now have DP guarantees, and all future launches of Gboard neural network LMs will require DP guarantees. We summarize our experience and provide concrete suggestions on DP training for practitioners.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.18465")
get_code_for_paper("2305.18465")
have("2305.18465")
Connect an agent — have() is free.