Hanwang Zhang, Xiaojuan Qi, Zhuotao Tian, Jiequan Cui, Zhisheng Zhong, Bei Yu
We lifted 11 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| jiequancui/dkl | canonical | 3 of 3 |
| jiequancui/LBGAT | canonical | 0 of 4 |
| FPNAS/LBGAT | pwc_unofficial | 2 of 2 |
| jiequancui/DKL | — | 1 of 1 |
| dvlab-research/LBGAT | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| IKL_KD | Ran | jiequancui/DKL/DKLv1/KD-dkl/mdistiller/distillers/IKL_KD.py pointer only (licence: NONE) · get_code("b32e10adc9bf6201") |
| cross_entropy | Ran | jiequancui/dkl/DKLv1/Adv-training-dkl/train_dkl_cifar10.py pointer only (licence: NONE) · get_code("59f6b4ce1de254e2") |
| dkl_loss | Ran | jiequancui/dkl/DKLv1/Adv-training-dkl/train_dkl_cifar10.py pointer only (licence: NONE) · get_code("eb1d0d5bb0f0c627") |
| dkl_loss | Ran | jiequancui/dkl/DKLv1/KD-dkl/mdistiller/distillers/IKL_KD.py pointer only (licence: NONE) · get_code("6ae3a3d2a8a1acc4") |
| l2_norm | Ran | FPNAS/LBGAT/lbgat.py code served (permissive licence) · get_code("bbffec6aadbbfd95") |
| lbgat_loss | Ran | dvlab-research/LBGAT/lbgat.py code served (permissive licence) · get_code("342a07cb13a5e0bc") |
| squared_l2_norm | Ran | FPNAS/LBGAT/lbgat.py code served (permissive licence) · get_code("724c88e3f81d2123") |
| ResNet152_cifar | Not yet run | jiequancui/LBGAT/models/resnetcifar.py code served (permissive licence) · get_code("87fe041ab6db516b") |
| ResNet18_cifar | Not yet run | jiequancui/LBGAT/models/resnetcifar.py code served (permissive licence) · get_code("60d3d2bb8b6c30e2") |
| ResNet34_cifar | Not yet run | jiequancui/LBGAT/models/resnetcifar.py code served (permissive licence) · get_code("4386c6d158344286") |
| lbgat_loss | Not yet run | jiequancui/LBGAT/lbgat.py code served (permissive licence) · get_code("2533491b22247ed9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels. Thanks to the decomposed formulation of DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of KL/DKL in scenarios like knowledge distillation by breaking its asymmetric optimization property. This modification ensures that the wMSE component is always effective during training, providing extra constructive cues. Secondly, we introduce class-wise global information into KL/DKL to mitigate bias from individual samples. With these two enhancements, we derive the Improved Kullback-Leibler (IKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training, and knowledge distillation tasks. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard -RobustBench and competitive performance on knowledge distillation, demonstrating the substantial practical merits. Our code is available at https://github.com/jiequancui/DKL.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.13948")
get_code_for_paper("2305.13948")
have("2305.13948")
Connect an agent — have() is free.