Yisen Wang, Stefanie Jegelka, Hu, Sharut Gupta, Yifei Wang, Kaiwen Mit
We lifted 7 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| kaotty/Understanding-ESSL | canonical | 2 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| get_lr | Ran | kaotty/Understanding-ESSL/ESSL/equivariant_tasks.py pointer only (licence: NONE) · get_code("190117b425ab9fe1") |
| nt_xent | Ran | kaotty/Understanding-ESSL/ESSL/equivariant_tasks.py pointer only (licence: NONE) · get_code("0931a280f76e561f") |
| extract_lr | Not yet run | kaotty/Understanding-ESSL/ESSL/utils.py pointer only (licence: NONE) · get_code("bde665611cfb0d27") |
| get_attention_mask | Not yet run | kaotty/Understanding-ESSL/ESSL/models.py pointer only (licence: NONE) · get_code("94d237ce37982cf4") |
| get_color_distortion | Not yet run | kaotty/Understanding-ESSL/ESSL/equivariant_tasks.py pointer only (licence: NONE) · get_code("fea67a5756eb68e5") |
| get_negative_mask | Not yet run | kaotty/Understanding-ESSL/ESSL/utils.py pointer only (licence: NONE) · get_code("b09a2cc2e21f2012") |
| get_optimizer | Not yet run | kaotty/Understanding-ESSL/ESSL/utils.py pointer only (licence: NONE) · get_code("825885e5fbd21be1") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (e.g., colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at https://github.com/kaotty/Understanding-ESSL.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.06508")
get_code_for_paper("2411.06508")
have("2411.06508")
Connect an agent — have() is free.