Sergio Escalera, Jun Wan, Yanyan Liang, Zhen Lei, Zhigang Chen, Benjia Zhou, Du Zhang, Albert Clapés
We lifted 7 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| zhoubenjia/gfslt-vlp | canonical | 6 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| cosine_scheduler | Ran | zhoubenjia/gfslt-vlp/utils.py code served (permissive licence) · get_code("04805aa3e54efe47") |
| make_head | Ran | zhoubenjia/gfslt-vlp/models.py code served (permissive licence) · get_code("dd6acfc78ad9755a") |
| sampler_func | Ran | zhoubenjia/gfslt-vlp/utils.py code served (permissive licence) · get_code("3ecbe9910f3ee824") |
| tokenize_13a | Ran | zhoubenjia/gfslt-vlp/metrics.py code served (permissive licence) · get_code("b5ec86d615fa1553") |
| tokenize_v14_international | Ran | zhoubenjia/gfslt-vlp/metrics.py code served (permissive licence) · get_code("7863c21b724029ba") |
| tokenize_zh | Ran | zhoubenjia/gfslt-vlp/metrics.py code served (permissive licence) · get_code("803bc06bd832fbfe") |
| count_parameters_in_MB | Not yet run | zhoubenjia/gfslt-vlp/utils.py code served (permissive licence) · get_code("0fee20319bd8425a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a twostage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pretraining (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (≥+5) and the CSL-Daily dataset (≥+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2307.14768")
get_code_for_paper("2307.14768")
have("2307.14768")
Connect an agent — have() is free.