Yu Wu, Fangyun Wei, Yutong Chen, Ronglai Zuo, Shujie Liu, Brian Mak
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden representations. RGB videos, however, are raw signals with substantial visual redundancy, leading the encoder to overlook the key information for sign language understanding. To mitigate this problem and better incorporate domain knowledge, such as handshape and body movement, we introduce a dual visual encoder containing two separate streams to model both the raw videos and the keypoint sequences generated by an off-the-shelf keypoint estimator. To make the two streams interact with each other, we explore a variety of techniques, including bidirectional lateral connection, sign pyramid network with auxiliary supervision, and frame-level selfdistillation. The resulting model is called TwoStream-SLR, which is competent for sign language recognition (SLR). TwoStream-SLR is extended to a sign language translation (SLT) model, TwoStream-SLT, by simply attaching an extra translation network. Experimentally, our TwoStream-SLR and TwoStream-SLT achieve stateof-the-art performance on SLR and SLT tasks across a series of datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily. Code and models are available at: https://github.com/FangyunWei/SLRT. * Equal contribution. † Corresponding author. 1 Glosses are the word-for-word transcription of sign language where each gloss is a unique label for a sign. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2211.01367")
get_code_for_paper("2211.01367")
have("2211.01367")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2211.01367.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2211.01367)
Connect an agent — have() is free.