Daniel Castro, Stephanie Hyland, Javier Alvarez-Valle, Aditya Nori, Benedikt Boecking, Shruthi Bannur, Ozan Oktay, Anton Schwaighofer, Hoifung Poon, Tristan Naumann, Naoto Usuyama, Maria Wetscherek
We lifted 4 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| microsoft/hi-ml | — | 3 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| BertProjectionHead | Ran | microsoft/hi-ml/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py code served (permissive licence) · get_code("639ba99d63e88bd1") |
| CXRBertConfig | Ran | microsoft/hi-ml/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py code served (permissive licence) · get_code("aa375ca39177b49a") |
| CXRBertOutput | Ran | microsoft/hi-ml/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py code served (permissive licence) · get_code("a00002413c3202bf") |
| CXRBertModel | Not yet run | microsoft/hi-ml/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py code served (permissive licence) · get_code("a219e03e037cfc36") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision-language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision-language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision-language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision-language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2204.09817")
get_code_for_paper("2204.09817")
have("2204.09817")
Connect an agent — have() is free.