SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2104.12763 · ICCV · 2021

MDETR -Modulated Detection for End-to-End Multi-Modal Understanding

Ishan Misra, Yann Lecun, Mannat Singh, Gabriel Synnaeve, Nicolas Carion, Aishwarya Kamath

arXiv · PDF · Open in the Atlas

Code that ran

7 of 11 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (4 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 1 violated it and 4 have no contract verdict; 2 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2104.12763.json · badge: /api/ran/2104.12763.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
b-faye/lightmdetr community (classifier label)
classifier basis not recorded on the link
3 ran · 4 no recorded run
facebookresearch/multimodal community
Papers with Code archive
3 ran · 0 no recorded run
AleDella/mdter_eval community
Papers with Code archive
1 ran · 0 no recorded run
FunctionStatusWhere it lives
MDETR Ran
ran on a synthesized input · constructed an object; no result was computed
facebookresearch/multimodal/torchmultimodal/models/mdetr/model.py
code served (permissive licence) · get_code("609d6e843c9168d9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, MDETR: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn, Tensor
from typing import List, Tuple, Optional

torch.manual_seed(0)

# Create minimal image backbone that returns embeddings and mask
class MinimalImageBackbone(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 4, kernel_size=3, padding=1)
    
    def forward(self, images: Tensor, image_mask: Tensor) -> Tuple[Tensor, Tensor]:
        # images shape: (batch, 3, h, w)
        # Returns: (batch*h*w, 4) embeddings and mask
        b, c, h, w = images.shape
        x = self.conv(images)  # (batch, 4, h, w)
        x = x.permute(0, 2, 3, 1).reshape(b * h * w, 4)  # flatten to (b*h*w, 4)
        return x, image_mask

# Create minimal text encoder
class MinimalTextEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(100, 4)
        self.hidden_size = 4
    
    def forward(self, text: Tensor, attention_mask: Tensor):
        x = self.embedding(text)  # (batch, seq_len, 4)
        # Return object with last_hidden_state attribute
        class TextEncoderOutput:
            def __init__(self, hidden_state):
                self.last_hidden_state = hidden_state
        return TextEncoderOutput(x)

# Create minimal transformer
class MinimalTransformer(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, *args, **kwargs):
        pass

# Create minimal positional embedding
class MinimalPosEmbed(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 4)
    
    def forward(self, mask: Tensor) -> Tensor:
        # mask shape: (batch, h, w)
        b, h, w = mask.shape
        return torch.randn(b, h, w, 4)

# Create minimal text projection
class MinimalTextProjection(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, x: Tensor) -> Tensor:
        return self.linear(x)

# Create minimal image projection
class MinimalImageProjection(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, x: Tensor) -> Tensor:
        return self.linear(x)

# Create minimal bbox and class embeddings
class MinimalBboxEmbed(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, x: Tensor) -> Tensor:
        return self.linear(x)

class MinimalClassEmbed(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 2)
    
    def forward(self, x: Tensor) -> Tensor:
        return self.linear(x)

# Construct all components
image_backbone = MinimalImageBackbone()
text_encoder = MinimalTextEncoder()
transformer = MinimalTransformer()
pos_embed = MinimalPosEmbed()
text_projection = MinimalTextProjection()
image_projection = MinimalImageProjection()
query_embed = nn.Embedding(10, 4)
bbox_embed = MinimalBboxEmbed()
class_embed = MinimalClassEmbed()
extra_query_embeddings = None

ARGS = {
    "image_backbone": image_backbone,
    "text_encoder": text_encoder,
    "transformer": transformer,
    "pos_embed": pos_embed,
    "text_projection": text_projection,
    "image_projection": image_projection,
    "query_embed": query_embed,
    "bbox_embed": bbox_embed,
    "class_embed": class_embed,
    "extra_query_embeddings": extra_query_embeddings,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MDETR — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a MDETR output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
MDETR Ran
ran on a synthesized input · constructed an object; no result was computed
AleDella/mdter_eval/models/mdetr.py
code served (permissive licence) · get_code("237c7b6df0869f30")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, MDETR: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch import nn as torch_nn

torch.manual_seed(0)

# Create a minimal backbone
class MinimalBackbone(nn.Module):
    def __init__(self):
        super().__init__()
        self.num_channels = 4
        self.conv = nn.Conv2d(3, 4, kernel_size=1)
    
    def forward(self, x):
        return self.conv(x)

# Create a minimal text encoder config
class TextEncoderConfig:
    def __init__(self):
        self.hidden_size = 8

# Create a minimal text encoder
class MinimalTextEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.config = TextEncoderConfig()
        self.embedding = nn.Embedding(100, 8)
    
    def forward(self, x):
        return self.embedding(x)

# Create a minimal transformer
class MinimalTransformer(nn.Module):
    def __init__(self):
        super().__init__()
        self.d_model = 16
        self.text_encoder = MinimalTextEncoder()
        self.nhead = 4
        self.encoder_layer = nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True)
        self.decoder_layer = nn.TransformerDecoderLayer(d_model=16, nhead=4, batch_first=True)
        self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=2)
        self.decoder = nn.TransformerDecoder(self.decoder_layer, num_layers=2)
    
    def forward(self, src, mask, query_embed, tgt):
        memory = self.encoder(src, src_key_padding_mask=mask)
        hs = self.decoder(tgt, memory)
        return hs, memory

# Create minimal backbone and transformer
backbone = MinimalBackbone()
transformer = MinimalTransformer()

# Set up ARGS dictionary
ARGS = {
    'backbone': backbone,
    'transformer': transformer,
    'num_classes': 91,
    'num_queries': 100,
    'aux_loss': False,
    'contrastive_hdim': 64,
    'contrastive_loss': False,
    'contrastive_align_loss': False,
    'qa_dataset': None,
    'split_qa_heads': True,
    'predict_final': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MDETR — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a MDETR output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
MDETRModelOutput Ran
ran on a synthesized input · constructed an object; no result was computed
facebookresearch/multimodal/torchmultimodal/models/mdetr/model.py
code served (permissive licence) · get_code("fb58fa6bb659e502")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, MDETRModelOutput: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Define the NamedTuple classes
class MDETRTransformerOutput:
    def __init__(self, decoder_hidden_states, text_memory):
        self.decoder_hidden_states = decoder_hidden_states
        self.text_memory = text_memory

class MDETRModelOutput:
    def __init__(self, transformer_output, pred_logits, pred_boxes, extra_embeddings):
        self.transformer_output = transformer_output
        self.pred_logits = pred_logits
        self.pred_boxes = pred_boxes
        self.extra_embeddings = extra_embeddings

# Create small tensors with distinct dimensions
decoder_hidden_states = torch.randn(2, 4, 8)  # batch=2, seq_len=4, hidden=8
text_memory = torch.randn(2, 5, 8)  # batch=2, text_len=5, hidden=8
pred_logits = torch.randn(2, 3, 10)  # batch=2, num_queries=3, num_classes=10
pred_boxes = torch.randn(2, 3, 4)  # batch=2, num_queries=3, box_coords=4
extra_embeddings = torch.randn(2, 6, 7)  # batch=2, num_extra=6, embed_dim=7

# Construct transformer output
transformer_output = MDETRTransformerOutput(
    decoder_hidden_states=decoder_hidden_states,
    text_memory=text_memory
)

# Construct the arguments dict
ARGS = {
    "transformer_output": transformer_output,
    "pred_logits": pred_logits,
    "pred_boxes": pred_boxes,
    "extra_embeddings": extra_embeddings
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MDETRModelOutput — the run constructed an instance of its own class and did not call it; no shape is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
MDETRTransformerOutput Ran
ran on a synthesized input · constructed an object; no result was computed
facebookresearch/multimodal/torchmultimodal/models/mdetr/model.py
code served (permissive licence) · get_code("5da926b2f714f541")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, MDETRTransformerOutput: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# MDETRTransformerOutput is a NamedTuple with two tensor fields
# We need to provide:
# - decoder_hidden_states: torch.Tensor
# - text_memory: torch.Tensor

# Create small tensors with distinct dimensions for probing
decoder_hidden_states = torch.randn(2, 4, 3)  # batch=2, seq_len=4, hidden_dim=3
text_memory = torch.randn(2, 5, 3)  # batch=2, text_seq_len=5, hidden_dim=3

ARGS = {
    "decoder_hidden_states": decoder_hidden_states,
    "text_memory": text_memory,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MDETRTransformerOutput — the run constructed an instance of its own class and did not call it; no shape is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
build_position_encoding Ran
ran; our contract draft was wrong, not the code
b-faye/lightmdetr/models/position_encoding.py
code served (permissive licence) · get_code("cea3f92defccfb9c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PositionEmbeddingSine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import math

torch.manual_seed(0)

class TensorList:
    """Mock tensor list object used by the position embedding functions."""
    def __init__(self, tensors, mask):
        self.tensors = tensors
        self.mask = mask

class Args:
    """Mock arguments object for build_position_encoding."""
    def __init__(self, hidden_dim=128, position_embedding='sine'):
        self.hidden_dim = hidden_dim
        self.position_embedding = position_embedding

# Create a small batch of tensors
batch_size = 2
height = 4
width = 3
channels = 16

# Create input tensor
tensors = torch.randn(batch_size, channels, height, width)

# Create mask (False means valid, True means padding)
mask = torch.zeros(batch_size, height, width, dtype=torch.bool)

# Create tensor_list object
tensor_list = TensorList(tensors=tensors, mask=mask)

# Create args for sine position embedding
args_sine = Args(hidden_dim=128, position_embedding='sine')

# Create args for learned position embedding
args_learned = Args(hidden_dim=128, position_embedding='learned')

ARGS = {
    'args': args_sine
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type PositionEmbeddingSine — a PositionEmbeddingSine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PositionEmbeddingSine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
dice_loss Ran
ran, violated the contract we drafted
b-faye/lightmdetr/models/segmentation.py
code served (permissive licence) · get_code("ac8fe530cdad4d8c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create sample inputs and targets tensors
# The function expects tensors that can be flattened and have sigmoid applied
# Shape: batch_size=2, spatial_dims flattened to create predictions
batch_size = 2
num_spatial = 4

# inputs: predictions before sigmoid, shape (batch_size, num_spatial)
inputs = torch.randn(batch_size, num_spatial)

# targets: binary labels matching inputs shape, values 0 or 1
targets = torch.randint(0, 2, (batch_size, num_spatial), dtype=torch.float32)

# num_boxes: number of boxes for normalization
num_boxes = 2

ARGS = {
    "inputs": inputs,
    "targets": targets,
    "num_boxes": num_boxes,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
sigmoid_focal_loss Ran
ran; our contract draft was wrong, not the code
b-faye/lightmdetr/models/segmentation.py
code served (permissive licence) · get_code("5c0711aada67957e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create small tensors for inputs and targets
# inputs: predictions (logits) - shape (2, 4) for batch_size=2, num_predictions=4
inputs = torch.randn(2, 4, requires_grad=True)

# targets: binary labels - shape (2, 4) matching inputs
targets = torch.randint(0, 2, (2, 4), dtype=torch.float32)

# num_boxes: number of boxes for normalization
num_boxes = 2

# alpha and gamma are already float parameters with defaults
alpha = 0.25
gamma = 2.0

ARGS = {
    "inputs": inputs,
    "targets": targets,
    "num_boxes": num_boxes,
    "alpha": alpha,
    "gamma": gamma,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
build_transformer Not yet run
unverified (harvested, no recorded run)
b-faye/lightmdetr/models/transformer.py
code served (permissive licence) · get_code("09e538653a528ac8")
How it ran

no invocation recorded.

build_transformer Not yet run
unverified (harvested, no recorded run)
b-faye/lightmdetr/models/transformer_plus.py
code served (permissive licence) · get_code("751cd5726698116b")
How it ran

no invocation recorded.

get_init_file Not yet run
unverified (harvested, no recorded run)
b-faye/lightmdetr/run_with_submitit.py
code served (permissive licence) · get_code("d71abefc7fbfd28f")
How it ran

no invocation recorded.

get_shared_folder Not yet run
unverified (harvested, no recorded run)
b-faye/lightmdetr/run_with_submitit.py
code served (permissive licence) · get_code("84691ff17ae1504d")
How it ran

no invocation recorded.

Also present in other papers' code

Of 11 samples lifted from this paper's linked repositories, 11 carry a function identity and 0 do not. Of the 10 distinct functions with an identity, 6 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 0 the same month, 6 later) and 4 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
sigmoid_focal_loss 56 other papers
4 dated earlier, 52 later · 1 only as an assembly brick
2005.12872 (2020-05)
End-to-End Object Detection with Transformers
2010.04159 (2020-10)
Deformable DETR: Deformable Transformers for End-to-End Object Detection
2103.11731 (2021-03)
Meta-DETR: Image-Level Few-Shot Object Detection with Inter-Class Correlation Exploitation
and 53 more not named here
dice_loss 40 other papers
3 dated earlier, 37 later · 1 only as an assembly brick
2005.12872 (2020-05)
End-to-End Object Detection with Transformers
2010.04159 (2020-10)
Deformable DETR: Deformable Transformers for End-to-End Object Detection
2103.12115 (2021-03)
End-to-End Trainable Multi-Instance Pose Estimation with Transformers
and 37 more not named here
build_position_encoding 8 other papers
8 later
2108.11084 (2021-08)
Transformer for Single Image Super-Resolution
2111.12085 (2021-11)
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
2207.13325 (2022-07)
SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding
and 5 more not named here
get_init_file 4 other papers
4 later
2112.08654 (2021-12)
Learning to Prompt for Continual Learning
2210.05668 (2022-10)
Understanding Embodied Reference with Touch-Line Transformer
2404.14721 (2024-04)
Dynamically Anchored Prompting for Task-Imbalanced Continual Learning
and 1 more not named here
build_transformer 2 other papers
2 later
2209.01814 (2022-09)
RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection
2505.11726 (2025-05)
Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures
get_shared_folder 2 other papers
2 later
2210.05668 (2022-10)
Understanding Embodied Reference with Touch-Line Transformer
2505.11726 (2025-05)
Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2104.12763")
get_code_for_paper("2104.12763")
have("2104.12763")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2104.12763.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2104.12763.svg)](https://syntology.ai/paper/2104.12763)

Connect an agent — have() is free.