SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.18667 · ACL · 2024

FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability

Caiming Xiong, Yihao Feng, Xinyi Yang, Chen Xing, Wenpeng Yin, Congying Xia, Ran Xu, Jiangshu Du

arXiv · PDF · Open in the Atlas

Code that ran

7 of 11 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 5 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.18667.json · badge: /api/ran/2402.18667.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
salesforceairesearch/fofo canonical (classifier label)
classifier basis: census
7 ran · 4 no recorded run
FunctionStatusWhere it lives
accuracy Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/metrics.py
code served (permissive licence) · get_code("e185cfb22182d830")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create sample preference data
# The function expects values of 0 (draw), 1 (base win), 2 (model win)
# It filters to keep only 0 and 1, so we'll create a mix of valid and invalid values
preferences_list = [0, 1, 1, 0, 1, 2, 1, 0, 1, 1]

ARGS = {
    "preferences": preferences_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_metrics_evaluator Ran
ran; our contract draft was wrong, not the code
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/analyze.py
code served (permissive licence) · get_code("fcb02c681ee32e8f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import pandas as pd

class MockAnalyzer:
    """Mock analyzer that implements the methods called by get_metrics_evaluator."""
    
    def agreement_of_annotations(self, annotations_1, n_majority_vote_1=None):
        """Return mock agreement metrics."""
        return {'accuracy': 0.85}
    
    def estimate_variance(self, df):
        """Return mock variance estimate."""
        return 0.12
    
    def estimate_bias(self, df):
        """Return mock bias estimate."""
        return 0.05
    
    def get_length_biases(self, df):
        """Return mock length bias metrics."""
        return {'probability_prefer_longer': 0.65}
    
    def get_list_biases(self, df):
        """Return mock list bias metrics."""
        return {'probability_prefer_list': 0.42}


# Create a mock dataframe with crossannotations
df_crossannotations = pd.DataFrame({
    'price_per_example': [0.10, 0.12, 0.11, 0.09],
    'time_per_example': [2.5, 3.0, 2.8, 2.2],
    'preference': [1.0, 2.0, 1.5, np.nan]
})

analyzer = MockAnalyzer()

ARGS = {
    'analyzer': analyzer,
    'df_crossannotations': df_crossannotations,
    'evaluator_name': 'humans'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_derangement Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/utils.py
code served (permissive licence) · get_code("6be086094b685940")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function is_derangement checks if two arrays are derangements of one another
# by verifying that no element at the same index is equal in both arrays.
# It uses zip() to pair elements and checks inequality with !=
# Both arr1 and arr2 should be iterables (lists, arrays, tensors, etc.)

# Create two simple lists that are derangements of each other
arr1 = [1, 2, 3, 4]
arr2 = [2, 1, 4, 3]

ARGS = {
    "arr1": arr1,
    "arr2": arr2,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
lmsys_parser Ran
ran, honoured the contract we drafted
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/completion_parsers.py
code served (permissive licence) · get_code("025e6fa1d9c72aa3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

ARGS = {
    "completion": "1 7\n some additional text here"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
random_seeded_choice Ran
ran; our contract draft was wrong, not the code
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/utils.py
code served (permissive licence) · get_code("6ebdee3cddbf3cdc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random
from typing import Any, Callable, Optional, Sequence, Union

def random_seeded_choice(seed: Union[int, str, float], choices, **kwargs):
    """Random choice with a (potentially string) seed."""
    return random.Random(seed).choices(choices, k=1, **kwargs)[0]

# Setup code to construct valid arguments
ARGS = {
    "seed": "my_seed_string",
    "choices": ["apple", "banana", "cherry", "date", "elderberry"]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ranking_parser Ran
ran, honoured the contract we drafted
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/completion_parsers.py
code served (permissive licence) · get_code("abb18ef90b2781c4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import ast
import logging
from typing import Any, Optional
import numpy as np

# The function takes one parameter: completion
# It expects either a string representation of a list of dicts,
# or a list of dicts directly.
# Each dict should have 'model' and 'rank' keys.
# The function looks for 'model_1' and checks if its rank is 1 or 2.

# Let's create valid test cases:

# Case 1: Valid completion where model_1 has rank 1
completion_valid_rank1 = "[{'model': 'model_1', 'rank': 1}, {'model': 'model_2', 'rank': 2}]"

# Case 2: Valid completion where model_1 has rank 2
completion_valid_rank2 = "[{'model': 'model_1', 'rank': 2}, {'model': 'model_2', 'rank': 1}]"

# Case 3: Valid completion as list of dicts (not string)
completion_as_list = [{'model': 'model_1', 'rank': 1}, {'model': 'model_2', 'rank': 2}]

# Use the first valid case as the main argument
ARGS = {
    "completion": completion_valid_rank1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_or_return Ran
ran; our contract draft was wrong, not the code
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/utils.py
code served (permissive licence) · get_code("e7d1e05fa7cb0da2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import logging
import os
import pathlib
from pathlib import Path
from typing import Any, Callable, Optional, Sequence, Union
import tempfile

AnyPath = Union[str, os.PathLike, pathlib.Path]

def read_or_return(to_read: Union[AnyPath, str], **kwargs):
    """Read a file or return the input if it is already a string."""
    try:
        with open(Path(to_read), **kwargs) as f:
            out = f.read()
    except FileNotFoundError as e:
        if Path(to_read).is_absolute():
            raise e
        logging.warning(f'Returning input because file not found. Error: {e}')
        out = to_read
    return out

# Create a temporary file for testing
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as tmp:
    tmp.write("test content")
    tmp_path = tmp.name

ARGS = {
    "to_read": tmp_path,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_annotations Not yet run
unverified (harvested, no recorded run)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/analyze.py
code served (permissive licence) · get_code("104b5b1c589a3175")
How it ran

no invocation recorded.

get_crossannotations Not yet run
unverified (harvested, no recorded run)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/analyze.py
code served (permissive licence) · get_code("35e948b2f38e9582")
How it ran

no invocation recorded.

pairwise_to_winrate Not yet run
unverified (harvested, no recorded run)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/metrics.py
code served (permissive licence) · get_code("226c75aa26f6221f")
How it ran

no invocation recorded.

regex_parser Not yet run
unverified (harvested, no recorded run)
salesforceairesearch/fofo/alpaca_eval/src/alpaca_eval/completion_parsers.py
code served (permissive licence) · get_code("b85db06aec0b0efc")
How it ran

no invocation recorded.

Also present in other papers' code

Of 11 samples lifted from this paper's linked repositories, 11 carry a function identity and 0 do not. Of the 11 distinct functions with an identity, 3 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 3 later) and 8 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_annotations 4 other papers
4 later
2406.04692 (2024-06)
Mixture-of-Agents Enhances Large Language Model Capabilities
2410.08964 (2024-10)
Language Imbalance Driven Rewarding for Multilingual Self-improving
2502.01534 (2025-02)
Preference Leakage: A CONTAMINATION PROBLEM IN LLM-AS-A-JUDGE
and 1 more not named here
is_derangement 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
random_seeded_choice 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper presents FOFO, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domainspecific formats, a crucial yet underexamined capability for their application as AI agents. Despite LLMs' advancements, existing benchmarks fail to assess their format-following proficiency adequately. FOFO fills this gap with a diverse range of real-world formats and instructions, developed through an AI-Human collaborative method. Our evaluation across both opensource (e.g., Llama 2, WizardLM) and closedsource (e.g., GPT-4, PALM2, Gemini) LLMs highlights three key findings: open-source models significantly lag behind closed-source ones in format adherence; LLMs' format-following performance is independent of their content generation quality; and LLMs' format proficiency varies across different domains. These insights suggest the need for specialized tuning for format-following skills and highlight FOFO's role in guiding the selection of domainspecific AI agents. FOFO is released here. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.18667")
get_code_for_paper("2402.18667")
have("2402.18667")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.18667.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.18667.svg)](https://syntology.ai/paper/2402.18667)

Connect an agent — have() is free.