SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2408.13467 · 2024

LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs

arXiv · PDF · Open in the Atlas

Code that ran

10 of 14 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 1 violated it and 2 have no contract verdict; 6 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2408.13467.json · badge: /api/ran/2408.13467.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
deep-diver/llamaduo canonical (classifier label)
classifier basis: census
10 ran · 4 no recorded run
FunctionStatusWhere it lives
call_service_llm Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/src/gen/utils.py
code served (permissive licence) · get_code("46b3bd49851e747b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import asyncio
import json


# Create a mock async client that simulates the LLM service
class MockLLMClient:
    async def generate_text(self, model, prompt, **gen_configs):
        # Return a response containing valid JSON with required keys
        response = '''
        Here is the analysis result:
        {
            "score": 8.5,
            "feedback": "Good response",
            "nested": {
                "detail": "Some detail"
            }
        }
        '''
        return response


# Set up the arguments
ARGS = {
    "client": MockLLMClient(),
    "eval_model": "gemini-pro",
    "prompt": "Please evaluate this response and provide feedback in JSON format with score, feedback, and nested.detail fields.",
    "gen_configs": {
        "temperature": 0.7,
        "max_tokens": 1024,
        "top_p": 0.95
    },
    "keys_to_check": [
        "score",
        "feedback",
        "nested.detail"
    ],
    "retry_num": 3,
    "job_num": 42
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
cleanup_ds Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/dataset_merge.py
code served (permissive licence) · get_code("5b051bbcaa7714d9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Dataset. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from datasets import Dataset

# Create a minimal Hugging Face Dataset for testing
# The function expects a Dataset with column_names attribute and methods:
# - remove_columns(list)
# - add_column(name, data)
# - __len__()

ds = Dataset.from_dict({
    'generator': ['gpt-4', 'gpt-3.5'],
    'prompt_id': ['p1', 'p2'],
    'extra_col_1': ['extra1', 'extra2'],
    'extra_col_2': ['data1', 'data2']
})

ARGS = {
    'ds': ds
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
datasets — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Dataset — a Dataset output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Dataset output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
collage_as_dataset Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/src/pipeline/synth_data_gen.py
code served (permissive licence) · get_code("0ab58512e22ed49f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DatasetDict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os
from datasets import Dataset, DatasetDict

# Create temporary JSON files with the expected structure
temp_dir = tempfile.mkdtemp()

# Create first JSON file
json_file_1 = os.path.join(temp_dir, "data_1.json")
data_1 = {
    "seed_prompt": "Tell me about AI",
    "contents": [
        {
            "instruction": "What is artificial intelligence?",
            "response": "AI is the simulation of human intelligence."
        },
        {
            "instruction": "How does machine learning work?",
            "response": "ML uses algorithms to learn from data."
        }
    ]
}
with open(json_file_1, 'w') as f:
    json.dump(data_1, f)

# Create second JSON file
json_file_2 = os.path.join(temp_dir, "data_2.json")
data_2 = {
    "seed_prompt": "Explain neural networks",
    "contents": [
        {
            "instruction": "What are neural networks?",
            "response": "Neural networks are computational models inspired by brains."
        }
    ]
}
with open(json_file_2, 'w') as f:
    json.dump(data_2, f)

ARGS = {
    "filenames": [json_file_1, json_file_2],
    "service_model_name": "gemini-pro",
    "topic": "artificial-intelligence",
    "split": "train"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
datasets — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DatasetDict — a DatasetDict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DatasetDict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
craft_prompt Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/src/gen/async_generate_data_gemini.py
code served (permissive licence) · get_code("966485e1f476ef37")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "instruction": "Write a Python function that finds the maximum element in a list.",
    "response": "To find the maximum element in a list, you can use the built-in max() function in Python. Here's an example:\n\n```python\ndef find_max(lst):\n    return max(lst)\n\nnumbers = [3, 7, 2, 9, 1]\nprint(find_max(numbers))  # Output: 9\n```\n\nAlternatively, you can iterate through the list manually:\n\n```python\ndef find_max(lst):\n    if not lst:\n        return None\n    max_val = lst[0]\n    for num in lst[1:]:\n        if num > max_val:\n            max_val = num\n    return max_val\n```\n\nThe first approach is more efficient and Pythonic."
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
find_json_snippet Ran
ran, honoured the contract we drafted
deep-diver/llamaduo/src/gen/collate_as_datasets.py
code served (permissive licence) · get_code("67e2dea755355a23")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json

def find_json_snippet(raw_snippet):
    """
    find_json_snippet tries to find JSON snippets in a given raw_snippet string
    """
    json_parsed_string = None
    json_start_index = raw_snippet.find('[')
    json_end_index = raw_snippet.rfind(']')
    if json_start_index >= 0 and json_end_index >= 0:
        json_snippet = raw_snippet[json_start_index:json_end_index + 1]
        try:
            json_parsed_string = json.loads(json_snippet, strict=False)
        except:
            raise ValueError('......failed to parse string into JSON format')
    else:
        raise ValueError('......No JSON code snippet found in string.')
    return json_parsed_string


# Setup code to construct valid arguments
raw_snippet_with_json = "Some prefix text [1, 2, 3, 4] some suffix text"

ARGS = {
    "raw_snippet": raw_snippet_with_json
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_response Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/src/gen/collate_as_datasets.py
code served (permissive licence) · get_code("5b706a41d8a78def")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import List, Dict

# Construct valid arguments for format_response function

# The function expects a List[Dict[str, str]] where each dict has a specific structure
# based on the code: response['candidates'][0]['content']['parts'][0]['text']

# Create sample responses that contain valid JSON
response_1 = {
    'candidates': [
        {
            'content': {
                'parts': [
                    {
                        'text': '''[
                            {
                                "instruction": "What is 2+2?",
                                "response": "The answer is 4"
                            },
                            {
                                "instruction": "What is the capital of France?",
                                "response": "The capital of France is Paris"
                            }
                        ]'''
                    }
                ]
            }
        }
    ]
}

response_2 = {
    'candidates': [
        {
            'content': {
                'parts': [
                    {
                        'text': '''Some prefix text [
                            {
                                "instruction": "How do you say hello in Spanish?",
                                "response": "You say 'Hola'"
                            }
                        ] some suffix text'''
                    }
                ]
            }
        }
    ]
}

response_3 = {
    'candidates': [
        {
            'content': {
                'parts': [
                    {
                        'text': '''[
                            {
                                "instruction": "What is Python?",
                                "response": "Python is a programming language"
                            }
                        ]'''
                    }
                ]
            }
        }
    ]
}

responses = [response_1, response_2, response_3]

ARGS = {
    'responses': responses
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_async Ran
ran on a synthesized input
deep-diver/llamaduo/src/gen/gemini.py
code served (permissive licence) · get_code("1067596b265450c1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from unittest.mock import AsyncMock, MagicMock


class MockGenerativeModel:
    """Stand-in for a generative model with async generate_content method."""
    
    def __init__(self):
        pass
    
    async def generate_content_async(self, prompt_list):
        """Mock async method that returns a response object."""
        response = MagicMock()
        response.text = "This is a generated response text."
        return response


# Create mock model instance
torch.manual_seed(0)
model = MockGenerativeModel()

# Define arguments for generate_async
ARGS = {
    'model': model,
    'prompt': 'What is the meaning of life?',
    'generation_config': None,
    'safety_settings': None,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_text Ran
ran on a synthesized input
deep-diver/llamaduo/src/gen/async_generate_data_gemini.py
code served (permissive licence) · get_code("a1849d123c6ba549")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import asyncio
import aiohttp

# Mock the GOOGLE_API_KEY since it's used in the function
GOOGLE_API_KEY = "test_key_12345"

# The function takes a single parameter: prompt
# prompt should be a string based on how it's used in the function

ARGS = {
    'prompt': 'What is the capital of France?'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
aiohttp — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_push_to_hf_hub_enabled Ran
ran, violated the contract we drafted
deep-diver/llamaduo/utils.py
code served (permissive licence) · get_code("5a05b366cd7e1ed8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    "push_enabled": True,
    "dataset_id": "my-dataset",
    "split": "train"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_all_json_files Ran
ran; our contract draft was wrong, not the code
deep-diver/llamaduo/src/gen/collate_as_datasets.py
code served (permissive licence) · get_code("a27454db22d18423")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import glob
import json
import tempfile
import os

# Create a temporary directory with JSON files
temp_dir = tempfile.mkdtemp()

# Create sample JSON files
json_data_1 = {"key1": "value1", "key2": 42}
json_data_2 = {"name": "test", "data": [1, 2, 3]}
json_data_3 = {"nested": {"inner": "value"}}

with open(os.path.join(temp_dir, "file1.json"), "w") as f:
    json.dump(json_data_1, f)

with open(os.path.join(temp_dir, "file2.json"), "w") as f:
    json.dump(json_data_2, f)

with open(os.path.join(temp_dir, "file3.json"), "w") as f:
    json.dump(json_data_3, f)

ARGS = {
    "path": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_local_lm_responses Not yet run
unverified (harvested, no recorded run)
deep-diver/llamaduo/src/pipeline/batch_inference.py
code served (permissive licence) · get_code("f533d60c6a9bbd90")
How it ran

no invocation recorded.

gen_model_outputs Not yet run
unverified (harvested, no recorded run)
deep-diver/llamaduo/src/gen/local_lm.py
code served (permissive licence) · get_code("433721422ea96a6b")
How it ran

no invocation recorded.

get_model Not yet run
unverified (harvested, no recorded run)
deep-diver/llamaduo/src/gen/local_lm.py
code served (permissive licence) · get_code("12e77bbcdff15306")
How it ran

no invocation recorded.

update_args Not yet run
unverified (harvested, no recorded run)
deep-diver/llamaduo/utils.py
code served (permissive licence) · get_code("d09622f684fbfbc1")
How it ran

no invocation recorded.

Also present in other papers' code

Of 14 samples lifted from this paper's linked repositories, 14 carry a function identity and 0 do not. Of the 14 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 14 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The widespread adoption of cloud-based proprietary large language models (LLMs) has introduced significant challenges, including operational dependencies, privacy concerns, and the necessity of continuous internet connectivity. In this work, we introduce an LLMOps pipeline, "LlamaDuo", for the seamless migration of knowledge and abilities from service-oriented LLMs to smaller, locally manageable models. This pipeline is crucial for ensuring service continuity in the presence of operational failures, strict privacy policies, or offline requirements. Our LlamaDuo involves fine-tuning a small language model against the service LLM using a synthetic dataset generated by the latter. If the performance of the fine-tuned model falls short of expectations, it is automatically improved through additional fine-tuning using extra similar data generated by the service LLM. This multi-turn process guarantees that the smaller model can eventually match or even surpass the service LLM's capabilities in specific downstream tasks, offering a practical and scalable solution for managing AI deployments in constrained environments. Extensive experiments with leading-edge LLMs are conducted to demonstrate the effectiveness, adaptability, and affordability of LlamaDuo across various downstream tasks. Our pipeline implementation is available at https://github.com/deep-diver/llamaduo.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2408.13467")
get_code_for_paper("2408.13467")
have("2408.13467")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2408.13467.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2408.13467.svg)](https://syntology.ai/paper/2408.13467)

Connect an agent — have() is free.