Tong Chen, Yejin Choi, Luke Zettlemoyer, Hannaneh Hajishirzi, Sewon Min, Akari Asai, Niloofar Mireshghallah, Pang Koh, James Grimmelmann, Mrs Dursley
We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| chentong0/copy-bench | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| call_gpt | Ran | chentong0/copy-bench/src/utils/openai_lm.py pointer only (licence: NONE) · get_code("87aa8dba7017b976") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Evaluating the degree of reproduction of copyright-protected content by language models (LMs) is of significant interest to the AI and legal communities. Although both literal and non-literal similarities are considered by courts when assessing the degree of reproduction, prior research has focused only on literal similarities. To bridge this gap, we introduce COPYBENCH, a benchmark designed to measure both literal and non-literal copying in LM generations. Using copyrighted fiction books as text sources, we provide automatic evaluation protocols to assess literal and non-literal copying, balanced against the model utility in terms of the ability to recall facts from the copyrighted works and generate fluent completions. We find that, although literal copying is relatively rare, two types of nonliteral copying-event copying and character copying-occur even in models as small as 7B parameters. Larger models demonstrate significantly more copying, with literal copying rates increasing from 0.2% to 10.5% and non-literal copying from 2.3% to 5.9% when comparing Llama3-8B and 70B models, respectively. We further evaluate the effectiveness of current strategies for mitigating copying and show that (1) training-time alignment can reduce literal copying but may increase non-literal copying, and (2) current inference-time mitigation methods primarily reduce literal but not non-literal copying.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2407.07087")
get_code_for_paper("2407.07087")
have("2407.07087")
Connect an agent — have() is free.