SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2503.20194 · ACL · 2025

GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization

Yanghua Xiao, Tao Wang, Tianyu Li, Zhouhong Gu, Hongwei Feng, Suhang Zheng, Xingzhou Chen, Xiaoran Shi

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recent advances in large language models have highlighted the critical need for precise control over model outputs through predefined constraints. While existing methods attempt to achieve this through either direct instructionresponse synthesis or preferential response optimization, they often struggle with constraint understanding and adaptation. This limitation becomes particularly evident when handling fine-grained constraints, leading to either hallucination or brittle performance. We introduce Generative Adversarial Policy Optimization (GAPO), a novel framework that combines GAN-based training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. GAPO leverages adversarial training to automatically generate training samples of varying difficulty while utilizing the encoder-only architecture to better capture prompt-response relationships. Extensive experiments demonstrate GAPO's superior performance across multiple benchmarks, particularly in scenarios requiring fine-grained constraint handling, where it significantly outperforms existing methods like PPO, DPO, and KTO. Our results suggest that GAPO's unique approach to preferential prompt learning offers a more robust and effective solution for controlling LLM outputs. Code is avaliable in https://github.com/MikeGu721/GAPO.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2503.20194")
get_code_for_paper("2503.20194")
have("2503.20194")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2503.20194.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2503.20194.svg)](https://syntology.ai/paper/2503.20194)

Connect an agent — have() is free.