SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2302.00111 · NeurIPS · 2023

Learning Universal Policies via Text-Guided Video Generation

Hanjun Dai, Bo Dai, U Berkeley, Joshua Tenenbaum, Dale Schuurmans, Pieter Abbeel, Yilun Du, Mit, Ofir Nachum, Mengjiao Yang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots 2 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2302.00111")
get_code_for_paper("2302.00111")
have("2302.00111")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2302.00111.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2302.00111.svg)](https://syntology.ai/paper/2302.00111)

Connect an agent — have() is free.