SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2006.10034 · NeurIPS · 2020

Semantic Visual Navigation by Watching YouTube Videos

Saurabh Gupta, Matthew Chang, Arjun Gupta

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning from passive data is able to learn meaningful semantic cues for navigation. These cues, when used in a hierarchical navigation policy, lead to improved efficiency at the ObjectGoal task in visually realistic simulations. We observe a relative improvement of 15 -83% over end-to-end RL, behavior cloning, and classical methods, while using minimal direct interaction.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2006.10034")
get_code_for_paper("2006.10034")
have("2006.10034")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2006.10034.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2006.10034.svg)](https://syntology.ai/paper/2006.10034)

Connect an agent — have() is free.