Prithviraj Ammanabrolu, Marc-Alexandre Côté, Peter Jansen, Ruoyao Wang
We lifted 8 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| allenai/macaw | canonical | 6 of 7 |
| allenai/scienceworld | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| build_simplification_str | Ran | allenai/scienceworld/examples/random_agent.py code served (permissive licence) · get_code("3c4d106b37c63938") |
| fix_t5_unk_characters | Ran | allenai/macaw/macaw/eval_utils.py code served (permissive licence) · get_code("20633ae7459cfada") |
| load_jsonl | Ran | allenai/macaw/macaw/utils.py code served (permissive licence) · get_code("c30d329a797e15bd") |
| replace_punctuation | Ran | allenai/macaw/macaw/eval_utils.py code served (permissive licence) · get_code("d2166ef001a9e416") |
| run_model | Ran | allenai/macaw/macaw/utils.py code served (permissive licence) · get_code("0ff0cabbc0770012") |
| run_model_with_outputs | Ran | allenai/macaw/macaw/batch_eval.py code served (permissive licence) · get_code("df88f11724e3cab5") |
| score_string_similarity | Ran | allenai/macaw/macaw/eval_utils.py code served (permissive licence) · get_code("00139853f0c3a89e") |
| load_model | Not yet run | allenai/macaw/macaw/utils.py code served (permissive licence) · get_code("a96d409802f5ecdc") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present SCIENCEWORLD, a benchmark to test agents' scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum. Despite the transformer-based progress seen in question-answering and scientific text processing, we find that current models cannot reason about or explain learned science concepts in novel contexts. For instance, models can easily answer what the conductivity of a known material is but struggle when asked how they would conduct an experiment in a grounded environment to find the conductivity of an unknown material. This begs the question of whether current models are simply retrieving answers by way of seeing a large number of similar examples or if they have learned to reason about concepts in a reusable manner. We hypothesize that agents need to be grounded in interactive environments to achieve such reasoning capabilities. Our experiments provide empirical evidence supporting this hypothesisshowing that a 1.5 million parameter agent trained interactively for 100k steps outperforms a 11 billion parameter model statically trained for scientific question-answering and reasoning from millions of expert demonstrations.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2203.07540")
get_code_for_paper("2203.07540")
have("2203.07540")
Connect an agent — have() is free.