Jack Hopkins Independent AI Safety Researcher

Factorio Learning Environment

Open-ended LLM agent benchmark built in Factorio NeurIPS 2025

Overview

The Factorio Learning Environment (FLE) is an open-source benchmark for evaluating large language model agents on complex, open-ended automation tasks. Built on top of Factorio, a factory-building simulation game, FLE provides a rich environment where agents must plan, build, and optimise increasingly complex production chains.

Unlike traditional benchmarks with fixed task sets, FLE offers unbounded evaluation — there is no ceiling on what an agent can attempt to build, making it a powerful testbed for measuring general-purpose reasoning and long-horizon planning.

Key Features

  • Unbounded evaluation: No fixed task ceiling; complexity scales exponentially as agents chain together production systems
  • Automation challenges: Agents must manage resource extraction, logistics, manufacturing, and research
  • Exponential complexity: Each layer of automation introduces compounding decision-making requirements
  • Verified by construction: The deterministic game engine means successful programs constitute correctness proofs — if it runs, it works
  • Reproducible: Deterministic game engine ensures consistent evaluation across runs

Stats

  • 935+ GitHub stars
  • NeurIPS 2025 accepted paper
  • Open-source under MIT licence

[Demo screenshot placeholder]

Collaborators

With thanks to Yasaman Ansari, Mart Bakler, Dipika Khullar, and Akbir Khan for their contributions.