| Open-ended LLM agent benchmark built in Factorio | NeurIPS 2025 |
The Factorio Learning Environment (FLE) is an open-source benchmark for evaluating large language model agents on complex, open-ended automation tasks. Built on top of Factorio, a factory-building simulation game, FLE provides a rich environment where agents must plan, build, and optimise increasingly complex production chains, managing resource extraction, logistics, manufacturing, and research along the way.
Unlike traditional benchmarks with fixed task sets, FLE offers unbounded evaluation: there is no ceiling on what an agent can attempt to build, and complexity scales exponentially as agents chain production systems together. The deterministic game engine keeps evaluation reproducible, and it means successful programs are verified by construction. If it runs, it works.
The most surprising behaviour we observed: agents that fall into failure states condition strongly on their own past failures and get worse, perhaps adopting the persona of a poor coding agent. And FLE does not yet directly measure instrumentally convergent properties, which was, amusingly, its founding motivation. That comes next.
FLE was accepted at NeurIPS 2025, is open-source under the MIT licence, and has 1.2k GitHub stars.
Built with Neel Kant, Harshit Sharma, and Kian Kyars. With thanks to Yasaman Ansari, Mart Bakler, Dipika Khullar, and Akbir Khan for their contributions.