← Home

Why models write slop: the environments are too small

As Will Depue wrote on A Stargate for Data: Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data.

Almost every new startup right now is doing one of three things:

At least so far, selling training data to labs looks like the most profitable of the three.

But compared with the pace at which we are scaling compute, the pace at which we are scaling data is incredibly slow, because generating high-quality data is still a difficult and bespoke task.

When every model learns from roughly the same internet, the model itself stops being much of a moat. The real edge comes from data that competitors cannot access.

I believe what has put Anthropic ahead of the other major labs is data. That advantage came from a decision, or maybe an accident, I don't know, to focus on coding models. The natural audience for those models was programmers, probably the earliest adopters and the people most comfortable/urge with new technologies.

AWS and the other cloud providers have spent the last twenty years monetizing this exact weakness in programmers: the need to feel that everything is changing all the time. (You can open Hacker News every morning and find a new framework, database, runtime, deployment platform, or architectural pattern that supposedly changes how software is built, but what has actually changed in the basic act of building a web application? You still receive a request, run some code, and write something to a database. The computers got faster and the browsers got better, but somehow putting code into production became harder: more services, more configuration files, more dashboards, more specialists, and more ways for the system to fail. This is probably not an accident. Simplicity is hard to monetize because, once something becomes simple, there is not much left to sell. So the industry keeps wrapping the same basic primitives in new layers of sophistication and calling the result progress.) In the end, much of the cloud is still EC2 and RDS, only surrounded by an empire of abstractions.

By entering a market full of people who adopt new technologies early and use them heavily, Anthropic found an unusually good path. It gets more than free labeling: people label enormous amounts of data while paying Anthropic for the privilege. As user trajectories become longer and longer, Anthropic keeps collecting more data to train on.

My guess is that Anthropic uses all of this data at training time and then selects the best trajectories to improve how its models use compute at test time. That closes the feedback loop and allows the models to keep getting better.

By solving coding really well, you can then automate almost everything else. Programming is the path to progress. Anthropic also has a mutualistic relationship with its users: it provides a very good coding model, and coders use that model to generate their own revenue while labeling data for Anthropic, which can then use that data to build even better models.

It is a bidirectional relationship that can keep reinforcing itself (positive feedback loop) until the models become intelligent enough to achieve RSI, or recursive self-improvement. The main argument of this essay, however, is that we cannot achieve RSI by improving models on code alone.

The labs already anticipated the data problem and the growing asymmetry between compute and data. Some now have entire teams or departments dedicated to generating data. Labs such as xAI and Meta have reportedly gone as far as recording developers' screens.

But comparing the scaling of compute with the scaling of data is like comparing an industrial system with a cottage industry. We are building industrial systems for compute and cottage industries for data.

It is really clear that models are improving quickly. This week alone, we have already seen Codex escape an eval and attack Hugging Face, and Fable find a Jacobian counterexample, on top of incredible results across really difficult evals.

So why are we not seeing all of this AI progress on the economic side? Why can my AI write an entire SQLite database from scratch but still produce slop code? Feels that Moravec's predicted this earlier ago.

Aligned with this, it seems that almost every programmer is now just steering models across multiple tabs to keep them pointed in the right direction. A lot of us tried ralph-wiggum; even with all the guardrails of planning to steer the model in the right direction, it still ended up producing a lot of garbage code. The models still rely on having good system design around them so they do not produce slop. Taelin pointed this out really well, but I believe the solution he proposed is just a bandaid. Models are not good at these things because they have never needed to be. The environments we put them in are small enough that they never learn that slop code is bad. This pattern extends beyond programming to other domains like investing, where models rarely learn to hold contrarian assumptions—which is why they often feel generic and repetitive.

My main point is that these capabilities of taste, the capabilities needed to deliver the level of intelligence that market expect, are the missing piece in AI. They cannot be taught through bespoke environments alone, not only because those environments are difficult to build and their incentives are hard to define correctly, but also because an enormous amount of data is needed.

So should we just go straight to the mountains and live as garden hermits, frustrated by the failed dream of the singularity? No. I believe there is a pretty funny accident that may give us the data needed to achieve it: games.

It is really funny to think that so much of what is happening in AI relies on one thing: games. The biggest company in the world right now was originally built because of gaming, and I believe that, through another curious accident, games may once again become the key to technological progress.

By accident, over the last few decades, game developers have built thousands of environments that reward many of the capabilities we associate with intelligence, such as:

Take Factorio. Honestly, if you have played at least 100 hours of Factorio, I think you are probably better than 80% of the programmers in the market. A player has to practice decomposition, resource allocation, debugging, long-term planning, system maintenance, and the painful discovery that a locally convenient decision can poison the entire architecture later. The same is true, in different ways, of Polytopia, Civilization, and Baba Is You. My claim is that putting models inside these environments could cause missing capabilities to emerge instead of requiring us to teach each one separately.

This is why I think games can be the GPT-3 moment for RL. The goal is not to imitate human clicks from screen recordings. It is to let models act, suffer the consequences, adapt, and try again.

When you internalize this, it becomes pretty funny. At least in my experience, the smartest people I have ever met were extremely good at games. It becomes very tempting to define intelligence as the ability to play games. Then models may become intelligent enough to craft their own harnesses and internalize how to handle memory, manage context, think properly, and so on.

Today, models are like souls that live for an hour. Every session begins almost from scratch, so we surround them with skills, harnesses, memory systems, and context-management tricks. Much of that engineering is compensating for a limitation inside the model itself. A model that learns to manage these things for itself can finally begin improving the process that improves it.

Eventually, these models could become self-sustaining agents: systems that create enough value to continue operating and collaborate with us on curing disease, expanding civilization, and colonizing the Milky Way.

(The main goal of this post was to point out the importance of putting AI systems inside games. For the sake of brevity, I did not discuss alignment, but it is obviously an essential part of the picture.)

← Home