# The Search for the Reward Function

*Instead of building agents, we should focus on the systems that reward them*

By [Nye's Digital Lab](https://paragraph.com/@nyewarburton.eth), 2025-12-07

systems, design, ai, play, reward

---

* * *

**Nye’s Digital Lab** is a weekly scribble on _creativity_ in the age of **AI** and **Distributed Systems.**

[Subscribe](https://paragraph.com/@nyewarburton.eth/subscribe)

  

This week I’m focusing on **the reward function**, and figuring out the best way AI can _optimize_ for it.

* * *

  
  

### Playgrounds

As soon as I parked, my kids opened the car doors and ran into the park.

There must have been twenty other kids scattered in twenty different directions. Most on swings, slides, the weird spinny thing that makes everyone nauseous. But some just went to the fence and started trading Pokémon. Others joined in a random game of _“chase.”_

No one told them where to go. _They just knew._

Each kid was solving their own optimization problem:

> maximize **fun**, minimize **boredom**, avoid **the big kid** who hogs the monkey bars.

I started to think of the park as **a system.** The reward function was built on _“fun.”_

For some, fun equals velocity and **air time** on the swings. For others, it’s **social connection**, huddling by the fence trading cards. For one kid I watched, it was **the pure satisfaction** of going down the slide several times in a row, because apparently that’s what peak performance looks like when you’re seven.

The playground works because it offers multiple reward pathways. Kids self-organize based on their individual motivations. Nobody needs to tell them where to go; the design of the space and their internal drives handle that _automatically._

Once you start looking for these patterns, understanding what motivates every agent in a system, you can’t unsee them. In a world inhabited by AI, understanding systems might be the most important skill we can develop.

  
  

* * *

![](https://storage.googleapis.com/papyrus_images/b3deb83b011768f05d11b67b5a1229baad040a25c868e9ef32062d0a836cd90e.jpg)

_Despite all my Rage, Stable Diffusion_

* * *

  
  

### I. What Motivates an Agent?

The first principle of systems thinking is simple:

> if you want to understand a **system**, figure out what its **agents** are **chasing.**

  

![](https://storage.googleapis.com/papyrus_images/e5f3b7a57ea9c2184a04c85ba38803ff0cfc428370ce4cc3c2569879c4ebfbc1.png)

_PacMan and Ghosts, Source: founditemclothing.com_

* * *

Here’s a classic [_reinforcement learning_ problem:](https://paragraph.com/@nyewarburton.eth/here-comes-reinforcement-learning) **teaching a robot to walk.**

Researchers create a virtual environment where a digital stick-figure flails around with joints in all the wrong places. At first, it’s _useless._ Its legs are spasming, torso flopping, going absolutely nowhere. But every time it moves forward, even a centimeter, you reward it with points. **Digital dopamine.** The robot doesn’t _“understand”_ walking.

It just knows:

> do thing, _get points_.
> 
> Do better thing, _get more points_.
> 
> Fall on face, _get no points._

* * *

[![](https://paragraph.com/editor/youtube/play.png)](https://www.youtube.com/watch?v=xAXvfVTgqr0)

* * *

Above is a video from youtube of someone training a quadruped robot to walk.

Through thousands of iterations, it _“figures out”_ walking. The reward function _is_ its motivation, its purpose, its entire reason for being.

* * *

  

**_Cut to:_** psychology experiments with rats in mazes.

The researcher doesn’t teach the rat a step-by-step choreography through corridors. They put cheese at the end.

The rat figures it out because the reward is clear:

> cheese exists, I **want** cheese, therefore...
> 
> I **will learn** this maze.

If you moved the cheese halfway through instead of at the end, the rat would optimize for a different path entirely. What if you electrified certain paths? Now you’ve introduced a _loss function_. That’s the opposite of the reward function. It’s the thing to avoid, the penalty for wrong moves.

The entire system includes the maze, the rat’s behavior, and the paths it takes. All of it is shaped by where you put the cheese and how you structure the penalties. Change the incentives, _change the system._

This is the fundamental insight that **unlocks everything else.**

* * *

![](https://storage.googleapis.com/papyrus_images/5992a4023ebcb7da87fb54fc5b570529df5c5efe5e1295b41a30eb842e74c1ba.jpg)

Standing Alone, Stable Diffusion

* * *

  
  
  

### II. Reward Functions

Once you start looking for **reward functions**, you see them **governing every system** around you.

  

![](https://storage.googleapis.com/papyrus_images/8092062c1b2500c08fae0523519ff9b4a3f18eafc5f70e4bd80bd4c7710f57ef.jpg)

Why don’t most people go 95 mph everywhere?

Not because cars can’t go that fast. Certainly not because we’re all responsible citizens who deeply contemplate traffic safety. It’s because there’s a speed limit sign, and behind that sign is **a system of enforcement** and penalties.

The **reward function** for speeding (getting there faster, feeling the rush) is outweighed by the **loss function** (tickets, insurance hikes, accidents, death).

> _This is_ **_governance_** _through_ **_systems design._**

We don’t rely on every driver to independently calculate optimal social outcomes, we build rules into the environment. The system guides behavior by making certain choices more or less attractive.

Beyond humans, this is important for artificial agents too. An AI optimizing for _“engagement”_ gave us social media’s infinite scroll and algorithmic radicalization. An AI optimizing for _“efficiency”_ might route delivery trucks through residential neighborhoods at 3 AM. An AI optimizing for _“profit”_ without constraints will find every loophole, every edge case, every technically-legal-but-morally-questionable shortcut available.

The problem isn’t the AI being evil.

The AI is just doing what the robot learning to walk does: _maximizing its reward function._ The problem is we pointed it at **the wrong thing** and forgot to add appropriate loss functions.

  
  

* * *

![](https://storage.googleapis.com/papyrus_images/068635bbed0341132e47cf1cf6a21338c91f2672009548b581d55c6558968541.jpg)

_Putting it together, Stable Diffusion_

* * *

  
  

### III. Designing Systems

  

> As we architect systems for an AI-inhabited world, what are we actually **incentivizing?**

Everyone’s obsessed with making agents smarter, **right?** We focus on better algorithms, more training data, faster processing. Maybe we stop with _the obsessive scaling?_

I think that we’re not considering _nearly enough about the systems these agents will operate within._

What does “success” look like? Who decides? What are the boundaries?

> What are we **maximizing** for?!?

The agent optimizes for whatever objective function we give it, with no inherent sense of whether that objective is actually useful or good.

A company that rewards **quarterly earnings** above all else will cut costs even as quality suffers. A social media platform that rewards **time-on-site** will optimize for addictiveness over wellbeing. A school that rewards **test scores** will teach to the test [instead of fostering curiosity.](https://paragraph.com/@nyewarburton.eth/finding-passion-in-china)

We are all rats in mazes designed by other people, by history, by markets, by culture. But we’re also the _maze designers._ We choose where to put the cheese. We decide what to reward and what to penalize. We architect the systems that shape behavior.

So how do you find a reward function? You observe.

You ask: what does this entity keep doing? What is it moving toward? What is it avoiding? If you released this agent into an environment with no instructions, where would it go?

The maze is being built right now.

**We get to choose where the cheese goes.**

  

Let’s try not to optimize for **the wrong thing?**

  
  

* * *

Thanks for reading. I do this every week. If you _vibe_ to the ideas I express, consider **subscribing** or **sharing** with friends. We'll see you next time.

  

[Subscribe](https://paragraph.com/@nyewarburton.eth/subscribe)

  

* * *

**Nye Warburton** is an educator and systems designer from Savannah, Georgia. This essay was improvised with [**Otter.ai**](http://Otter.ai) and refined and edited using **Claude Sonnet 4.5.**

For more information visit: [https://nyewarburton.com](https://nyewarburton.com)

* * *

### More Essays from the Digital Lab

*   [Here Comes Reinforcement Learning](https://paragraph.com/@nyewarburton.eth/here-comes-reinforcement-learning), March 30, 2025
    
*   [Finding Passion in China](https://paragraph.com/@nyewarburton.eth/finding-passion-in-china), November 30, 2025
    
*   [Why you should consider Savannah, GA](https://paragraph.com/@nyewarburton.eth/why-you-should-consider-savannah-ga), June 22, 2025
    

* * *

[Share](https://paragraph.com/@nyewarburton.eth/IqrHBD6iNpXnFTYCdQKJ)

​​​​​​​​​​​​​​​​

---

*Originally published on [Nye's Digital Lab](https://paragraph.com/@nyewarburton.eth/the-search-for-the-reward-function)*