Starting from REINFORCE, the original deep reinforcement learning algorithm, we will trace the evolution of policy gradient methods to the Group Relative Policy Optimization algorithm used to train Deepseek r1.

This post ignores the LLM side of things, less-related developments in RL, and most of the equations used for these algorithms, but captures the essence and intuition of the RL-timeline without wasting your time. This is all self-study, so feel free to send me any corrections/suggestions.¹

I don’t think r1 will get us to artificial super intelligence, but whatever comes next probably will.

We are reaching a familiar bottleneck in AI. Previously, humans had to manually hardcode the patterns that AI could recognize. With deep learning, machines began to learn patterns on their own, without human assistance. With (relatively) expensive humans out of the loop, we threw machines at the world’s data until they began to talk, code, and paint. Many people believed this would be sufficient to reach artificial super intelligence–but it wasn’t.

It used to be good enough just to copy others. Now, with AI in the hands of billions, there’s little value in copying.

For instance, take programming. Five years ago, building apps, websites, or games required a non-trivial amount of skill, and getting your first project off the ground was an accomplishment. Now, AI can generate most starter projects in hours, if not minutes. I think this decimates the reward, both internal and external, of actually completing the first few projects.

I’ve been an author on a few papers before, but I recently published the first research project where I was responsible for most of the work and direction. It’s in the first 2024 issue of the journal Artificial Life, which you can find here. You can find a non-paywalled version here Below, I tell the chronology of the project and summarize our findings.

We explore the conditions under which trade can emerge between four deep reinforcement learning agents that pick up and put down resources in a 2D foraging environment. Agents are rewarded for having both resources once, but the resources are distributed far apart from each other. To maximize reward, agents need to split up the work - agent 1 goes to resource A, agent 2 goes to resource B, etc, and then they meet to exchange resources, since meeting halfway can get them the most of each resource in the shortest amount of time.

An ever-expanding list of concepts in the field of AI to give myself and others an easy reference. Each item in the list contains a short, rudimentary definition I’ve written, as well as a link to a resource that can explain it better.

Ablation Study: Removing some parts of a machine learning model to measure impact on performance

Advantage Function: The difference between a Q-value for a state-action pair and a value for the state. Useful to determine how good an action is relative to its state.

Here’s the problem with advertising AI-based technology that doesn’t exist:

You cannot promise anything about your product.

We’ve all seen AI advertised to the masses that doesn’t work as advertised, just look at any voice-to-text system. When I got my Apple Watch, I hoped to use it to respond to messages without getting distracted by my phone. I quickly realized that wasn’t a viable solution: I had to repeat my message multiple times per text in order to get the correct dictation.