It’s straightforward for an AI agent to have the thought “I want to kill that annoying human now”, but how can we actually get it to pick up a gun in-game and shoot someone? In today’s post, we discuss the steps needed to enable an AI agent to interact with its environment.

Lets talk about this in todays thread 1/many

Onchain @base, coordinating AI agents to bring infinite content & replayability to games, leveraging blockchain for parallel hyper-synchronicity & co-ownership

2. Awareness of the current game state

The agent needs a live update of the current game state to understand what it should do next. Its inventory, health bar, any enemies close by, etc. There are a few ways of doing this:

- the game relays information to the agent via an API
- the agent uses computer vision to understand what’s happening on the screen (like a human)

3. Figuring out the next move

Based on the current game state, the agent then reasons through what it should do next and refers to the action space for an appropriate action to carry out.

Here’s an example of what that looks like in Minecraft, as illustrated in the Voyager paper (Wang et al., 2023).

4. Interfacing with the game

The agent then needs a way to relay its desired action to the game. This is done via the game API or, in the case of Google’s SIMA agent, directly via keyboard and mouse actions.

5. Feedback/learning loop

The agent then receives feedback on the success/failure of its actions and decides what to do next. While this has been a huge focus in the reinforcement learning field, with LLMs in-context learning methods have also emerged. Techniques such as chain-of-thought (CoT) prompting, ReAct and Reflexion have been commonly cited in literature.

The future of AI agents playing games alongside and on behalf of humans has never been nearer. Join Virtuals Protocol to shape this exciting future as a game developer, ecosystem partner, or community member.

1. Understanding the action space

The agent needs to know what the available actions are that it can take in-game. E.g., “mine for gems” is a valid move but “punch zombies” is not. Think of it as an encyclopedia that the agent can refer to at any time.