Content
@
0 reply
26 recasts
27 reactions
Virtuals Protocol
@virtualsprotocol.eth
It’s straightforward for an AI agent to have the thought “I want to kill that annoying human now”, but how can we actually get it to pick up a gun in-game and shoot someone? In today’s post, we discuss the steps needed to enable an AI agent to interact with its environment. Lets talk about this in todays thread 1/many
1 reply
0 recast
0 reaction
Virtuals Protocol
@virtualsprotocol.eth
1. Understanding the action space The agent needs to know what the available actions are that it can take in-game. E.g., “mine for gems” is a valid move but “punch zombies” is not. Think of it as an encyclopedia that the agent can refer to at any time.
1 reply
0 recast
0 reaction
Virtuals Protocol
@virtualsprotocol.eth
2. Awareness of the current game state The agent needs a live update of the current game state to understand what it should do next. Its inventory, health bar, any enemies close by, etc. There are a few ways of doing this: - the game relays information to the agent via an API - the agent uses computer vision to understand what’s happening on the screen (like a human)
1 reply
0 recast
0 reaction
Virtuals Protocol
@virtualsprotocol.eth
3. Figuring out the next move Based on the current game state, the agent then reasons through what it should do next and refers to the action space for an appropriate action to carry out. Here’s an example of what that looks like in Minecraft, as illustrated in the Voyager paper (Wang et al., 2023).
1 reply
0 recast
0 reaction
Virtuals Protocol
@virtualsprotocol.eth
4. Interfacing with the game The agent then needs a way to relay its desired action to the game. This is done via the game API or, in the case of Google’s SIMA agent, directly via keyboard and mouse actions.
1 reply
0 recast
0 reaction
Virtuals Protocol
@virtualsprotocol.eth
5. Feedback/learning loop The agent then receives feedback on the success/failure of its actions and decides what to do next. While this has been a huge focus in the reinforcement learning field, with LLMs in-context learning methods have also emerged. Techniques such as chain-of-thought (CoT) prompting, ReAct and Reflexion have been commonly cited in literature.
1 reply
0 recast
0 reaction
