Most of our AI research focuses on making difficult tasks a little easier but the SIMA AI gaming partner by Google is now offering to keep us company with the fun tasks as well. The new DeepMind co-op gaming AI, SIMA or “Scalable, Instructable, Multiworld Agent,” is designed to learn how to play different games and participate in them as a player you can team up with. Through natural language instructions and image recognition, the Google AI gaming partner has the ability to understand the environment of a game and interact with it as a player would, and this isn’t restricted to linear strategy games like Chess or Go. The AI is also capable of learning how to navigate open-world gaming spaces and other environments in order to move the game forward.
Image: Google DeepMind
Understanding SIMA: The AI gaming partner by Google
Strategy games with a computer as your opponent is not a new concept and many have long employed such automated mechanics where the device competes to win against you. The Google SIMA AI gaming companion is a little different. Instead of being trained on hours of material to master a single game, the AI is trained on competencies that it requires for a range of games. With 600 basic skills under its belt, the AI is being developed to really “play” the game like a regular human would. The skills range across categories like navigation, object interaction, and menu use, and the AI can execute simple tasks that take about 10 seconds to perform.
The SIMA AI gaming partner by Google shouldn’t be confused with regular NPCs either. These non-player characters usually have a fixed role and can’t do too much beyond delivering their lines or utilizing basic in-game mechanics to move the game forward. Here, the Google AI learns how to interact with these NPCs just like a real player would, creating a multiplayer experience for those on its team.
Training The Google SIMA AI Gaming Companion
To train the DeepMind co-op gaming AI, nine video games were used to expose it to different gaming contexts and their resulting rules and regulations. This made it possible for the AI to pick up on in-game tasks that it could accomplish, such as gathering resources and flying spaceships. The company focused on open-world games or sandbox ones that did not include any violent or gruesome content but instead had interesting mechanisms that the AI could be trained on. These games included Goat Simulator 3, Hydroneer, No Man’s Sky, Satisfactory, Teardown, Valheim, and Wobbly Life.
Google collaborated with eight game studios to gain access to these interactive setups and used four research environments to test the AI. The research process was evidently no small feat. One of the environments the team used was a new one they built on the Unity engine—the Construction Lab. As a part of the training material, pairs of humans were recorded playing games where one of them instructed the other on how to play the game. Solo players were also recorded playing the game. These players later rewatched the recording and added instructions on how to make those in-game actions possible. The natural language videos were a central part of the training for Google’s SIMA AI gaming companion.
Image: Google
Benefits of Sima AI Integration in Video Games
Essentially, the Google AI gaming partner can learn to react to environments and then follow instructions to complete tasks. Instead of changing the game’s source code to integrate the AI or accessing it through APIs, the AI model only requires an on-screen image and simple instructions to get the task of learning done, playing the game via keyboard and mouse inputs. Currently, it appears the SIMA AI gaming partner by Google can only follow simple instructions, but the goal appears to be executing multi-level prompts that require sequential tasks to be completed.
Google posits that training agents on multiple games were found to be infinitely better than training them on a single game, as these AI agents were capable of generalizing their training beyond the game that was presented to them upfront. Instruction and language make up a significant part of this learning and application. While the SIMA AI integration in video games is a fun application of AI, such advanced learning and application capabilities could have a significant application in everyday life.
“Ultimately, our research is building towards more general AI systems and agents that can understand and safely carry out a wide range of tasks in a way that is helpful to people online and in the real world,” Google states, and we can see the vision they have in mind. With more complex versions of the AI, we could migrate the in-game ability to drive a car and generalize it to the same real-world potential to do it flawlessly compared to the incomplete self-drive versions we have today. The possibilities are endless.