If you’re tired of hearing people say the future is AI, gear up because you’re going to continue hearing it everywhere now. Google Deepmind’s Genie 2 is a new foundational tool that can create 3D environments designed for training and evaluating embodied agents. With a single prompt, you can access the pipeline from text to playable game AI, which is something groundbreaking to be sure.
The AI tool is capable of creating diverse, playable 3D environments using prompts, and a user can interact with the manifested environment with a mouse or keyboard. Unsurprisingly, the user doesn’t have to be human as an AI can provide the movement inputs as well. The provision of an AI-generated interactive world is impressive all on its own but its ability to provide a training ground for AI is equally useful.
Image: Google DeepMind
Google DeepMind Genie 2—Why Should We Care About AI-Generated Video Game Worlds?
Games can be fun and they can be informative, but another thing games can achieve is to create a safe and controlled environment for understanding and training AI. DeepMind has been leading the pack when it comes to the training and building of AI on the backs of games but there are often limitations on the kinds of environments that can be made available for these training sessions.
The real world is exceedingly complex and it’s nearly impossible to keep recreating the kinds of contexts necessary to fully prepare an AI. Whether it is the completion of simple tasks or the addressing of real-world problems, AI and most other programs can benefit from comprehensive testing to work out the kinks in the code or capability.
How Does Genie 2 Work Differently?
DeepMind’s Genie 1 has been revolutionary in rendering 2D worlds to train AI and now Genie 2 is taking it a step further by generating diverse 3D worlds. Referred to as a “world model” the tool can simulate virtual worlds where we can set up actions and consequences within the environment.
Trained on large-scale video datasets, the model “demonstrates various emergent capabilities at scale, such as object interactions, complex character animation, physics, and the ability to model and thus predict the behavior of other agents.” Light, smoke, gravity, and other elements accurately reflect reality.
Genie 2 seems to be in par with @worldlabs in terms of fidelity, but has much more advanced in-world interactions. Very impressive. pic.twitter.com/4Nru2T7vw6
— Chris McKay (@cmcky) December 4, 2024
With its AI-generated video game world, Genie 2 can respond to actions taken by the users whether input by a keyboard or a mouse, and move the right components on the screen. The AI is also able to generate counterfactual experiences depending on the reaction of the users, and the generated content keeps with reality.
Image: Google DeepMind
The AI-Generated Interactive Worlds Are Diverse and Complex
The playable 3D environments are exceedingly diverse and also support the generation of different angles and views, whether you want a first-person perspective or need a third-person view. Just like one would expect in a video game, users can interact with the objects and have them respond appropriately. Slashing a balloon can burst it and pushing a door will allow you to open it, as designed.
Interestingly, not only does DeepMind Genie 2 generate objects, but there is room for NPC creation and character design as well. The AI-generated video game environments also showcase long-horizon memory allowing the world that is no longer in view to be rendered correctly again when you face it. The tool can also continue to generate the world as you move forward for up to a minute.
Introducing Genie 2: our AI model that can create an endless variety of playable 3D worlds – all from a single image. 🖼️
These types of large-scale foundation world models could enable future agents to be trained and evaluated in an endless number of virtual environments. →… pic.twitter.com/qHCT6jqb1W
— Google DeepMind (@GoogleDeepMind) December 4, 2024
The rapid prototyping abilities allow for a much faster test of an AI’s capabilities. An AI-generated image can be expanded into a virtual world with DeepMind Genie 2, and another AI can be prompted to explore and interact with this virtual world. The interesting outtakes that Google kindly attached with the Genie 2 reveal show us that the AI isn’t perfect, but it is still one of the most efficient tools we’ve seen recently.
Thankfully, Genie 2 is poised to strictly be a research tool and not one that’s planning to put game developers out of a job. The 3D environments that are generated are set up to aid other AI tools in becoming more sophisticated while also improving their own capabilities as the trials progress.