Not many know that Elon Musk, as an intern many summers ago, wrote the software for video games at a company called Rocket Science. This was long before SpaceX was even conceived.
As a summer intern many years ago, I wrote software for video games at a company called Rocket Science. This was long before SpaceX was even conceived.
Didn’t realize until a friend gave me a copy today that they included me in the credits 🥰 pic.twitter.com/adU1J81ESq
— Elon Musk (@elonmusk) October 3, 2023
And 12, Musk created his first video game, called Blastar, which was a simple space invaders-inspired side-scroller. He sold the game’s code to a PC magazine in South Africa for $500.
The Tesla CEO has often admitted that playing video games helps him unwind. In a recent podcast with Lex Fridman, Musk said, “Killing the demons in a video game calms the demons in my mind.”
Interestingly, now would be a great time for Musk to start a hyper-realistic gaming company.
Text-to-video games seem to be becoming a reality. Google DeepMind recently introduced GameNGen, the world’s first game engine driven entirely by neural models capable of simulating complex video games like DOOM in real-time without traditional coding.
It uses a neural network to generate game frames at over 20 frames per second, producing visuals that are nearly indistinguishable from the original game.
The AI model in GameNGen uses reinforcement learning to learn from recorded sessions, followed by a diffusion model that generates subsequent frames based on past actions. This process enables seamless and realistic gameplay that enhances the player experience.
An AI agent was trained to play DOOM using reinforcement learning, with its gameplay data serving as the foundation for the GameNGen model. The model underwent training on 128 TPU v5e accelerators for 700,000 steps, generating a total of 900 million training frames.
“Tesla can do something similar with real-world video,” said Musk. Comma.ai founder George Hotz responded, “So can Comma.ai.”
Agreeing with Musk, NVIDIA senior manager Jim Fan said, “Not surprising, the Autopilot team likely has trillions of (camera feed, steering wheel action) pairs. Again, data is the hard part! With such rich real-world data, it’s entirely possible to learn a general driving sim that covers all kinds of edge cases and use that to deploy and verify a new FSD build without physical cars.”
Musk recently gave a sneak peek into Tesla’s supercomputer Cortex, which has enough compute resources to train video game models. “Just did a walkthrough of the Tesla supercomputer cluster at Giga Texas (aka Cortex). This will be ~100k H100/H200 with massive storage for video training of FSD & Optimus,” Musk posted on X.
While Musk is focusing on FSD and Optimus, pivoting to video games and entertainment seems like a promising prospect. And while OpenAI hasn’t released Sora yet, Musk said that Tesla has already been doing real-world video generation with accurate physics for about a year.
Earlier this year, Musk shared Tesla’s video generation capabilities, to which a user replied, “Tesla should make a video game.” Musk instantly responded, “I’ve wanted to do that for a long time :).”
He said that while Tesla’s real-world simulation and video-generation capabilities are the best in the world, making a game can only come after they release unsupervised FSD, which is far safer than supervised FSD.
I’ve wanted to do that for a long time 🙂
Our real-world simulation and video generation is the best in the world, but unfortunately making a game can only come after we release unsupervised FSD that is far safer than even supervised FSD.
— Elon Musk (@elonmusk) February 18, 2024
Moreover, with Neuralink, the possibilities are endless, as users would be able to create and control their own realities within video games using their thoughts.
Noland Arbaugh, a 30-year-old man who became the first human to receive a Neuralink brain implant, now browses the web, plays computer games, and performs tasks previously impossible for him.
Text-to-Video Games Coming Soon
GameNGen has given hope that building games won’t be difficult in the future.
“GameNGen will take a while to be fully adopted, but I can already see the massive impact it’ll have. Imagine games that are fully customised for each player, where you can feed the AI with your own photos and generate the story/characters/maps your own way, while maintaining the general storyline,” posted a user on X.
Beyond GameNGen, another notable model called MarioVGG has recently emerged. It is a text-to-video diffusion model designed for controllable video generation within the Super Mario Bros game.
MarioVGG showcases the ability to continuously produce consistent and meaningful scenes and levels while accurately simulating the physics and movements of a controllable player throughout the video.
MarioVGG
Video Game Generation: A Practical Study using Mario
model: https://t.co/rEBEwTpkEr
present MarioVGG, a text-to-video diffusion model for controllable video generation on the Super Mario Bros game. MarioVGG demonstrates the ability to continuously generate consistent… pic.twitter.com/LKjTgZIX8d
— AK (@_akhaliq) September 4, 2024
Recently, Anthropic introduced Artifacts in the Claude iOS and Android apps, allowing users to easily create mobile games by simply providing their ideas in the form of a prompt. The best part is, you can play the game instantly, right then and there. It’s safe to say that now, anyone can become a game developer.
Earlier this year, Google introduced Genie, an interactive game environment. Genie is a ‘foundation world model’ trained on over 200,000 hours of unlabeled video footage, primarily from 2D platformer games. It can create fully playable and interactive 2D game environments from single images, photos, sketches, or text descriptions.
With generative AI now easily capable of creating 3D worlds, game engines like Unity, Unreal, Roblox, and Godot might face some competition.
“The next Pixar will be an AI company built on interactive video – a blend of film & games where frames are generated entirely by a neural net in real-time, with no code, assets, or game engines,” said Jon Lai, partner at Andreessen Horowitz.
Moreover, NVIDIA recently announced NVIDIA ACE, a suite for creating digital humans with generative AI, now including the on-device Nemotron-4 4B Instruct model powered by RTX AI. This model enhances role-play, retrieval-augmented generation, and function-calling, allowing game characters to better understand and respond to player instructions.
Only time will tell if Musk beats everyone to creating the first hyper-realistic game with generative AI, or if we’ll be playing GTA 6 first.
