Nvidia's Facial Animation Technology to Go Open Source, Simplifying Game Development
Nvidia announces the open-source release of its Audio2Face technology, enabling game developers to create more realistic AI characters with lifelike facial animations.
Nvidia has announced that its Audio2Face animation technology is going open source. On paper, that should make it much easier for a wide range of game developers to create AI characters with convincing facial expressions, including during real-time conversations with gamers.
To recap and according to Nvidia’s own words, “by using large language and speech models, generative AI is creating intelligent 3D avatars that can engage users in natural conversation, from video games to customer service. To make these characters truly lifelike, they need human-like expressions.”
Enter Nvidia’s Audio2Face. “Audio2Face accelerates the creation of realistic digital characters by providing real-time facial animation and lip-sync driven by generative AI, Nvidia says.”
In short, by open sourcing Audio2Face, Nvidia aims to “accelerate adoption of AI-powered avatars in games and 3D applications.”
Part of Nvidia’s broader ACE platform, which is all about creating more convincing digital human avatars, our own Jacob R sampled Audio2Face last year and came away impressed. Fuelled by some LLM-generated responses, Jacob found the end result, “frighteningly good.”
The only really obvious giveaway that you’re dealing with an early, experimental system is the slight delay in responses, which resulted in “awkward pauses” during conversation.
In terms of what’s being released in open source form, we’re talking the Audio2Face SDK, audio plugins for inputting voice streams, training frameworks, sample training data, a library of facial models and a specific Unreal 5 Engine plugin. The open-source release also includes Audio2Emotion Models, which can “infer” emotional state from audio in real time.
Nvidia highlights that among game developers who already use Audio2Face are Codemasters, GSC Games World, NetEase, and Perfect World Games, while ISVs include Convai, Inworld AI, Reallusion, Streamlabs, and UneeQ.
But it’s worth noting that Nvidia’s broader ACE platform is, inevitably, tied at least to some extent to Nvidia’s GPUs, although there aren’t any obvious reasons why ACE features shouldn’t operate on non-Nvidia GPUs. As is often the case with exciting technology from Nvidia, part of the motivation behind its development seems to be to encourage gamers to stick with Nvidia GPUs.
