Engine version: 5.5 - 5.6

Advanced Real-Time Lip Synchronization

Runtime MetaHuman Lip Sync (AI for NPCs) (+ CC4, Genesis, ARKit, and more) provides a powerful solution for bringing digital characters to life. This tool enables zero-latency, real-time lip synchronization for MetaHumans and custom characters alike. Furthermore, the system operates completely offline, ensuring privacy and reliability without requiring an internet connection. Therefore, developers can create immersive and believable conversations with minimal setup time.

Three High-Quality Animation Models

The developer has included three distinct models to suit various project requirements. Each model offers a different level of fidelity and compatibility. Specifically, these models include:

Mood-Enabled Realistic Model: This version provides emotion-aware facial animation specifically for MetaHuman characters. It features 12 different moods to enhance storytelling.
Realistic Model: This model focuses on enhanced visual fidelity for MetaHumans. It utilizes 81 facial controls to achieve natural mouth movements.
Standard Model: This model offers broad compatibility. It works with MetaHumans and various custom characters using 14 visemes.

However, users should note that the Standard model requires a small extension plugin for full functionality. These options allow creators to balance performance and visual detail based on their specific needs.

Broad Compatibility and Customization

This plugin is designed to be versatile and works with a wide range of character systems. Beyond MetaHumans, it supports Reallusion CC3 and CC4, Daz Genesis 8 and 9, and Mixamo. Additionally, it is compatible with FACS, ARKit, and custom morph targets. Consequently, developers can apply high-quality lip sync to almost any character rig. Moreover, the tool supports dynamic laughter animations detected directly from audio cues.

Technical Features and Platform Support

The system is highly optimized for real-time performance across multiple environments. It supports Windows, Mac, iOS, Linux, Android, and Meta Quest. One standout feature is the support for Pixel Streaming. This allows for live lip sync from browser-based microphone input. Additionally, the plugin handles multiple audio sources, including live microphone input, audio files, and synthesized speech. Therefore, it is ideal for complex AI chatbot integrations and interactive NPCs.

Integration and Workflow

For those looking to build advanced workflows, this resource integrates seamlessly with other tools from the creator. For instance, it works alongside the Runtime Audio Importer and Runtime Text To Speech plugins. While these additional plugins are required to replicate specific demo projects, developers can also implement their own audio input solutions. The demo projects provided by the developer showcase full speech-to-speech workflows, including AI chatbots and speech recognition in Unreal Engine 5.6.

Conclusion

Runtime MetaHuman Lip Sync (AI for NPCs) (+ CC4, Genesis, ARKit, and more) is a comprehensive solution for real-time character animation. By offering offline processing and cross-platform support, it provides the flexibility needed for modern game development. Whether you are using MetaHumans or custom CC4 characters, this plugin ensures your NPCs communicate naturally and emotionally.