Interactive Virtual Human Streamer

About project

Introduction:

I present this groundbreaking project that seamlessly integrates Twitch, Unreal Engine’s MetaHuman, Edge-TTS, RVC, and OpenAI’s GPT to create an unparalleled immersive digital experience. The objective was to create a virtual human, live stream it into Twitch and make it respond chat messages using a custom voice. An andalusian wise fortune teller was selected as the personality for the the meta human, we thought it could be fun to have a livestreaming guessing people’s future, and it was.

Technologies:

• Unreal Engine was used to build the whole project. We also used MetaHuman Creator to create the physical appearence of our virtual streamer. It was fully created using blueprints. Some plugin were used to achieve the results:
  • TwitchInteractionUE provided the logic to connect to twitch services and read chat messages.
  • Rama’s Victory BP Plugin provided blueprints to launch and manage external processes from Unreal Engine.
  • Fetch allowed us to create HTTP requests to send prompts to GPT and receive responses.
  • Nvidia Omniverse Audio2Face was used to animate avatar visomes (speaking) based on audio in realtime.
• Blender as 3D model tool. Used to edit metahuman’s clothes.

• Text-to-Speech (TTS) with Edge-TTS and RVC: Edge-TTS allwed to produce TTS with good quality results without paying a cloud service provider. RVC-WebUI was a surprising easy way to generate custom voice models from existing audios and also offered a infer tool that allow us to adapt edge-tts audios to our custom voice model.

• AI Responses with OpenAI GPT: Azure OpenAI services were used instead of official OpenAI API. It was consumed by HTTP requests from Unreal Engine. It works really good when offering it a solid context and a good example of how-to responses should be.

Key Features:

• Real-time virtual avatar reactions using Unreal Engine’s MetaHuman.
• Quite natural and expressive Text-to-Speech synthesis with Edge-TTS and RVC.
• Intelligent and context-aware AI responses powered by OpenAI’s GPT.
• Immersive viewer engagement through live Twitch streaming.

Conclusion:

Embark on a journey into the future of digital engagement with this immersive project. By combining the power of Twitch, Unreal Engine, Edge-TTS, RVC, and OpenAI’s GPT, we have created an experience that shows what is easily feasible nowadays in the world of live streaming, avatars, and artificial intelligence.