Hi everyone, I'm Gary Caogan, a self-media newcomer who has been exploring AI video and content globalization for two years.
On the evening of October 15th, I stumbled upon a livestream room called "Tiangong Kaizhen" (Author: Tiangong Kaizhen, GitHub: https://github.com/beiyege-01, X: @yijt1339303). The author had been streaming for a while, and now every time they go live, thousands of people are online. There have also been many reposts and comments about this livestream room on X.
Typical digital human livestreams often have stiff movements and voice, with pre-set scripts. For better effects, you usually need a real person driving it in the background. But why did this livestream room break out? Because the effect is indeed far superior to previous digital human livestreams. There is no real person driving it behind the scenes; the reactions are real-time. When viewers send a message, she replies. Movements, expressions, and interactions are all generated on the spot. My first reaction was that the imagination potential of this solution is huge:
Game streaming, e-commerce streaming, entertainment streaming, companion streaming—basically all scenarios can be deeply combined with this solution. So the question arises: how did he implement it? Can I build such a livestream room myself?
After four days of research and polishing, my avatar "Wanwan" has already officially streamed on my Douyin account. We briefly restarted the service twice during the stream, totaling over 70 minutes. What score do you think this effect deserves?
Some screen recordings:
https://ncnmu5q1eqm9.feishu.cn/file/Z5Spb8GIlog8qlxaWWKcGB7Xnbg
Okay, so this article will share how I built and tuned the AI real-time livestream host Wanwan from 0 to 1 in these few days, including how I broke down the solution, how I implemented it, what pitfalls I encountered, and how I solved them.
At the same time, the entire AI real-time livestream tool framework has been open-sourced under the MIT license. Address: https://github.com/garycaogan/ai-live-avatar.
This article was completed by me using the previously open-sourced content workbench and Claude.
1. Breaking Down the Original Author's Livestream Room
This is the original author Xiaoya's livestream room.
Break it down into inner and outer layers.
The inner layer is the host herself. Movements, expressions, and speech are all generated in real-time. There is a Vidu S label hanging above the screen, which makes it easy to understand that the model is Vidu S2, Vidu's latest digital human model. This basically rules out various guesses about local deployment of open-source models.
The outer layer is the circle around the host. The instruction card says "Continue testing bullet comment commands today," and below there is a column for singing, dancing, selling goods, and interaction filtering, some checked and some empty. This means that after bullet comments come in, there is a program in the middle filtering them, turning them into commands, and then handing them to the digital human to act. This layer is a tool made by the author himself.
For the inner layer, Vidu's digital human solution has three paths: one is real-time digital human, where face, voice, and responses are all handed to it; one is preparing the audio and feeding it in, letting it only handle lip-syncing, which can even sing; and one is using a real-person video to drive the digital human, replacing the person inside.
Talk is cheap. Overthinking these technical evaluations is meaningless, so I immediately got to work and quickly ran through a demo. Based on my familiarity with YouTube, I first connected the loop of livestreaming on YouTube and reading bullet comments.
Interested parties can watch the replay: https://www.youtube.com/live/fG84tzLOxLc
But the effect of the first livestream was really poor. My effect wasn't much different from conventional digital human technology; movements were robotic, replies were robotic, and voice was robotic. If Xiaoya's recording gets a 9, scoring my early attempts across the three routes resulted in failing grades.
Comparing the three routes, the replacement route has the most natural movements but cannot accept bullet comments in real-time. The audio-feed route requires building another set of responses and dubbing. Combined with the Vidu S label on the screen and her speed in responding to bullet comments, I judged that Xiaoya took the real-time route. So the gap is not in the path, but in the base image, voice, and persona.
What makes Xiaoya feel like a living person, I think, is mainly the following points:
First, movements. She spends most of her time looking down, as if reading bullet comments on her phone, and smiles when she sees something interesting, showing teeth. Her hands sometimes rest on her chin, sometimes reach for snacks. She looks very much like an ordinary girl coming home from work to start a livestream, rather than a "digital human" standing upright reciting a script.
Second, visuals. I initially suspected the whole screen was Photoshopped with a hazy filter. Later, zooming in on screenshots revealed that the edges of the text overlaid in the frame were sharp, and the small window of the host was clear. Only her face was soft. So this haziness is not a post-production filter but inherent to the digital human screen: a soft light hits only the face, surroundings are darkened, skin looks white, pink, and flat like a beauty-filtered mobile livestream. Placed in the small horizontal frame on a phone, details are shrunk away, hiding the AI's flaws. This effect resembles the lighting of a real host.
Third, rhythm. Subtitles are pressed at the bottom of the frame, replying to a bullet comment roughly every 20 seconds. It sounds slow, but this slowness just covers the few seconds AI takes to react. Real hosts don't reply instantly either, right?
Fourth, real-person window. There is a living person in the corner. The entire livestream room doesn't look like a hung machine. When chatting with her, viewers know someone is watching behind the scenes.
Breaking it down here, the optimization path is clear. Use Vidu's existing solution for the host layer. What I need to do is the outer layer: landscape mode, instruction cards, real-person window, etc., learning from Xiaoya's setup.
2. Vidu Handles the Host, I Build the Floor Control Layer
To use a plain analogy: In a real livestream room, next to the host there is usually a floor control assistant who watches bullet comments, kicks out advertisers, passes viewer questions to the host, reminds her to change clothes, and offers topics when things get quiet.
In Wanwan's livestream room, the host role belongs to Vidu. What Wanwan looks like, how she opens her mouth, and what she says are all generated in real-time by Vidu.
So the key to this real-time livestream project is the floor control layer, specifically involving these tasks:
- Filtering: Vulgar or advertising bullet comments are simply not read aloud.
- Queueing: Bullet comments come one by one. If needed, logic can be added for gift-givers to jump the queue.
- Understanding intent: Distinguishing whether a comment is chatting, requesting a clothing change, asking for a prop, changing background, or asking for a heart gesture or wave.
- Presetting: For example, the wardrobe logic. Each outfit is a complete photo. Changing clothes means switching to a new photo and restarting the session.
- Idle commands: Vidu supports idle voice. To prevent the host from just staring blankly when there are no comments, topics for idle times can be set.
- Instruction cards and labels: Learned from Xiaoya's room. Write on the screen what can be done, what cannot, and how to send comments.
- A local workbench: Change personas, manage wardrobes, switch voices, adjust actions—all via web clicks.
- Livestream configuration: I used OBS to combine the digital human screen, instruction cards, and my camera feed, then pushed it horizontally via Douyin Live Companion. My camera is placed in the bottom left corner.
All these logics were created through rounds of testing, discussion, and optimization with Claude. He handles tech and product; I provide opinions and requirements.
AI can build the first version overnight, running through clothing changes, queueing, and pushing streams. But it cannot hear the robotic tone in the voice, nor does it know how awkward it is to loop those three or four sentences during silence. You only feel this by actually watching the character interact with you in the livestream.
Regarding costs, they are mainly concentrated on Vidu; others are negligible.
Vidu's real-time digital human is charged per second, 1.5 credits per second, converting to about 169 RMB per hour. Tonight's stream lasted 1 hour and 7 minutes, costing 191.47 RMB in total, including pre-stream warm-up, sessions restarted for clothing changes, and mid-stream restarts. For me, 100-200 RMB per hour is acceptable during testing, but the cost is there. If you want to monetize with this solution, calculate your ROI clearly.
3. Pitfalls Encountered These Few Days
Appearance (Basically the Base Image)
The first successful run was Tasha's CG version. Tasha is a character from the game Ace Combat 8. Who doesn't like pink hair? But the digital human felt heavy in the stream because the photo was CG-style, far from realistic.
Then came the real-person version of Tasha, drawn by an image generation model into a realistic photo. It was better. I also made a horizontal composition for her mimicking Xiaoya's setup, sitting at a desk resting her chin, with a lamp nearby. But it still lacked something, feeling obviously like an AI face.
When switching to original characters, I wanted more life-like, ordinary roles. I made four girls with accents: Xiaoman from Northeast China, Chengzi from Changsha, Awen from Minnan, and An'an with a Taiwanese accent.
The problems with these were mainly aesthetics and base images. Comparing Chengzi's base image with Xiaoya's screen, the skin tone was too yellow and saturated, texture too detailed, camera angle eye-level frontal, arms spread like an ID photo. Persona issues also arose: I wrote sample lines in the persona, and she read them verbatim.
Finally, frustrated, I looked for inspiration from blogger Doumiao's X posts (@nima2qisi2), using prompts to generate a similar version, which became Wanwan.
The persona is a 24-year-old girl working in graphic design at a small company in Hangzhou. She comes home past 10 PM, starts a livestream to chat, has a black cat named Meiqiu (not shown, but she complains about its mischief). Personality is slow-to-warm, slightly dry humor. When asked if she's real, she directly says she's an AI host.
Changing Clothes
Initially, I tried letting AI modify clothes on her body directly. Once was okay, but after twice, the face started breaking. Later, I switched to making a complete photo for each outfit beforehand, including person, clothes, and room. Changing clothes meant restarting the session from that photo, which was faster and more stable. The wardrobe now has 7 outfits; the black turtleneck is default. One change takes about 5 seconds.
This logic can also switch scenes and actions, provided enough preset base images are prepared.
Voice Tone
Initially, I made several accented characters (Northeast, Changsha, Minnan, Taiwanese) because I found accents cover up AI vibes. A girl speaking with an accent feels subconsciously like a living person. A small incident occurred: an AI listened to a voice twice blind and said it was standard Mandarin, but I heard a Northeastern twang immediately. Since then, I pick voices myself.
Later, I found a critical issue: many voices sound like real people alone, but lose their charm in real-time livestreams.
Wanwan's final voice is one I designed on Volcano Engine, providing raw audio material to clone onto Vidu.
Repetitive Idle Voice
After the first 6-minute test stream, my biggest takeaway was that when no one commented, she looped the same three or four warm-up phrases. It was awkward even for me. Before the second stream, I quickly prepared 100 cold-start topics based on her persona. Repetition only happens after all are exhausted.
Background Change Removed Clothes
Tonight she changed backgrounds 9 times. The first 8 were fine. At the end, wearing a black camisole, changing from room to snowy cabin kept the clothes (see image below). But the 9th change, from snowy cabin to city night view, resulted in missing shoulder straps and top, looking naked. I ended the stream immediately.
Investigating the cause: Clothing changes are stable because each outfit is a pre-made complete photo, restarting the session. Background changes and props take another path: handing a reference image to Vidu during the stream to redraw the scene. But her current outfit wasn't included in the reference, nor did we ask it to preserve clothes. Vidu could redraw her body. First 8 times it drew correctly; the 9th time it erased clothes. The scariest part is inconsistency. Testing multiple times without issues doesn't mean safety. This is a confirmed vulnerability. Whether it's the sole cause is uncertain, but the hole must be plugged. We discovered earlier that modifying reference images mid-stream causes drift and irreversibility, but didn't expect it to happen in a public stream.
Now, background changes and eating/drinking props are disabled by default. Instruction cards move them to the "cannot do yet" section until we switch to "restart from full photo" or add visual checks.
Vidu Disconnection, Real Person Holds On
Vidu disconnected once for about 3 minutes. During that time, Wanwan disappeared from the screen, leaving only me in the bottom-left window. Comments flooded with "Why did it end?" "Gary is gone." This shows the real-person window isn't just for presence; when things break, someone must hold the fort.
Handling Bullet Comments and Gifts
Bullet comments currently use a temporary method reading webpage comments.
Gifts initially had queue-jumping logic, but I removed it because Douyin counts it as inducing tipping. Technically, gift-based queue jumping is feasible.
Unresolved Issues
- Lip-sync singing and large movements are temporarily impossible.
- Even with persona and topic presets, responses sometimes retain AI vibes.
- From viewer comment to her speaking takes 10-20 seconds.
- Long sentences get interrupted by new dialogue or action commands, though optimizable programmatically.
- Memory is short. After reconnection, when asked why it stopped, she replied, "It didn't stop, aren't we still chatting?"
4. Development Potential of This Gameplay?
I'll mention directions I'm familiar with and visualize well. Obvious ones like e-commerce hosts won't be discussed. Future explorations may follow these paths; results will be shared. Readers can check for intersections with their expertise:
1. Game Streaming. As a gamer with 20+ years experience, I have affection for game streaming. I've streamed Honor of Kings and LOL before. Have you seen Bilibili's "Doubao Teaches Games" series? Playing Delta Force, LOL, Minecraft, many went viral. E.g., Doubao teaching Minecraft hit 16 million plays. Imagine if Doubao had a real-time interactive, good-looking character. Videos and streams would be even better.
- Infinite Loop Streaming. MiniMax previously did a real-time generated screen stream, feeling like infinite loops. But lacking a defined theme, it didn't run within a framework. Bridging Vidu's solution with an outer shell, generating scene images and dialogue in real-time based on comments, likely enables interactive plot streaming. Real hosts wouldn't dare imagine this. Picture a stream where the main line is the host completing a trip: transport, companions, encounters, words, choices—all directed by comments. Isn't that interesting?
3. Entertainment Streaming. No explanation needed. Xiaoya's case proves potential in entertainment hosting. I highly suspect soon, after similar models open-source, adult platforms will be occupied by AI.
Boundaries: Douyin requires labeling AI-generated content, mandates real-person appearance and interaction, discourages unmanned/hanging streams. So Wanwan's screen always displays "AI Real-Time Generated Screen," and my camera stays in the bottom left. Tonight's crash is a reminder. Among 18 filtered comments, several requested swimsuits, bikinis, slip dresses, blocked by keywords (keywords also falsely flagged "someone is slacking off" as suggestive, later fixed). Yet the accident wasn't from these, but a normal background change.
Text can be blocked by keywords; screens cannot. AI-generated visuals can unexpectedly produce platform-prohibited content. Without supervision, it might create violations. Real-person presence isn't just a platform requirement; it protects you.
Those hoping for 24-hour passive income with digital humans still have distance to go.
Also, don't use celebrities, others' characters, or faces for digital humans. Making your own isn't hard.
6. Open Source Notes
This project's tool framework is open-sourced under MIT license: https://github.com/garycaogan/ai-live-avatar.
Includes bullet comment filtering/queueing, intent recognition, wardrobe, instruction cards/screen overlays, local workbench, OBS/Douyin horizontal push integration. You can dry-run without spending money to see filtering, queueing, and overlay implementation. Webpage scraping part is excluded due to risk; implementers should add it themselves.
Tools are available to all, but characters must be yours. Swap in your face, write your persona, and it runs.
If you want to build one, take the code, make your character, and welcome comments for help or sharing successes.
I'm Gary. If this article helped, remember to triple-click!
