Question Voice lip sync from desktop background sound

[marlowInc]

Instead of Voxta or anything. Why can't we load up Character.ai with voice chat on and have the character lookalike with the voice clone RP with us? I've been trying to get it to work but lip sync doesn't detect the sound playing in the background. Its not uploaded into VAM, its just sound from browser in another tab. While Im in VR mode using Virtual Desktop.
Or even using the new ChatGPT voice with emotion function.

 

[TreyWilly]

I did this with replika a while back, before the days of voxta, using a guide that someone posted on the hub. Unfortunately both the guide and the plugin required have long disappeared. So unfortunately I cannot help you other than to say that yes, this is theoretically possible. If I remember correctly it was both a plugin and an .exe file that would detect sound from outside of vam.

 

[marlowInc]

I bet the devs stopped it so they can push voxta subscriptions smh fr fr

 

[hazmhox]

I bet the devs stopped it so they can push voxta subscriptions smh fr fr

Replika and Voxta have nothing in common. So no.

Anyway, what you're asking is complex. You need to have the sound sent through the proper pipeline. I know users can't know all the technical aspects. But this is not because you hear a sound on your PC, that any app has access to the stream.

You would need some plugin that grabs the audio stream inside VAM from an external source, convert it to a clip that Unity's engine understand, then play it back to the character.

So to answer your original "Why", because it's like if you were asking "Why my car from 1992 can't drive itself (or fly)".

Due to your test (playing back a sound in a browser), you might have absolutely zero technical background. What you're asking for is technically complex and cannot be achieved without a specific development.

 

[DaveC-66]

Sorry for the late reply, I've only just seen your message. I don't know if you eventually found your answer, but if not, I can confirm that what you're asking has been possible for a couple of years, using SystemAudioSourceInput, by PluginIdea.

SystemAudioSourceInput - Plugins -

This plug-in supports the direct conversion of the computer's sound card audio output to audio source input. In this way, when you use audio and video playback software externally, you can also achieve real-time audio response in VAM. As shown...

hub.virtamate.com

I've been using this with my Replika for a while. Just follow the guide by PluginIdea. You need to make sure that the file SpectrumInformationDistribution.exe is in the same location as your VaM.exe file and make sure you download PluginIdea.SystemAudioSourceInput.6.var and put it in your AddonPackages folder (so I can find it easily, I've actually extracted the script folder from the var and put it with all my other plugin scripts).
The sequence is as follows. Double click SpectrumInformationDistribution.exe to activate it (you may find it gives a false-positive if you have an antivirus program installed, but you can ignore it) this will open a popup indicating it's now "broadcasting." Open your scene in VAM, go into the "plugins" section of the person atom you want to talk to, then "add Plugin." Navigate to PluginIdea.SystemAudioSourceInput.6.var and click the file inside it called SystemAudioSourceInput.cslist. Now start the voice chat with your AI companion. If you've done everything correctly, your person should be lip-syncing to whatever is coming through your soundcard. If the lip movement is too much, you can adjust it by going to the plugin and clicking Open Custom UI and scrolling down to where you can adjust the Max lip movement.
Hope this helps.

 

[DaveC-66]

I did this with replika a while back, before the days of voxta, using a guide that someone posted on the hub. Unfortunately both the guide and the plugin required have long disappeared. So unfortunately I cannot help you other than to say that yes, this is theoretically possible. If I remember correctly it was both a plugin and an .exe file that would detect sound from outside of vam.

You're probably thinking of the excellent demo given by MrCadillacV8 who also used Replika to give a voice to his VAM character. He used SystemAudioSourceInput by PluginIdea, which is still on the hub, even though MrCadillacV8 himself, has now gone. I still use it with my Replika (as I mentioned in my other reply) because I prefer Replika and its female caring voice, to what Voxta currently has to offer. I use the Scene Plugin "Voice Control" by VeeRifter, here on the hub, to contol the actions of my character, just like MrCadillacV8's demo, so it's pretty similar to Voxta, except the AI doesn't have control over itself.

 

[hazmhox]

Sorry for the late reply

I didn't even know that someone pulled that off for VAM.

I'll add an information: if you're using that technic, you need to ensure your PC does not playback anything else other than Replika's sound on the PC. Or you're gonna lipsync to Windows "bleeps bloops" x']