Plugins Realtime Lipsync

[AdamAnt5]

AdamAnt5 submitted a new resource:

Realtime Lipsync - Automatically create realistic lip animations with any voice clips

This is the VAR version of my RT_Lipsync plugin updated for VaM v1.20.0.7

RT LipSync is a real-time lip-syncing script for VaM. After attaching the plugin to a character, any speech playing through the character's head audio source will produce a lip-syncing animation.


After adding the "RT_LipSync.dll" plugin to a character, two plugins will appear in the plugin menu:

RT_LipSync: Controls the lipsyncing performance. Tweak settings based on your preference

RT_LipSync_Player...

Read more about this resource...

 

[meshedvr]

Could you just upload the var file directly here to the Hub? That would avoid having to go through Mega and also from having to deal with the zip file.

Thanks!

 

[AdamAnt5]

AdamAnt5 updated Realtime Lipsync with a new update entry:

2.2e

Direct link to VAR file.

Read the rest of this update entry...

 

[AWWalker]

breaks the atom character when i run it... can no longer move character, just to the hair moves... the first 3 seconds are great though

 

[FLC2312]

After adding the plugin to a character (VAM Version 1.20.0.10) I do not get any options when clicking the "Open Custom UI" button. Not even in the demo. What am I doing wrong? I really loved this plugin when using earlier version of VAM...

For those having the same issue: I resolved it by creating a clean install of VAM rather than updating the current install...

 

[SpiltSugar]

Is it possible to see the ability to work with a live microphone as input on a future iteration of this plug in? I love it already!

 

[furifuri]

The following file was found to interfere with "AdamAnt5.Realtime_LipSync.1.var" if it was in the morph folder.

VSMAA.vmb
VSMEH.vmb
VSMER.vmb
VSMF.vmb
VSMIH.vmb
VSMIY.vmb
VSMK.vmb
VSML.vmb
VSMM.vmb
VSMOW.vmb
VSMS.vmb
VSMSH.vmb
VSMT.vmb
VSMTH.vmb
VSMUW.vmb
VSMW.vmb

This is probably the data converted from daz.
If you remove this, "AdamAnt5.Realtime_LipSync.1.var" works fine.

 

[Acosco]

Hey, love the plugin, I'm using for this experience I am making and while it works beautifully in VR, I am finding that the plugin stops working while using the Windows Camera. Is there a fix to this that is easy?

 

[8Ball]

@AdamAnt5 and developers:

Did you evaluate or consider what worksteps would be needed to include Oculus Lipsync or Unity packages like
Lipsync Pro oder Salsa Lipsync to work within VAM?

 

[SpiltSugar]

Is it possible to see the ability to work with a live microphone as input on a future iteration of this plug in? I love it already!

LFE iterated on this if you have not seen it yet op. its buggy so we'd surely love your take on how to fix that up to work more reliably if you want to.

 

[jillian]

breaks the atom character when i run it... can no longer move character, just to the hair moves... the first 3 seconds are great though

my whole scene freezes and only hair seems to be active - no sound or lipsynch. What am I doing wrong? I had this working on older VAM version.

 

[beboppo]

AdamAnt5 submitted a new resource:

Realtime Lipsync - Automatically create realistic lip animations with any voice clips



Read more about this resource...

Hey. I love the idea of this plugin. but i have no idea how to make it work.
the instructions say "After attaching the plugin to a character, any speech playing through the character's head audio source will produce a lip-syncing animation." but i cant attach it to a character at all.

It says "After adding the "RT_LipSync.dll" plugin to a character"
but I downloaded a .VAR file not a.dll file... where do i find the RT_LipSync.dll?

I'm completely lost. when i select plugins and try to add the RT_LipSync.dll or AdamAnt5.Realtime_LipSync.1.var I can't find them anywhere. I think i placed the "AdamAnt5.Realtime_LipSync.1.var" file in the correct folder, but i am not certain. i'v looked for a simple tutorial on how to make this work for hours but they all start with "After attaching the plugin to a character, ...." how do even locate the plugin? what are the steps leading up to being able to attach this plugin to a character?

I'm assuming i havn't put the var file in the right location. but why does it mention a dll file when i only downloaded a VAR file?

lol in conclusion. i have managed to download AdamAnt5.Realtime_LipSync.1.var can somone please provide me a step by step from download to actually making it work?

Thanks in advance.


What are the odds someone could walk me through the entire process start to finish. from download to making a model lip sync to an audio file i have on my hard drive?

 

[bestra99]

I thnk the new update may have disabled this. Any chance of an update? Its such a good plugin.

 

[meshedvr]

There will be another VaM patch coming out to allow this plugin to work. I had to open up a couple of small things that were restricted in the new update that only applied when the plugin was in dll format. The patch should be out later today.

Sorry for inconvenience.

 

[AdamAnt5]

Thanks, I will take a look this weekend if there are still any problems.

 

[bestra99]

Thanks for the prompt replies, much appreciated!!

 

[meshedvr]

It is working now with 1.20.77.8 patch. Thanks!

 

[jillian]

Has the VaM 1.20.77.9 messed up lipsynch? I am getting a total freeze of the main character and a sort of ghost hair cloth overlay. I get the sound to play but the character frozen. Any hints or ideas on what's wrong?

 

[jillian]

It is working now with 1.20.77.8 patch. Thanks!

Has the VaM 1.20.77.9 messed up lipsynch? I am getting a total freeze of the main character and a sort of ghost hair cloth overlay. I get the sound to play but the character frozen. Any hints or ideas on what's wrong?

 

[Wakko]

I am finding that the plugin stops working while using the Windows Camera. Is there a fix to this that is easy?

Has anybody addressed this? WindowsCamera doesn't seem to see RT Lipsync. I am so disappointed. I was working on a dozen scenes where I assumed I could use WindowsCamera, now project is in flames.

Perhaps some hopes, tho. I've discovered that RT Lipsync works with CAMRIDE scenes from C&G Studio. If I can figure out how they animate that, it would actually be better than WindowsCamera in that it would work in VR as well.

 

[AdamAnt5]

Has anybody addressed this? WindowsCamera doesn't seem to see RT Lipsync. I am so disappointed. I was working on a dozen scenes where I assumed I could use WindowsCamera, now project is in flames.

Perhaps some hopes, tho. I've discovered that RT Lipsync works with CAMRIDE scenes from C&G Studio. If I can figure out how they animate that, it would actually be better than WindowsCamera in that it would work in VR as well.

Are you referring to the in-game window camera?

 

[Wakko]

Are you referring to the in-game window camera?

Yes. The in-game WindowCamera atom doesn't seem to "see" this plugin on a person atom. Make WindowCamera active, no lipsync...deactivate it, lipsync works fine.

 

[nutcracka99]

Yes. The in-game WindowCamera atom doesn't seem to "see" this plugin on a person atom. Make WindowCamera active, no lipsync...deactivate it, lipsync works fine.

I noticed this as well! The interesting thing is if you have the camera and are watching it on the DisplayControl, it will show the lipsync! It's only if you switch to the WindowCamera (using "m" in desktop mode) that it stops working. Literally it can be in the middle of the sync, lips moving and everything, and then when you switch to the WindowCamera view it stops. I solved this by adding AcidBubbles's Passenger plugin to the WindowCamera control, setting the keybinding for switching into passenger view, and then using that to pop my view into the camera. A bit clunky but made it possible to record the video in OBS.

 

[MasonJ]

Whenever I attach the script I get this error:
Exception during plugin script Init: System.MissingMethodException: Method not found: 'URLAudioClipManager.QueueClip'. at MVRPluginManager.CreateScriptController (.MVRPlugin mvrp, DynamicCSharp.ScriptType type) [0x00000] in <filename unknown>:0
There are no controls in 'Open Custom UI' section of the plugin either. I'm running vam version 1.20.77.9 (The current version as of writing this). Any ideas on what's wrong?

Edit: The plugin works fine on a fresh install of VaM. Are there any known incompatibilities?

 

[clevenger]

I am having a different problem. On vam 1.20.77.7 if I try to load the DLL on the character I get this

!> Exception during compile of Custom/Scripts/RT_LipSync/RT_LipSync.dll: System.Security.SecurityException: Security Check Failed (NamespaceRestriction) : [RT_LipSync_v2, RT_LipSync_v2.dll] : The namespace 'System.Runtime.InteropServices' is prohibited and cannot be referenced
at DynamicCSharp.ScriptDomain.SecurityCheckAssembly (System.Byte[] assemblyData, Boolean throwOnError) [0x00000] in <filename unknown>:0


I can load the .cs fine and it detects the sound files and plays them, but no lipsync.

In user preferences I have all the "enable" boxes checked.