Plugins + Scripts VaM ControlNet Capture

[harry kant]

harry kant submitted a new resource:

VaM ControlNet Capture - Session plugin for exporting matching Beauty, Depth, and OpenPose ControlNet images from

VaM ControlNet Exporter

VaM ControlNet Exporter is a Virt-A-Mate 1.x session plugin for AI image generation workflows, especially with ComfyUI and ControlNet.

It exports matching reference images from a selected WindowCamera: Beauty render, Depth map, and OpenPose / COCO-18 pose image.

All exports use the same camera and resolution, so the generated Beauty, Depth, and Pose images line up correctly and can be used together in ControlNet-based workflows...

Read more about this resource...

 

[rabbit1qaz]

im not into AI but intrested in start to using it, what could you achieve in the end with that? could you post some end images/movies with how was it in vam and endresult

 

[harry kant]

im not into AI but intrested in start to using it, what could you achieve in the end with that? could you post some end images/movies with how was it in vam and endresult

Hey, good question!

What is ControlNet?

In Stable Diffusion (A1111 / ComfyUI) you normally only have a text prompt. ControlNet adds an extra image to control how the image is generated.

This plugin exports some crutial control images directly from VaM:

• Beauty - the normal VaM render... you can make edge pictures in Comfy UI via Preprocessor very easy
• Depth - a grayscale image where white = close, black = far. This keeps the exact body shape, pose and camera distance from VaM in the final AI image.
  I had to build a custom render pass for this, because raw depthTexture in VaM misses hair (transparent queue, ZWrite Off) and clothing (cutout queue) - they come from different meshes.
• Pose / OpenPose - the skeleton of your Person atoms. Keeps arms, legs, hands and face position.

What can you achieve with it?

VaM is perfect for posing, but you want a different style in the end. You pose everything in VaM, export Depth + Pose with this plugin, then load those into ControlNet in ComfyUI and generate the final image with any model you like
- same pose as in VaM, but with a completely different look, cartoony or photorealistic.

I will post some end results (VaM viewport vs. final SD result) when I am home, and also include example workflows for ComfyUI.

How and if the ControlNets work depends a lot on the model you use

More details here: ControlNet on GitHub
Good intro for ComfyUI: ComfyUI ControlNet Examples

 

[harry kant]

harry kant updated VaM ControlNet Capture with a new update entry:

Depth Capture and Preview Fixes

Improved the Depth capture pipeline after version .4.var: active Person bodies, clothing, and hair are now rebuilt from VaM’s current GPU buffers so they better match the Beauty render. Preview timing was also adjusted to reduce duplicate/old-pose Person artifacts, UI layers are excluded from ControlNet passes, and Beauty preview/export now use the same selected active WindowCamera more consistently.

Read the rest of this update entry...

 

[harry kant]

NOTE: Update uploaded and pending moderator approval - not live yet.
Improved Depth capture pipeline:
- Active Person bodies, clothing, hair now rebuilt from VaM's current GPU buffers (matches Beauty)
- Fixed preview timing to reduce duplicate/old-pose artifacts
- Excluded UI layers from ControlNet passes
- Beauty preview/export now use same active WindowCamera consistently

 

[harry kant]

harry kant updated VaM ControlNet Capture with a new update entry:

Depth Ghost Figure Fix and Developer Documentation

Fixed ghost/duplicate figure artifacts in the Depth pass by improving the capture timing and VaM-specific proxy handling, while also adding clearer C# XML documentation for developers.

Read the rest of this update entry...

 

[mightkc]

This is really cool. @harry kant , what do you think would be necessary to put together a plugin that takes video content controlnet exports from AI into animation ready data for VAM characters?

 

[VaMStaff]

There has been some misunderstanding about what this plugin actually does, and so the moderation team feels the need to comment here publicly.

This tool aids in AI static image generation (and possibly video) using already established workflows. It does not create 3d assets, morphs, textures, or any other form of content that can be loaded into VaM.

If it did create morphs, clothing, textures, etc, it would not be allowed as it would violate our licensing policies. As long as the sole purpose is to aid in image/video generation, there is no problem downloading and using this plugin for your own personal use.