VaM 1.x VaM Character Generator

Thanks, everybody. All very relevant and helpful comments.

To understand the true limitations of the current approach, I'm running a brand new capture set of 500K samples with a slightly enhanced morph library.
The process is semi-automated and will take about 4 days (and nights) to finish.

Also, the Model has been updated to a Dual KNN architecture.
The morph mapper has been split into two separate models:

• Front model — trained on front-facing captures, handles overall face shape, eye spacing, nose width, jaw, brows, etc.
• Side model — trained on side-view captures, handles depth-related features, nose projection, lip depth, chin projection, forehead slope, etc.

The reason for the split:
A single combined model trained on both front and side features was dominated by the 21 front measurements, effectively drowning out the 7 side measurements.
Separating them gives the side view its own dedicated model with full weight on depth features, rather than competing against the front measurements in a shared feature space.

At inference time, both models run independently, and their predictions are blended, with the front model weighted slightly higher.
This gives the generator a genuine understanding of facial depth rather than approximating it from the front view alone.

The result is noticeably more accurate profile silhouettes.
I should be able to present some examples to you within a week.

VaM Character Generator — Development Update


The project has moved away from random morph training data toward a systematic single-morph sweep approach for calibration.

The old approach generated 360,000 training captures by randomising all morphs simultaneously, then used a KNN model to find the nearest matching VaM face in feature space. The problem: with 687 morphs randomised together, individual morph contributions were buried in noise, correlations were weak, and the KNN was effectively guessing from a haystack.

The new approach sweeps each morph individually through its native range (-1.0, -0.5, 0.0, +0.5, +1.0), capturing front and side photos at each value against a fixed camera. 260 hand-curated morphs. Around 2,600 captures total, completed in under an hour.

This gives something that wasn't available before: a true response curve per morph, exactly how much each morph moves each facial landmark measurement.
From that, a direct lookup can be built: measure a feature in a reference photo, find which morphs control it, and compute the exact value needed.

Why this is better:

• No statistical approximation, each morph's effect is measured directly
• Conflict detection becomes explicit, and morphs that affect the same measurements are identified and ranked by strength
• New morphs can be added incrementally without rerunning the entire dataset
• The resulting mapper is deterministic and fully auditable

The morph list was curated using a custom tool that reads the full VaM morph catalogue, filtered to visible face and head morphs only, with expression, pose, fantasy and internal morphs excluded.

Updates will follow as results improve.

As I said before, the core problem is morph crosstalk. If the morphs are not isolated, every correction can destroy points that were already matched by previous corrections.

With such morphs, KNN can only find the nearest possible approximation inside the available morph space. It cannot produce an exact copy of the photo-derived mesh.

For a proper measurement-based reconstruction, you would need an independent morph basis and a response matrix/Jacobian that describes exactly how each morph affects each measured point. Without that, the solver is always fighting the morph set itself.

btw. when i run training sessions, I use services like RunPod rather than having to wait for days on end, even on my 5090.

LeChuck99 said:

As I said before, the core problem is morph crosstalk. If the morphs are not isolated, every correction can destroy points that were already matched by previous corrections.

With such morphs, KNN can only find the nearest possible approximation inside the available morph space. It cannot produce an exact copy of the photo-derived mesh.

For a proper measurement-based reconstruction, you would need an independent morph basis and a response matrix/Jacobian that describes exactly how each morph affects each measured point. Without that, the solver is always fighting the morph set itself.

btw. when i run training sessions, I use services like RunPod rather than having to wait for days on end, even on my 5090.

Click to expand...

You called it exactly right, and that's precisely where I ended up. Moved away from KNN and correlation-based approaches and built a Jacobian solver using a systematic single-morph sweep calibration.
Each morph individually swept through its range, response curves fitted per feature. The solver handles all morphs simultaneously via bounded least-squares which addresses the crosstalk problem directly.

Still hitting a ceiling on the domain gap between real photo measurements and VaM render measurements, but the output is meaningfully better than KNN ever was.

Good tip on RunPod. I've been running training locally which is painful. Will look into it for the next sweep expansion.

@CraSh.it

Hi.
That makes sense. At that point the main bottleneck probably shifts away from the morph solver itself and into the measurement domain gap.

If the Jacobian is calibrated on VaM-rendered measurements but the target comes from real photo measurements, the solver can only be as accurate as the feature correspondence between both domains. Even a good bounded least-squares solver will converge to the wrong compromise if the photo landmarks and VAM landmarks are not measuring the same effective geometry.

You might need either a render-matched measurement pipeline, stronger normalization between photo and render space, or an iterative render-compare-correct loop where the generated VAM face is re-rendered under controlled camera conditions (for the Landmark) and measured again. Otherwise the Jacobian can be mathematically correct inside VAM but still biased against real photo input.
I think your aproach is very promising.


btw. Runpod is easy to use. There are also youtube tutorials for good setups for different needs. But there are also other service provider like runpod.

LeChuck99 said:

@CraSh.it

Hi.
That makes sense. At that point the main bottleneck probably shifts away from the morph solver itself and into the measurement domain gap.

If the Jacobian is calibrated on VaM-rendered measurements but the target comes from real photo measurements, the solver can only be as accurate as the feature correspondence between both domains. Even a good bounded least-squares solver will converge to the wrong compromise if the photo landmarks and VAM landmarks are not measuring the same effective geometry.

You might need either a render-matched measurement pipeline, stronger normalization between photo and render space, or an iterative render-compare-correct loop where the generated VAM face is re-rendered under controlled camera conditions (for the Landmark) and measured again. Otherwise the Jacobian can be mathematically correct inside VAM but still biased against real photo input.
I think your aproach is very promising.


btw. Runpod is easy to use. There are also youtube tutorials for good setups for different needs. But there are also other service provider like runpod.

Click to expand...

That's exactly the problem, the Jacobian is working correctly inside VaM space but the input measurements from real photos don't land in the same space as the calibration data.

The render-compare-correct loop is an interesting idea I hadn't considered. Generate a result, render it in VaM under the same controlled camera conditions used for calibration, measure it, compare to the target photo measurements, and iterate. It would self-correct the domain gap rather than trying to pre-calibrate it away. Could be slow but might be the most principled solution.

Will look into it. Thanks for the detailed breakdown.

you will made it. i am sure! by the might of grayskull! ehhmm or skynet.

CraSh.it said:

Hey gang
I wanted to give you a sneak peek into something I've been working on for the past few weeks.
The model is still in training — updates and releases will follow as results improve.

"VaM Character Generator" - (Drop in a photo > Get a VaM face)

View attachment 589280


VaM Character Generator analyses a reference photo using MediaPipe's 478-point 3D face mesh, extracts 28 precise facial measurements from both front and side views, and maps them to a VaM appearance preset using a K-Nearest Neighbour model trained on 160,000 VaM renders so far.
Each training sample is a real VaM render with randomised face morphs — the model has seen virtually every combination of face shape the engine can produce.
The result is a .vap preset that loads directly into VaM with matching face morphs, skin tone, lip colour, and eye colour extracted from the reference photo.

This is a face/head generator only.
Body, clothing, hair and expressions are yours to customise. All generated morph values are fully accessible as sliders in the app. You can fine-tune before export.

Stay tuned...

Click to expand...

Thanks for the tool!

You've gotta give him credit. I can't even imagine creating something like that. Looking forward to the finished version.

Hi, is there a github/gitlab/ other alternative repository to follow development, try beta versions or for people contribute? Would love to see the project develop! Can appreciate models may need to be stored via a 3rd party source.

Litim said:

Hi, is there a github/gitlab/ other alternative repository to follow development, try beta versions or for people contribute? Would love to see the project develop! Can appreciate models may need to be stored via a 3rd party source.

Click to expand...

No, not yet.
The current state of development is very turbulent. Once I get a solid footing, I will consider GitHub.

Following closely on the development of this tool, super curious on the training side of things and overall technical approach!

Vholf3X said:

Following closely on the development of this tool, super curious on the training side of things and overall technical approach!

Click to expand...

Stay tuned

any updates?

好东西

zhy00509 said:

好东西

Click to expand...

You are on an english speaking forum, on an english thread. Please use a translation service.

any news, updates?
just curious

This looks like its gonna be better than FaceGen Artist Pro. Waiting for any news.

Tomblack said:

This looks like its gonna be better than FaceGen Artist Pro. Waiting for any news.

Click to expand...

It's unlikely.. FaceGen was developed by an entire company and the model was configured not just on photographs, but on tens of thousands of real-life simulated faces. The technology was made without the help of AI and, as far as I know, has only one modern competitor. - Face Transfer 2. all other generators work in unity or fbx formats.

Demorfeus said:

It's unlikely.. FaceGen was developed by an entire company and the model was configured not just on photographs, but on tens of thousands of real-life simulated faces. The technology was made without the help of AI and, as far as I know, has only one modern competitor. - Face Transfer 2. all other generators work in unity or fbx formats.

Click to expand...

You gotta explain to me why "an entire company with tens of thousands real life simulated faces" succeeded to make a software outputting these abysmal results : ')
Let's be honest: FaceGen does not output clean or nice results.

hazmhox said:

You gotta explain to me why "an entire company with tens of thousands real life simulated faces" succeeded to make a software outputting these abysmal results : ')
Let's be honest: FaceGen does not output clean or nice results.

Click to expand...

Let's be honest, when FaceGen came out in 2001y
It was an outstanding technology for that time. Of course, for the time being, this is an outdated program and already looks like garbage compared to modern plug-ins on blender.

So I'm not sure I'm following your reasoning here and your initial answer. But that's a bit out of the subject anyway.

I meant that it was a cool technology for its time. But now, of course, FaceGen is outdated and looks terrible.

First of all, I think this is a fantastic idea, and bold experiments like this are always worth encouraging. Without people willing to try difficult ideas, projects like VaM would never move forward in the first place.

That said, I think it's also worth having an objective discussion about how realistic the end goal is. I don't think the technical direction is wrong at all. In fact, I think it's probably the right direction. My concern is that achieving the level of results people are expecting is a much bigger challenge than it initially appears.

Even with AI assistance, I don't think this is the kind of task that current frontier LLMs are particularly good at. This isn't fundamentally a language or reasoning problem. It's a highly specialised inverse rendering and computer vision problem. I think it would require a dedicated model trained specifically for this purpose, not simply adding an LLM into the workflow.

The challenge also goes far beyond detecting facial landmarks or matching the outline of a face. A model would need to infer depth, facial projection, soft tissue, camera perspective, lens distortion, lighting, shadows, and even geometry that is partially hidden from one or two 2D images. In other words, it's trying to reconstruct 3D information that simply isn't present in the input.

On top of that, mapping those features into VaM's morph system is another major challenge. The same 2D appearance can often be produced by many different morph combinations. A deeper-looking eye socket, for example, could come from eye depth, brow projection, cheek projection, nose bridge shape, lighting, or a combination of all of them. Matching landmarks is one thing. Producing a face that actually looks like the same person is a completely different problem.

The biggest challenge, in my opinion, isn't the neural network itself. It's the data.

To achieve the level of accuracy people seem to expect, I don't think something like 160,000 synthetic VaM renders would be enough. The theoretical morph space is enormous, and randomly generated VaM faces don't necessarily represent the distribution of real human faces. More importantly, synthetic VaM renders and real photographs belong to completely different visual domains.

The ideal training data would probably need something much closer to real photographs, accurate 3D facial scans, verified VaM morph parameters, multiple camera angles, different lighting conditions, and human validation that the generated character actually resembles the original person. Building that dataset is arguably harder than building the model itself.

And even then, this isn't the kind of model you train once and call it finished. It would likely require countless rounds of experimentation with network architectures, loss functions, hyperparameter tuning, data cleaning, domain adaptation between real photos and VaM renders, and continuous evaluation. The compute cost and engineering effort would be enormous.

That's why I think this is on a completely different scale from what an individual developer, or even a typical small software company, could realistically take on. When I think about the amount of training data, parameter tuning, compute resources, and engineering involved, it feels much closer to the development process behind modern foundation models like SeedDance than a typical community project.

Don't get me wrong, I absolutely believe AI can improve this workflow. I think it could become an excellent starting-point generator that automatically gets creators 60 to 80% of the way there, which would already save a huge amount of manual work.

I'm just much more sceptical about the idea of a reliable one-click photo-to-VaM reconstruction system. In my opinion, reaching that level isn't simply a matter of using a bigger model or adding AI. It would require a highly specialised model trained on an enormous, carefully curated dataset, together with a level of compute, engineering, and iteration that is well beyond the reach of most individuals, and probably beyond what an ordinary company could realistically justify as well.

So I think the direction is absolutely worth exploring. I just think the gap between a promising prototype and a truly production-quality system is much larger than many people realise. The bottleneck here isn't AI itself. It's data, training scale, and engineering.

One additional thought.

Personally, I think a more realistic long-term direction would be to train a specialised AI model that predicts vertex positions directly on the DAZ Genesis 2 base mesh used by VaM, rather than trying to reconstruct a face by blending existing morphs. Morphs are ultimately an artist-friendly control system, not necessarily the ideal representation for machine learning.

Even then, I would still see that as a way to generate a solid starting point rather than a finished character. At least with today's technology, I think there would still be a significant amount of manual sculpting and refinement afterwards.

More importantly, this still doesn't solve what I see as the biggest challenge. Whether you're predicting morph weights or directly predicting mesh geometry, you still need a highly specialised model capable of reconstructing accurate 3D facial geometry from very limited 2D information. That means building an enormous, carefully curated training dataset, designing and tuning the model, handling domain adaptation between real photos and the DAZ/VaM mesh, and training with a level of compute that is well beyond what most individuals, and probably even an ordinary software company, could realistically afford.

So in my opinion, switching from morph prediction to direct mesh prediction makes more technical sense, but it doesn't really change the scale of the AI problem. The biggest bottleneck is still the specialised training pipeline, the data, and the engineering effort required to make it work reliably.