Idea for easier/better facial expression animations

[henshin]

I love VAM, but sadly facial expressions are a big problem, and I think is the most important part to mimic a real experience

That's why I want to create a library of many facial expression so we can reuse them in any scene later

I've tried copyn from real XXX videos and with Timeline plugin, but the process it's really slow

Is possible to control timeline plugin with another plugin (I can code it)?

My idea is this:
1. Modify the facil expression using the morphs UI
2. When I get the expression I want, create a frame with those morphs in timeline plugin (with just a button or key, going to the timeline and searching and selecting the morphs and also the frame is too slow)

 

[hazmhox]

Control timeline as in what? Play/stop/scrub, of course. Anything more advanced, probably not. You would need to fork Timeline to make your modifications.
What you want to do is simply make an "add" target but with current active pose morphs if I understand properly.

 

[henshin]

Control timeline as in what? Play/stop/scrub, of course. Anything more advanced, probably not. You would need to fork Timeline to make your modifications.
What you want to do is simply make an "add" target but with current active pose morphs if I understand properly.

Yes, the action should be something like:
1. Detect active facial expression morhps (only these, not all active morphs)
2. Create a new frame (maybe 1 second later, extend the timeline if necessary)
3. Copy morphs ti that frame (and reset to 0 any other already in timeline)

 

[henshin]

Maybe it's easier just to create a .json like the one created with "export animation".
Is there any documentation to understand the format? I don't know what are the "values":

{
   "SerializeVersion" : "283",
   "AtomType" : "Person",
   "Clips" : [
      {
         "AnimationName" : "Anim 1",
         "AnimationLength" : "6",
         "BlendDuration" : "1",
         "Loop" : "1",
         "NextAnimationRandomizeWeight" : "1",
         "AutoTransitionPrevious" : "0",
         "AutoTransitionNext" : "0",
         "SyncTransitionTime" : "1",
         "SyncTransitionTimeNL" : "0",
         "EnsureQuaternionContinuity" : "1",
         "AnimationLayer" : "Main",
         "Speed" : "1",
         "Weight" : "1",
         "Uninterruptible" : "0",
         "AnimationSegment" : "Segment 1",
         "FloatParams" : [
            {
               "Storable" : "geometry",
               "Name" : "morph: AA Blow Job Rig Fix",
               "Value" : [
                  "C0000000003",
                  "B9A99993E9885823E",
                  "BCDCC4C3FCD417F3E",
                  "B6666A63F3364B33E",
                  "BCDCCCC3F00000000",
                  "A0000C040"
               ],
               "Min" : "0",
               "Max" : "1"
            },
            {
               "Storable" : "geometry",
               "Name" : "morph: AA Drunk 5",
               "Value" : [
                  "C0000000003",
                  "ACDCC0C40",
                  "BCDCC4C40A403033E",
                  "B0000C04000000000"
               ],
               "Min" : "0",
               "Max" : "1"
            },

I think this is the class doing that "\src\AtomAnimations\Serialization\AtomAnimationSerializer.cs"
I will take a look, but if anyone has already the info or any tip is welcome

I guess these are the methods:

private static JSONNode SerializeCurve(BezierAnimationCurve curve, int serializeMode)
        {
            var curveJSON = new JSONArray();
            var lastV = 0f;
            var lastC = -1;
            for (var key = 0; key < curve.length; key++)
            {
                var keyframe = curve.GetKeyframeByKey(key);
                if (serializeMode == Modes.Optimized)
                {
                    var encoded = EncodeKeyframe(keyframe, lastV, lastC);
                    lastV = keyframe.value;
                    lastC = keyframe.curveType;
                    curveJSON.Add(encoded);
                }
                else
                {
                    var curveEntry = new JSONClass
                    {
                        ["t"] = keyframe.time.ToString(CultureInfo.InvariantCulture),
                    };
                    if (serializeMode == Modes.Full || keyframe.value != lastV)
                    {
                        curveEntry["v"] = keyframe.value.ToString(CultureInfo.InvariantCulture);
                        lastV = keyframe.value;
                    }
                    if (serializeMode == Modes.Full || keyframe.curveType != lastC)
                    {
                        curveEntry["c"] = keyframe.curveType.ToString(CultureInfo.InvariantCulture);
                        lastC = keyframe.curveType;
                    }
                    if (keyframe.curveType == CurveTypeValues.LeaveAsIs)
                    {
                        curveEntry["i"] = keyframe.controlPointIn.ToString(CultureInfo.InvariantCulture);
                        curveEntry["o"] = keyframe.controlPointOut.ToString(CultureInfo.InvariantCulture);
                    }
                    curveJSON.Add(curveEntry);
                }
            }
            return curveJSON;
        }

private static void DeserializeCurveFromArray(BezierAnimationCurve curve, JSONArray curveJSON, int version, ref bool dirty)
        {
            if (curveJSON.Count == 0) return;
            var lastT = -1f;
            var lastV = 0f;
            var lastC = CurveTypeValues.SmoothLocal;
            var brokenKeyframes = 0;
            foreach (JSONNode keyframeJSON in curveJSON)
            {
                try
                {
                    BezierKeyframe keyframe;
                    if (version >= 230 && !string.IsNullOrEmpty(keyframeJSON.Value))
                    {
                        // Compressed time and value
                        var value = keyframeJSON.Value;
                        keyframe = DecodeKeyframe(value, lastV, lastC, version);
                        if (keyframe.time < 0)
                        {
                            brokenKeyframes++;
                            continue;
                        }
                    }
                    else if(keyframeJSON is JSONClass)
                    {
                        var keyframeObject = (JSONClass)keyframeJSON;
                        // Separate time and value
                        keyframe = new BezierKeyframe
                        {
                            time = float.Parse(keyframeJSON["t"], CultureInfo.InvariantCulture).Snap(),
                            value = DeserializeFloat(keyframeJSON["v"], lastV),
                            curveType = keyframeObject.HasKey("c") ? int.Parse(keyframeJSON["c"]) : lastC,
                            controlPointIn = DeserializeFloat(keyframeJSON["i"]),
                            controlPointOut = DeserializeFloat(keyframeJSON["o"])
                        };
                        if (version < 230)
                        {
                            // Backward compatibility, tangents are not supported since bezier conversion.
                            if (keyframeObject.HasKey("ti"))
                            {
                                dirty = true;
                                if (keyframe.curveType == CurveTypeValues.LeaveAsIs)
                                    keyframe.curveType = CurveTypeValues.SmoothLocal;
                            }
                        }
                    }
                    else
                    {
                        throw new InvalidOperationException($"Invalid keyframe type {keyframeJSON.GetType()} with version {version}\n{keyframeJSON}");
                    }
                    if (Math.Abs(keyframe.time - lastT) <= float.Epsilon) continue;
                    lastT = keyframe.time;
                    lastV = keyframe.value;
                    lastC = keyframe.curveType;
                    if (lastC != CurveTypeValues.LeaveAsIs)
                        dirty = true;
                    curve.AddKey(keyframe);
                }
                catch (IndexOutOfRangeException exc)
                {
                    throw new InvalidOperationException($"Failed to read curve: {keyframeJSON}", exc);
                }
            }
            if (brokenKeyframes > 0)
            {
                SuperController.LogError("Timeline: " + brokenKeyframes + " broken keyframes were removed from the curve.");
            }
        }

Lastly these are the functions to encode the keyframe, I think we need only time and value

private static readonly StringBuilder _encodeSb = new StringBuilder();
        [MethodImpl(256)]
        private static string EncodeKeyframe(BezierKeyframe keyframe, float lastV, int lastC)
        {
            _encodeSb.Length = 0;
            // Time and Value encoding
            var hasValue = Math.Abs(lastV - keyframe.value) > float.Epsilon;
            var hasCurveType = lastC != keyframe.curveType;
            // Encoding sizes
            _encodeSb.Append(EncodeSizes(hasValue, hasCurveType));
            WriteBytes(keyframe.time, _encodeSb);
            if(hasValue) WriteBytes(keyframe.value, _encodeSb);
            if(hasCurveType) WriteBytes((byte)keyframe.curveType, _encodeSb);
            return _encodeSb.ToString();
        }
        [MethodImpl(256)]
        private static void WriteBytes(float value, StringBuilder sb)
        {
            var bytes = BitConverter.GetBytes(value);
            for (var i = 0; i < bytes.Length; i++)
            {
                var b = bytes[i];
                WriteBytes(b, sb);
            }
        }
        [MethodImpl(256)]
        private static void WriteBytes(byte value, StringBuilder sb)
        {
            sb.Append(value.ToString("X2"));
        }
        [MethodImpl(256)]
        private static char EncodeSizes(bool hasValue, bool hasCurveType)
        {
            var encodedValue = 0;
            if(hasValue) encodedValue |= (1 << 0);  // Assigns '01' in binary (1 in decimal)
            if(hasCurveType) encodedValue |= (1 << 1);  // Assigns '10' in binary (2 in decimal)
            return (char)('A' + encodedValue);
        }

 

[henshin]

OK I see there is another legacy format easier to create:

{
   "SerializeVersion" : "283",
   "AtomType" : "Person",
   "Clips" : [
      {
         "AnimationName" : "Anim 1",
         "AnimationLength" : "6",
         "BlendDuration" : "1",
         "Loop" : "1",
         "NextAnimationRandomizeWeight" : "1",
         "AutoTransitionPrevious" : "0",
         "AutoTransitionNext" : "0",
         "SyncTransitionTime" : "1",
         "SyncTransitionTimeNL" : "0",
         "EnsureQuaternionContinuity" : "1",
         "AnimationLayer" : "Main",
         "Speed" : "1",
         "Weight" : "1",
         "Uninterruptible" : "0",
         "AnimationSegment" : "Segment 1",
         "FloatParams" : [
            {
               "Storable" : "geometry",
               "Name" : "morph: AA Blow Job Rig Fix",
               "Value" : [
                  {
                     "t" : "0",
                     "c" : "3"
                  },
                  {
                     "t" : "0.3",
                     "v" : "0.2549255"
                  },
                  {
                     "t" : "0.8",
                     "v" : "0.2492744"
                  },
                  {
                     "t" : "1.3",
                     "v" : "0.3503738"
                  },
                  {
                     "t" : "1.6",
                     "v" : "0"
                  },
                  {
                     "t" : "6"
                  }
               ],
               "Min" : "0",
               "Max" : "1"
            },
            {
               "Storable" : "geometry",
               "Name" : "morph: AA Drunk 5",
               "Value" : [
                  {
                     "t" : "0",
                     "c" : "3"
                  },

I will be back with the plugin for facial expressions soon if it works fine