What is it?
This guide is about using VaM to create a 3D avatar for an AI running via SillyTavern.
The SillyTavern application is a client for backend applications with inference engines such as oobabooga/text-generation-webui api and KoboldAI, which provide access to a text generation AI (aka Large Language Models, LLMs). SillyTavern provides additional features for processing, visualizing and voicing the LLM output, focused on role-playing fictional characters/waifu (AI role-playing). This is a fork (version) of TavernAI and a free analogue of Voxta AI.
SillyTavern can work with both the API of an offline (local) model running on the user’s computer, and the API of an online model, for example, KoboldAI Horde and OpenAI.
The main useful features of SillyTavren are:
1) Basic functions and additional modules for processing LLM responses (such as Classify and Objective), allowing the results of these responses to be used in an application to run scripts. This type of module parses LLM responses received during role-play using the LLM itself and converts them into a structured JSON format for later use in the application. See below for more information on possible uses of Classify and Objective;
2) Basic functions and additional modules to improve the quality of role-play and character memory (for example, Persona Management, Smart Context, Summarize and World Info), taking into account the limited size of the prompt. Modules of this type, before each request, automatically modify the prompt used by the LLM, adding the most relevant information to it, taking into account keywords, as well as using a brief summary;
3) Basic functions and additional modules for visualization and high-quality voicing of a character (text to speech, TTS), as well as a voice recognition module;
4) Open source and the possibility to make your own modules/plugins in JavaScript;
5) Browser of APIs of online and offline models available for connection, and easy connection to any supported API;
6) A large database of ready-to-use presets for role-playing popular fictional characters, already configured to work with SillyTavern;
7) A convenient chat history editor with functions of saving and loading the chat history along with the module settings;
8) A relatively convenient and understandable GUI of the client (browser) part of the software.
Basic features and additional modules are described in detail in the project wiki: https://docs.sillytavern.app/
SillyTavern has a built-in system for creating and animating character avatars, but these avatars are 2D sprites and have extremely limited functionality. Using a VaM Person as an avatar for a role-playing AI, which, unlike a 2D sprite, has ample opportunities to interact with the player and the 3D game world, can be no less interesting.
On YouTube you can find a few tutorials on using SillyTavern together with VaM , for example:
However, the available guides do not reveal all the interesting features that the joint use of these programs provides.
This guide covers the following features:
1) How to broadcast the voice generated by the TTS module in SillyTavern from the Head Source of a VaM Person with lip synchronization (using Silero TTS module for voicing a VaM Person);
2) How to make the facial expression (morph) of a VaM Person match the mood of the character’s response (Character Expression) in SillyTavern (exporting and using data from the Classify module in VaM);
3) How to make the scene in VaM change in accordance with plot turns in SillyTavern (exporting and using data from the Objective module in VaM).
4) How to make the avatar (person atom) in VaM perform some action that was requested by the player from AI. For example, actions such as sitting on a chair or taking a book from a shelf.
Installing SillyTavern and VaM
You can install SillyTavern from the developers' website (see https://github.com/SillyTavern/SillyTavern). Various versions of this software are available for download. For this guide SillyTavern version 1.11.2 'staging' (24cd072e) was used.
There are many guides on the Internet for installing this software, for example:
But it is best to use the instructions from the developers on the website (github.com/SillyTavern/SillyTavern), since they were written for the actual version of this software.
For additional options such as Silero TTS and Classify, you also will need to install SillyTavern extras (see https://github.com/SillyTavern/SillyTavern-Extras) (I used the "Option 2 - Vanilla" way).
If you are reading this guide on Virt-A-Mate Hub, then probably, installing VaM does not require additional explanation. If you are new to Virt-A-Mate, please visit the Wiki on this website. VaM version 1.22.0.1 was used.
1. Voice and Lip Synchronization
To broadcast sound from SillyTavern to VaM, this version of the guide suggests using a virtual audio cable. Of course, this option is not without its drawbacks and requires some configuration of the virtual cable software to suit the configuration of a specific PC. However, it does not require writing additional scripts, which I think is important for many users, especially since processing audio clips requires some knowledge of Unity. First, let's look at how to broadcast the TTS output from SillyTavern to VAM. To do this you need to do the following:
1. Install Virtual Audio Cable (see https://vac.muzychenko.net/en/, full version is needed). With some settings, the program may transmit sound with bugs (clicks, squeaks, repeats). See the screen capture below for the setup with which everything works correctly for my PC:
Virtual Audio Cable settings
2. Launch VaM in windowed mode. To switch from full-screen to windowed mode, you can use either the key combination left Alt + Enter or the VaM (Config) batch file. Check that Lip Sync in the “Auto Behaviors” menu of a Person atom is set to “enabled”.
VaM Person settings, Auto Behaviors/Lip Sync tab
3. If you use SillyTavern extras, launch the SillyTavern extras server with the modules used. For example, to run the Classify and Silero-TTS modules, you need to enter the following commands in the Command Prompt/batch file:
4 Launch the SillyTavern browser client. Select the character, model, TTS and voice that will be used. For demos of this guide, the model used is koboldcpp/LLaMA2-13B-TiefighterLR. The voice used is Silero en12.
A large library of character presets compatible with SillyTavren is available, for example, on chub.ai website. For demos, of this guide the following 2B character preset was used: https://www.chub.ai/characters/cptpants/yorha-2b
This character, like the original 2B from Nier, is more focused on her duty to save mankind and her current mission and is not very willing to give in to attempts to turn the dialogue into an intimate plane. For NSFW scenes, perhaps another character is more suitable: https://www.chub.ai/characters/Koze/yorha-2b-cb6372ef
In the game, you will probably want the AI to provide short chat-style responses without long descriptions. To do this, it is useful to perform the following settings. In the character card, fill out the first message and examples of dialogue in the desired style. In the AI response formatting section, in the System Prompt and Last Output Sequence, specify your desired response style and length.
5. Go to the Windows sound setting. Select the option “App volume and device preferences”. For the browser, set Output as Line 1 (Virtual Audio Cable). For VaM, set Input as Line 1 (Output Audio Cable) (tested on Windows 10).
Audio Mixer Settings in Windows 10
6. For the Person atom in VaM, in the Head Audio menu, select the “Start Microphone Input” option. The microphone name should be displayed: Line 1 (Virtual Audio Cable).
Person settings in VaM, Head Audio tab
To check that everything is working correctly, you can open a page with an audio source in the browser, for example a podcast on YouTube. If everything is done correctly, the sound will come from the Person Head Audio Source, and the Person will move its lips in accordance with the sounds of the speech. Thus, when voice responses are generated by the TTS module from SillyTavern, the Person in VaM will voice them.
The video shows an example of how to run VaM and SillyTavern together:
2. Changing Person's morphs in VaM, in accordance with Character Expression in SillyTavern
So, the VaM Person voices text lines from SillyTavern, but its stony-neutral expression is disappointing. Let's now make the Person's facial morph change, in accordance with the mood of the LLM response in SillyTavern. The mood of the response is determined in the Classify module (see https://docs.sillytavern.app/extras/extensions/expression-images/), which requires the SillyTavern Extras server to run. In the SillyTavern browser client, you need to check the “Local server classification” checkbox in the Extensions/Character Expressions menu.
How to enable the Classify module in the Extensions/Character Expressions menu.
If the Classify module is active, then in the SillyTavern server window after each LLM response, in addition to the prompt and application parameters, the JSON content with expressions (moods) of the response with weighting coefficients (Classify output) will be shown. For the 2D avatar from SillyTavern, only the expression with the highest weight is used. We will do the same for the 3D avatar (Person atom) from VaM.
An example of the result of processing the response of LLM by the Classify module in the SillyTavern server window. Expression 'caring' that has the highest weighting coefficient will be displayed by the avatar.
Further, unfortunately, we cannot continue without editing the scripts. You can use any text editor to create and edit scripts. To make it easier to visually perceive the script, you can use a color-coded editor, such as Visual Studio Code. For even greater convenience, you can install extensions for JavaScript and C# in the editor: in this case, you can see the presence of syntax errors in the program before it is launched.
To make the result of the Classify module available in VaM, let’s edit the classify.js source script in the “...\SillyTavern\src\endpoints” folder. Before editing this file, it is recommended to save a backup of it, which you can roll back to if something goes wrong.
Location of the classify.js script in the SillyTavern folder
After the line in the classify.js script
Add the following:
In the path '.../Custom/Scripts/output.txt' the ellipsis needs to be replaced with the path to the folder with VaM, so that you get something like 'C:/.../VaM/Custom/Scripts/output.txt'. If all permissions are given (see user preferences/security menu in VaM), VaM should have no problem reading a file from this directory.
The "result" variable stores the entire output of the Classify module in JSON format. But we only need to store the first argument, which contains the name of the mood with the largest weighting factor, which is what the lines do:
Thus, when generating a response, SillyTavern with the Classify module running will also save a text file with the name of the current mood in the folder VaM/Custom/Scripts. See the classify.js script modified in this way below:
You will also need to make a plugin for VaM with a script that reads the output.txt file and, accordingly, changes the character's morph. This script must be saved in the “..Custom/Scripts” folder inside the VaM directory. See an example of the implementation in the script classify_0.cs below:
Location of the classify_0.cs script and the output.txt file in the VaM folder.
This version of the script uses only morphs built into VaM (grey colored ones), but they can also be replaced with morphs from AddonPackages (pink colored ones) if the appropriate add-ons are installed. Moreover, if the morph has a full morph identifier it looks like this: “ascorad.asco_Expressions.12:/Custom/Atom/Person/Morphs/female/asco - Expressions/asco - Regal Smile.vmi” (can be viewed by clicking the “Copy Uid” button” next to the slider of the corresponding morph in the Female morphs menu), then in the script you can select this morph like this:
Before activating the script in VaM, you need to modify the following line by writing the actual path to the VaM folder instead of the ellipsis:
This plugin must be added to the VaM Person in the "Plugins" menu. If everything is done correctly, the blue message log window in the VaM interface will constantly display the name of the current expression from SillyTavern.
Once all the files are installed, the Person's morph in VaM should change according to the mood of the response in SillyTavern after each new response.
3. Changing the VaM scene when colmeting tasks prescribed by AI in SillyTavren
An example of what a task tree might look like in the Objective module: to complete an AI script, you need to perform 3 tasks in sequence. The first task ("Take the last human survivor to the Resistance camp") has already been completed, the second task is active ("Upgrade your equipment at the Resistance camp").
3.1. Writing variable values from the SillyTavern client to a file for reading by VaM
A special feature of SillyTavern is its client-server (frontend-backend) architecture. At the same time, in client (browser) part procedures written in javascript, actions with the file system, such as writing to a file and reading from a file, are not directly available. Therefore, to create and overwrite files, you need to make a request to the server implemented in Node.JS.
Since, unlike the Classify module, the output of the Objective module is not sent to the application backend, we need to implement this ourselves. First, let’s add a script to the ‘.../scr/endpoints’ folder in the SillyTavern directory that writes a string variable to a file. The path to the file is written inside this request handler:
See the source script objective1.js for processing a request to write to a file below:
We also need to add a declaration of our request to write to the file to the server script (the server.js file in the root folder of the SillyTavern directory):
In the client module (in this case, this is the index.js module in the directory ‘..\SillyTavern\public\scripts\extensions\third-party\Extension-Objective’), the value of the variable from which needs to be written to the file, the corresponding function is declared:
An example call to this function for a file write request looks like this:
3.2. Using the output of the Objective module in VaM
After we have dealt with writing variable values from the SillyTavern client side to files, we will make the name of the current task available for reading in VaM. In order for the value of this variable to be available in VaM, add the index.js script in the directory “...\SillyTavern\public\scripts\extensions\third-party\Extension-Objective” the following request function to the server:
After the line
also add the following:
See the index.js script of the Objective module modified in this way below:
In the VaM script we add reading a file with the result of the Objective module and a scene transformation operation for each of the possible options, for example:
In this case, if the current task changes to “Upgrade your equipment at the Resistance camp”, then the trigger of the plugin attached to the Person atom (the MacGruber.Relay plugin from the Logic Bricks set) and all the actions attached to it are launched in the scene.
See the script classify_3.cs for VaM, modified to take into account the change in the scene based on the output of the Objective file below:
The video at the link below shows an example of the result with lip&expression synch, as well as the scene change caused by the output of the Objective module:
This video shows the following:
- AI plays the role of an android, which must guide the player to the Resistance camp in order to improve equipment there and replenish supplies, which is necessary to complete the global goal;
- At first, the dialogue with the AI is in the "Take the last human survivor to the Resistance camp" state. The application is configured so that the AI will check the completion of the current task after every 3 requests;
- 1:18 After responding to the player’s next request, the AI begins checking the completion of the current goal “Take the last human survivor to the Resistance camp” (the notification “Checking for task completion” pops up);
- 1:28 The check completes with an affirmative result (true), the current AI goal changes to "Upgrade your equipment at the Resistance camp" and the application changes the scene for this dialogue state. In this case, the Person's pose simply changes, and an object [box] is added to the scene, indicating that this part of the story takes place in the Resistance camp. But you can, of course, come up with and implement something more complex and interactive.
The character look is 2B by Xspada. The idle character animation is obtained by using the Alive plugin by SPQR.
The following video shows another demo in which not only the Person's pose, but also the background image switches depending on the state of the task tree. The current task is indicated in an UIText and the current character expression is indicated in the bottom left:
4. Function calling
Let's look at how you can make the avatar (person atom) in VaM perform some action that was requested by the player from AI. For example, actions such as sitting on a chair or taking a book from a shelf. To do this, we will define in advance a set of actions (functions) available to the character, write down their names and descriptions in the SillyTavern script, and give the AI instructions for selecting one of these actions based on the content of the last phrases of the dialogue. The idea was adopted from Action List Manager (ALM) plugin for Voxta by vaan20.
In order to skip implementing a new extension for SillyTavern when testing this functionality, let’s modify the existing Objective module. To do this, we will edit the index.js script in the ‘..\SillyTavern\public\scripts\extensions\third-party\Extension-Objective’ directory.
Please note that the selection and configuration of the LLM in this case is carried out in the same way as when using the original Objective module. I got the best results using the aphrodite/derceto-labs/Psycho-Crystal-16B model. Although smaller models, such as koboldcpp/LLaMA2-13B-TiefighterLR or koboldcpp/Silicon-Maid-7B, also showed acceptable results.
We will process the response of the role-playing AI to select one of the functions using the GenerateQuietPrompt() procedure. When using this feature, SillyTavern makes a request to the LLM engine, but does not record the result in the chat history, which is useful for performing AI-assisted internal functions.
We will use the following addition to the main prompt:
The ### Input tag contains all the functions available to the character that will be used in the scene. In this example, the functions are responsible for equipping various weapons (sword/bow/spear/nothing).
Next, change {{char}} in the prompt to the current character name, request an AI response via the GenerateQuietPrompt() function and save the response to a file.
See below script index.js from the Objective directory modified this way:
If everything works correctly, the AI response should contain only a name of one of the specified functions, as shown for example in the pictures below:
An example of Function calling working correctly with the aphrodite/derceto-labs/Psycho-Crystal-16B model
An example of Function calling working correctly with the koboldcpp/Silicon-Maid-7B model
In the script for processing SillyTavern responses in VaM, we will add the procedure SwitchWeapon1(), which reads the name of the function from the file “path3” (a variable containing the path to the file) and launches the corresponding trigger in VaM:
Add a call to function SwitchWeapon1() to the Update() function.
The video below shows an example of Function Calling in VaM based on requests to AI in SillyTavern:
In this video example, the player first asks the character to change her weapons to a bow, then remove all weapons, then equip a spear. The AI correctly calls the appropriate functions in accordance with the player's requests. The weapon assets are made by Torvald.
Another example shows the implementation of Function Calling in a scene with a nurse in a hospital. The functions are specified for situations where a nurse is interviewing a patient, conducting a physical examination, or performing treatment:
Prompt is similar to the previous example, but the following functions are specified:
Model used in this example is aphrodite/jebcarter/psyonic-cetacean-20B:
This guide is about using VaM to create a 3D avatar for an AI running via SillyTavern.
The SillyTavern application is a client for backend applications with inference engines such as oobabooga/text-generation-webui api and KoboldAI, which provide access to a text generation AI (aka Large Language Models, LLMs). SillyTavern provides additional features for processing, visualizing and voicing the LLM output, focused on role-playing fictional characters/waifu (AI role-playing). This is a fork (version) of TavernAI and a free analogue of Voxta AI.
SillyTavern can work with both the API of an offline (local) model running on the user’s computer, and the API of an online model, for example, KoboldAI Horde and OpenAI.
The main useful features of SillyTavren are:
1) Basic functions and additional modules for processing LLM responses (such as Classify and Objective), allowing the results of these responses to be used in an application to run scripts. This type of module parses LLM responses received during role-play using the LLM itself and converts them into a structured JSON format for later use in the application. See below for more information on possible uses of Classify and Objective;
2) Basic functions and additional modules to improve the quality of role-play and character memory (for example, Persona Management, Smart Context, Summarize and World Info), taking into account the limited size of the prompt. Modules of this type, before each request, automatically modify the prompt used by the LLM, adding the most relevant information to it, taking into account keywords, as well as using a brief summary;
3) Basic functions and additional modules for visualization and high-quality voicing of a character (text to speech, TTS), as well as a voice recognition module;
4) Open source and the possibility to make your own modules/plugins in JavaScript;
5) Browser of APIs of online and offline models available for connection, and easy connection to any supported API;
6) A large database of ready-to-use presets for role-playing popular fictional characters, already configured to work with SillyTavern;
7) A convenient chat history editor with functions of saving and loading the chat history along with the module settings;
8) A relatively convenient and understandable GUI of the client (browser) part of the software.
Basic features and additional modules are described in detail in the project wiki: https://docs.sillytavern.app/
SillyTavern has a built-in system for creating and animating character avatars, but these avatars are 2D sprites and have extremely limited functionality. Using a VaM Person as an avatar for a role-playing AI, which, unlike a 2D sprite, has ample opportunities to interact with the player and the 3D game world, can be no less interesting.
On YouTube you can find a few tutorials on using SillyTavern together with VaM , for example:
However, the available guides do not reveal all the interesting features that the joint use of these programs provides.
This guide covers the following features:
1) How to broadcast the voice generated by the TTS module in SillyTavern from the Head Source of a VaM Person with lip synchronization (using Silero TTS module for voicing a VaM Person);
2) How to make the facial expression (morph) of a VaM Person match the mood of the character’s response (Character Expression) in SillyTavern (exporting and using data from the Classify module in VaM);
3) How to make the scene in VaM change in accordance with plot turns in SillyTavern (exporting and using data from the Objective module in VaM).
4) How to make the avatar (person atom) in VaM perform some action that was requested by the player from AI. For example, actions such as sitting on a chair or taking a book from a shelf.
Installing SillyTavern and VaM
You can install SillyTavern from the developers' website (see https://github.com/SillyTavern/SillyTavern). Various versions of this software are available for download. For this guide SillyTavern version 1.11.2 'staging' (24cd072e) was used.
There are many guides on the Internet for installing this software, for example:
But it is best to use the instructions from the developers on the website (github.com/SillyTavern/SillyTavern), since they were written for the actual version of this software.
For additional options such as Silero TTS and Classify, you also will need to install SillyTavern extras (see https://github.com/SillyTavern/SillyTavern-Extras) (I used the "Option 2 - Vanilla" way).
If you are reading this guide on Virt-A-Mate Hub, then probably, installing VaM does not require additional explanation. If you are new to Virt-A-Mate, please visit the Wiki on this website. VaM version 1.22.0.1 was used.
1. Voice and Lip Synchronization
To broadcast sound from SillyTavern to VaM, this version of the guide suggests using a virtual audio cable. Of course, this option is not without its drawbacks and requires some configuration of the virtual cable software to suit the configuration of a specific PC. However, it does not require writing additional scripts, which I think is important for many users, especially since processing audio clips requires some knowledge of Unity. First, let's look at how to broadcast the TTS output from SillyTavern to VAM. To do this you need to do the following:
1. Install Virtual Audio Cable (see https://vac.muzychenko.net/en/, full version is needed). With some settings, the program may transmit sound with bugs (clicks, squeaks, repeats). See the screen capture below for the setup with which everything works correctly for my PC:
Virtual Audio Cable settings
2. Launch VaM in windowed mode. To switch from full-screen to windowed mode, you can use either the key combination left Alt + Enter or the VaM (Config) batch file. Check that Lip Sync in the “Auto Behaviors” menu of a Person atom is set to “enabled”.
VaM Person settings, Auto Behaviors/Lip Sync tab
3. If you use SillyTavern extras, launch the SillyTavern extras server with the modules used. For example, to run the Classify and Silero-TTS modules, you need to enter the following commands in the Command Prompt/batch file:
Code:
cd C:\...\SillyTavern-extras (directory where SillyTavern extras is installed)
python server.py --enable-modules=classify,silero-tts4 Launch the SillyTavern browser client. Select the character, model, TTS and voice that will be used. For demos of this guide, the model used is koboldcpp/LLaMA2-13B-TiefighterLR. The voice used is Silero en12.
A large library of character presets compatible with SillyTavren is available, for example, on chub.ai website. For demos, of this guide the following 2B character preset was used: https://www.chub.ai/characters/cptpants/yorha-2b
This character, like the original 2B from Nier, is more focused on her duty to save mankind and her current mission and is not very willing to give in to attempts to turn the dialogue into an intimate plane. For NSFW scenes, perhaps another character is more suitable: https://www.chub.ai/characters/Koze/yorha-2b-cb6372ef
In the game, you will probably want the AI to provide short chat-style responses without long descriptions. To do this, it is useful to perform the following settings. In the character card, fill out the first message and examples of dialogue in the desired style. In the AI response formatting section, in the System Prompt and Last Output Sequence, specify your desired response style and length.
5. Go to the Windows sound setting. Select the option “App volume and device preferences”. For the browser, set Output as Line 1 (Virtual Audio Cable). For VaM, set Input as Line 1 (Output Audio Cable) (tested on Windows 10).
Audio Mixer Settings in Windows 10
6. For the Person atom in VaM, in the Head Audio menu, select the “Start Microphone Input” option. The microphone name should be displayed: Line 1 (Virtual Audio Cable).
Person settings in VaM, Head Audio tab
To check that everything is working correctly, you can open a page with an audio source in the browser, for example a podcast on YouTube. If everything is done correctly, the sound will come from the Person Head Audio Source, and the Person will move its lips in accordance with the sounds of the speech. Thus, when voice responses are generated by the TTS module from SillyTavern, the Person in VaM will voice them.
The video shows an example of how to run VaM and SillyTavern together:
2. Changing Person's morphs in VaM, in accordance with Character Expression in SillyTavern
So, the VaM Person voices text lines from SillyTavern, but its stony-neutral expression is disappointing. Let's now make the Person's facial morph change, in accordance with the mood of the LLM response in SillyTavern. The mood of the response is determined in the Classify module (see https://docs.sillytavern.app/extras/extensions/expression-images/), which requires the SillyTavern Extras server to run. In the SillyTavern browser client, you need to check the “Local server classification” checkbox in the Extensions/Character Expressions menu.
How to enable the Classify module in the Extensions/Character Expressions menu.
If the Classify module is active, then in the SillyTavern server window after each LLM response, in addition to the prompt and application parameters, the JSON content with expressions (moods) of the response with weighting coefficients (Classify output) will be shown. For the 2D avatar from SillyTavern, only the expression with the highest weight is used. We will do the same for the 3D avatar (Person atom) from VaM.
An example of the result of processing the response of LLM by the Classify module in the SillyTavern server window. Expression 'caring' that has the highest weighting coefficient will be displayed by the avatar.
Further, unfortunately, we cannot continue without editing the scripts. You can use any text editor to create and edit scripts. To make it easier to visually perceive the script, you can use a color-coded editor, such as Visual Studio Code. For even greater convenience, you can install extensions for JavaScript and C# in the editor: in this case, you can see the presence of syntax errors in the program before it is launched.
To make the result of the Classify module available in VaM, let’s edit the classify.js source script in the “...\SillyTavern\src\endpoints” folder. Before editing this file, it is recommended to save a backup of it, which you can roll back to if something goes wrong.
Location of the classify.js script in the SillyTavern folder
After the line in the classify.js script
JavaScript:
console.log('Classify output:', result);Add the following:
JavaScript:
var arr2 = Object.values(result[0]);
const fs = require('fs')
let data = JSON.stringify(arr2[0]);
fs.writeFile('.../Custom/Scripts/output.txt', data, (err) => {
if (err) throw err;
});In the path '.../Custom/Scripts/output.txt' the ellipsis needs to be replaced with the path to the folder with VaM, so that you get something like 'C:/.../VaM/Custom/Scripts/output.txt'. If all permissions are given (see user preferences/security menu in VaM), VaM should have no problem reading a file from this directory.
The "result" variable stores the entire output of the Classify module in JSON format. But we only need to store the first argument, which contains the name of the mood with the largest weighting factor, which is what the lines do:
JavaScript:
var arr2 = Object.values(result[0]);
let data = JSON.stringify(arr2[0]);Thus, when generating a response, SillyTavern with the Classify module running will also save a text file with the name of the current mood in the folder VaM/Custom/Scripts. See the classify.js script modified in this way below:
JavaScript:
const express = require('express');
const { jsonParser } = require('../express-common');
const TASK = 'text-classification';
const router = express.Router();
const cacheObject = {};
router.post('/labels', jsonParser, async (req, res) => {
try {
const module = await import('../transformers.mjs');
const pipe = await module.default.getPipeline(TASK);
const result = Object.keys(pipe.model.config.label2id);
return res.json({ labels: result });
} catch (error) {
console.error(error);
return res.sendStatus(500);
}
});
router.post('/', jsonParser, async (req, res) => {
try {
const { text } = req.body;
async function getResult(text) {
if (Object.hasOwn(cacheObject, text)) {
return cacheObject[text];
} else {
const module = await import('../transformers.mjs');
const pipe = await module.default.getPipeline(TASK);
const result = await pipe(text, { topk: 5 });
result.sort((a, b) => b.score - a.score);
cacheObject[text] = result;
return result;
}
}
console.log('Classify input:', text);
const result = await getResult(text);
console.log('Classify output:', result);
var arr1 = Object.keys(result[0]);
var arr2 = Object.values(result[0]);
const fs = require('fs')
let data = JSON.stringify(arr2[0]);
fs.writeFile('C:/.../VaM/Custom/Scripts/output.txt', data, (err) => { if (err) throw err; });
return res.json({ classification: result });
} catch (error) {
console.error(error);
return res.sendStatus(500);
}
});
module.exports = { router };
classify.js
Displaying classify.js.You will also need to make a plugin for VaM with a script that reads the output.txt file and, accordingly, changes the character's morph. This script must be saved in the “..Custom/Scripts” folder inside the VaM directory. See an example of the implementation in the script classify_0.cs below:
C#:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using UnityEngine;
using UnityEngine.UI;
//using System.Collections;
//using SimpleJSON;
//using MacGruber;
namespace Test
{
class Test01: MVRScript
{
protected UIDynamicButton myButton;
public DAZMorph FaceMorph1;
protected String path1;
public override void Init()
{
pluginLabelJSON.val = "";
path1 = @"C:\...\VaM\Custom\Scripts\output.txt";
// Add a button and a click handler
myButton = CreateButton("Reset Expression", false);
myButton.height = 100;
myButton.label = "Reset Expression";
myButton.button.onClick.AddListener(delegate ()
{
SuperController.singleton.SaveStringIntoFile(path1, "neutral");
});
}
// Runs once when plugin loads - after Init()
protected void Start()
{
// show a message
SuperController.LogMessage(pluginLabelJSON.val + " Loaded");
}
// A Unity thing - runs every physics cycle
public void FixedUpdate()
{
// put code here
}
// Unity thing - runs every rendered frame
public void Update()
{
SwitchTr1();
SuperController.LogMessage("running");
}
void SwitchTr1()
{
JSONStorable geo = containingAtom.GetStorableByID("geometry");
DAZCharacterSelector character = geo as DAZCharacterSelector;
GenerateDAZMorphsControlUI morphControl = character.morphsControlUI;
//reset morphs
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Excitement"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Angry"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Happy"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Confused"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Desire"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Contempt"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Fear"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 2"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - For Me"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 3"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Taste Surprise"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("Curious1"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Bedroom Smile"); FaceMorph1.morphValue = 0f;
//FaceMorph1 = morphControl.GetMorphByDisplayName("AAeyesquintbrowworry2"); FaceMorph1.morphValue = 0f;
//path for the outpur file with expression
string expr1 = SuperController.singleton.ReadFileIntoString(path1);
//revome quote symbols if any
expr1 = expr1.Replace("\"", string.Empty);
SuperController.LogMessage(expr1);
//put a morph on Person according to Classify output
if (expr1 == "admiration") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "amusement") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Excitement");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "anger") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Angry");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "annoyance") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "approval") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Happy");
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - For Me");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "caring") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 2");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "confusion") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Confused");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "curiosity") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
//FaceMorph1 = morphControl.GetMorphByDisplayName("Curious1");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "desire") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Desire");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disappointment") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disapproval") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Contempt");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disgust") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "embarrassment") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "grief") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "gratitude") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Taste Surprise");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "joy") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "love") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile");
FaceMorph1.morphValue = 0.6f;
}
if (expr1 == "nervousness") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Fear");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "neutral") {
//
}
if (expr1 == "optimism") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "pride") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Desire");
// FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile");
FaceMorph1.morphValue = 0.6f;
}
if (expr1 == "realization") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
//FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Bedroom Smile");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "relief") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "remorse") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "sadness") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "surprise") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
FaceMorph1.morphValue = 0.3f;
}
}
}
}Location of the classify_0.cs script and the output.txt file in the VaM folder.
This version of the script uses only morphs built into VaM (grey colored ones), but they can also be replaced with morphs from AddonPackages (pink colored ones) if the appropriate add-ons are installed. Moreover, if the morph has a full morph identifier it looks like this: “ascorad.asco_Expressions.12:/Custom/Atom/Person/Morphs/female/asco - Expressions/asco - Regal Smile.vmi” (can be viewed by clicking the “Copy Uid” button” next to the slider of the corresponding morph in the Female morphs menu), then in the script you can select this morph like this:
C#:
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile");Before activating the script in VaM, you need to modify the following line by writing the actual path to the VaM folder instead of the ellipsis:
C#:
path1 = @"...\Custom\Scripts\output.txt";This plugin must be added to the VaM Person in the "Plugins" menu. If everything is done correctly, the blue message log window in the VaM interface will constantly display the name of the current expression from SillyTavern.
Once all the files are installed, the Person's morph in VaM should change according to the mood of the response in SillyTavern after each new response.
3. Changing the VaM scene when colmeting tasks prescribed by AI in SillyTavren
Let's look at how you can change the scene in VaM in accordance with the course of history in SillyTavern. One way to implement such a scene is to use the output of the Objective module (see https://docs.sillytavern.app/extras/extensions/objective/). This module allows you to set goals for AI within the current history, track their implementation and move along a given tree of goals (see screen capture below). In particular, it outputs the character's current task. Let's make the name of the current task available for reading in VaM.
Please note that not all models can run correctly with the Objective extension and provide a stable correct output. It seems to me that better results are provided by instruct models with a relatively large number of parameters in Instruct Mode (see https://docs.sillytavern.app/usage/core-concepts/instructmode/). Instruct Mode is supported, for example, by koboldcpp/LLaMA2-13B-TiefighterLR model:
Please note that not all models can run correctly with the Objective extension and provide a stable correct output. It seems to me that better results are provided by instruct models with a relatively large number of parameters in Instruct Mode (see https://docs.sillytavern.app/usage/core-concepts/instructmode/). Instruct Mode is supported, for example, by koboldcpp/LLaMA2-13B-TiefighterLR model:
KoboldAI/LLaMA2-13B-TiefighterLR · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
There is information that less resource-demanding models can also solve this problem, for example:
SanjiWatsuki/Silicon-Maid-7B · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
TheBloke/Zarafusionex-1.1-L2-7B-GGUF · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
TheBloke/MLewd-L2-Chat-13B-GPTQ · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
An example of what a task tree might look like in the Objective module: to complete an AI script, you need to perform 3 tasks in sequence. The first task ("Take the last human survivor to the Resistance camp") has already been completed, the second task is active ("Upgrade your equipment at the Resistance camp").
3.1. Writing variable values from the SillyTavern client to a file for reading by VaM
A special feature of SillyTavern is its client-server (frontend-backend) architecture. At the same time, in client (browser) part procedures written in javascript, actions with the file system, such as writing to a file and reading from a file, are not directly available. Therefore, to create and overwrite files, you need to make a request to the server implemented in Node.JS.
Since, unlike the Classify module, the output of the Objective module is not sent to the application backend, we need to implement this ourselves. First, let’s add a script to the ‘.../scr/endpoints’ folder in the SillyTavern directory that writes a string variable to a file. The path to the file is written inside this request handler:
JavaScript:
fs.writeFile('C:/.../VaM/Custom/Scripts/output33.txt', text, (err) => { if (err) throw err; });See the source script objective1.js for processing a request to write to a file below:
JavaScript:
const express = require('express');
const { jsonParser } = require('../express-common');
const TASK = 'text-classification';
const router = express.Router();
const cacheObject = {};
router.post('/labels', jsonParser, async (req, res) => {
try {
const module = await import('../transformers.mjs');
const pipe = await module.default.getPipeline(TASK);
const result = Object.keys(pipe.model.config.label2id);
return res.json({ labels: result });
} catch (error) {
console.error(error);
return res.sendStatus(500);
}
});
router.post('/', jsonParser, async (req, res) => {
try {
const fs = require('fs/promises');
const { text, path } = req.body;
console.log('Write to file:', text);
console.log('File path:', path);
await fs.writeFile(path, text);
return res.sendStatus(200);
} catch (error) {
console.error(error);
return res.sendStatus(500);
}
});
module.exports = { router };We also need to add a declaration of our request to write to the file to the server script (the server.js file in the root folder of the SillyTavern directory):
JavaScript:
// Write to file
app.use('/api/extra/objective1', require('./src/endpoints/objective1').router);In the client module (in this case, this is the index.js module in the directory ‘..\SillyTavern\public\scripts\extensions\third-party\Extension-Objective’), the value of the variable from which needs to be written to the file, the corresponding function is declared:
JavaScript:
//Write to file
async function Write1(text) {
const apiResult1 = await fetch('/api/extra/objective1', {
method: 'POST',
headers: getRequestHeaders(),
body: JSON.stringify({ text: text }),
});An example call to this function for a file write request looks like this:
JavaScript:
Write1(JSON.stringify(arr3[1]));3.2. Using the output of the Objective module in VaM
After we have dealt with writing variable values from the SillyTavern client side to files, we will make the name of the current task available for reading in VaM. In order for the value of this variable to be available in VaM, add the index.js script in the directory “...\SillyTavern\public\scripts\extensions\third-party\Extension-Objective” the following request function to the server:
JavaScript:
//Write to file
async function Write1(text) {
const apiResult1 = await fetch('/api/extra/objective1', {
method: 'POST',
headers: getRequestHeaders(),
body: JSON.stringify({ text: text }),
});}After the line
JavaScript:
console.info(`Current task in context.extensionPrompts.Objective is ${JSON.stringify(context.extensionPrompts.Objective)}`);also add the following:
JavaScript:
var arr3 = Object.values(currentTask);
Write1(JSON.stringify(arr3[1]));See the index.js script of the Objective module modified in this way below:
JavaScript:
import { chat_metadata, callPopup, saveSettingsDebounced, is_send_press, getRequestHeaders } from '../../../../script.js';
import { getContext, extension_settings, saveMetadataDebounced } from '../../../extensions.js';
import {
substituteParams,
eventSource,
event_types,
generateQuietPrompt,
} from '../../../../script.js';
import { registerSlashCommand } from '../../../slash-commands.js';
import { waitUntilCondition } from '../../../utils.js';
import { is_group_generating, selected_group } from '../../../group-chats.js';
const MODULE_NAME = 'Objective';
let taskTree = null;
let globalTasks = [];
let currentChatId = '';
let currentObjective = null;
let currentTask = null;
let checkCounter = 0;
let lastMessageWasSwipe = false;
const defaultPrompts = {
'createTask': 'Pause your roleplay. Please generate a numbered list of plain text tasks to complete an objective. The objective that you must make a numbered task list for is: "{{objective}}". The tasks created should take into account the character traits of {{char}}. These tasks may or may not involve {{user}} directly. Include the objective as the final task.',
'checkTaskCompleted': 'Pause your roleplay. Determine if this task is completed: [{{task}}]. To do this, examine the most recent messages. Your response must only contain either true or false, and nothing else. Example output: true',
'currentTask': 'Your current task is [{{task}}]. Balance existing roleplay with completing this task.',
};
let objectivePrompts = defaultPrompts;
//###############################//
//# Task Management #//
//###############################//
// Return the task and index or throw an error
function getTaskById(taskId) {
if (taskId == null) {
throw 'Null task id';
}
return getTaskByIdRecurse(taskId, taskTree);
}
function getTaskByIdRecurse(taskId, task) {
if (task.id == taskId) {
return task;
}
for (const childTask of task.children) {
const foundTask = getTaskByIdRecurse(taskId, childTask);
if (foundTask != null) {
return foundTask;
}
}
return null;
}
function substituteParamsPrompts(content, substituteGlobal) {
content = content.replace(/{{objective}}/gi, currentObjective.description);
content = content.replace(/{{task}}/gi, currentTask.description);
if (currentTask.parent) {
content = content.replace(/{{parent}}/gi, currentTask.parent.description);
}
if (substituteGlobal) {
content = substituteParams(content);
}
return content;
}
// Call Quiet Generate to create task list using character context, then convert to tasks. Should not be called much.
async function generateTasks() {
const prompt = substituteParamsPrompts(objectivePrompts.createTask, false);
console.log('Generating tasks for objective with prompt');
toastr.info('Generating tasks for objective', 'Please wait...');
const taskResponse = await generateQuietPrompt(prompt);
// Clear all existing objective tasks when generating
currentObjective.children = [];
const numberedListPattern = /^\d+\./;
// Create tasks from generated task list
for (const task of taskResponse.split('\n').map(x => x.trim())) {
if (task.match(numberedListPattern) != null) {
currentObjective.addTask(task.replace(numberedListPattern, '').trim());
}
}
updateUiTaskList();
setCurrentTask();
console.info(`Response for Objective: '${currentObjective.description}' was \n'${taskResponse}', \nwhich created tasks \n${JSON.stringify(currentObjective.children.map(v => { return v.toSaveState(); }), null, 2)} `);
toastr.success(`Generated ${currentObjective.children.length} tasks`, 'Done!');
}
// Call Quiet Generate to check if a task is completed
async function checkTaskCompleted() {
//console.log('test2');
//const fs = require('fs')
//let data = 'example';
//fs.writeFile('.../Custom/Scripts/output1.txt', data, (err) => { if (err) throw err; });
// Make sure there are tasks
if (jQuery.isEmptyObject(currentTask)) {
return;
}
try {
// Wait for group to finish generating
if (selected_group) {
await waitUntilCondition(() => is_group_generating === false, 1000, 10);
}
// Another extension might be doing something with the chat, so wait for it to finish
await waitUntilCondition(() => is_send_press === false, 30000, 10);
} catch {
console.debug('Failed to wait for group to finish generating');
return;
}
checkCounter = $('#objective-check-frequency').val();
toastr.info('Checking for task completion.');
const prompt = substituteParamsPrompts(objectivePrompts.checkTaskCompleted, false);
const taskResponse = (await generateQuietPrompt(prompt)).toLowerCase();
// Check response if task complete
if (taskResponse.includes('true')) {
console.info(`Character determined task '${currentTask.description} is completed.`);
currentTask.completeTask();
} else if (!(taskResponse.includes('false'))) {
console.warn(`checkTaskCompleted response did not contain true or false. taskResponse: ${taskResponse}`);
} else {
console.debug(`Checked task completion. taskResponse: ${taskResponse}`);
}
}
function getNextIncompleteTaskRecurse(task) {
if (task.completed === false // Return task if incomplete
&& task.children.length === 0 // Ensure task has no children, it's subtasks will determine completeness
&& task.parentId !== '' // Must have parent id. Only root task will be missing this and we dont want that
) {
return task;
}
for (const childTask of task.children) {
if (childTask.completed === true) { // Don't recurse into completed tasks
continue;
}
const foundTask = getNextIncompleteTaskRecurse(childTask);
if (foundTask != null) {
return foundTask;
}
}
return null;
}
// Set a task in extensionPrompt context. Defaults to first incomplete
function setCurrentTask(taskId = null, skipSave = false) {
const context = getContext();
//console.log('test1');
// TODO: Should probably null this rather than set empty object
currentTask = {};
// Find the task, either next incomplete, or by provided taskId
if (taskId === null) {
currentTask = getNextIncompleteTaskRecurse(taskTree) || {};
} else {
currentTask = getTaskById(taskId);
}
// Don't just check for a current task, check if it has data
const description = currentTask.description || null;
if (description) {
const extensionPromptText = substituteParamsPrompts(objectivePrompts.currentTask, true);
// Remove highlights
$('.objective-task').css({ 'border-color': '', 'border-width': '' });
// Highlight current task
let highlightTask = currentTask;
while (highlightTask.parentId !== '') {
if (highlightTask.descriptionSpan) {
highlightTask.descriptionSpan.css({ 'border-color': 'yellow', 'border-width': '2px' });
}
const parent = getTaskById(highlightTask.parentId);
highlightTask = parent;
}
// Update the extension prompt
context.setExtensionPrompt(MODULE_NAME, extensionPromptText, 1, $('#objective-chat-depth').val());
console.info(`Current task in context.extensionPrompts.Objective is ${JSON.stringify(context.extensionPrompts.Objective)}`);
var arr3 = Object.values(currentTask);
//let data = JSON.stringify(arr2[0]);
//Write1(JSON.stringify(currentTask));
Write1(JSON.stringify(arr3[1]));
//console.log(currentTask);
} else {
context.setExtensionPrompt(MODULE_NAME, '');
console.info('No current task');
}
// Save state if not skipping
if (!skipSave) {
saveState();
}
}
function getHighestTaskIdRecurse(task) {
let nextId = task.id;
for (const childTask of task.children) {
const childId = getHighestTaskIdRecurse(childTask);
if (childId > nextId) {
nextId = childId;
}
}
return nextId;
}
//###############################//
//# Task Class #//
//###############################//
class ObjectiveTask {
id;
description;
completed;
parentId;
children;
// UI Elements
taskHtml;
descriptionSpan;
completedCheckbox;
deleteTaskButton;
addTaskButton;
constructor({ id = undefined, description, completed = false, parentId = '' }) {
this.description = description;
this.parentId = parentId;
this.children = [];
this.completed = completed;
// Generate a new ID if none specified
if (id == undefined) {
this.id = getHighestTaskIdRecurse(taskTree) + 1;
} else {
this.id = id;
}
}
// Accepts optional index. Defaults to adding to end of list.
addTask(description, index = null) {
index = index != null ? index : index = this.children.length;
this.children.splice(index, 0, new ObjectiveTask(
{ description: description, parentId: this.id },
));
saveState();
}
getIndex() {
if (this.parentId !== null) {
const parent = getTaskById(this.parentId);
const index = parent.children.findIndex(task => task.id === this.id);
if (index === -1) {
throw `getIndex failed: Task '${this.description}' not found in parent task '${parent.description}'`;
}
return index;
} else {
throw `getIndex failed: Task '${this.description}' has no parent`;
}
}
// Used to set parent to complete when all child tasks are completed
checkParentComplete() {
let all_completed = true;
if (this.parentId !== '') {
const parent = getTaskById(this.parentId);
for (const child of parent.children) {
if (!child.completed) {
all_completed = false;
break;
}
}
if (all_completed) {
parent.completed = true;
console.info(`Parent task '${parent.description}' completed after all child tasks complated.`);
} else {
parent.completed = false;
}
}
}
// Complete the current task, setting next task to next incomplete task
completeTask() {
this.completed = true;
console.info(`Task successfully completed: ${JSON.stringify(this.description)}`);
this.checkParentComplete();
setCurrentTask();
updateUiTaskList();
}
// Add a single task to the UI and attach event listeners for user edits
addUiElement() {
const template = `
<div id="objective-task-label-${this.id}" class="flex1 checkbox_label">
<input id="objective-task-complete-${this.id}" type="checkbox">
<span class="text_pole objective-task" style="display: block" id="objective-task-description-${this.id}" contenteditable>${this.description}</span>
<div id="objective-task-delete-${this.id}" class="objective-task-button fa-solid fa-xmark fa-2x" title="Delete Task"></div>
<div id="objective-task-add-${this.id}" class="objective-task-button fa-solid fa-plus fa-2x" title="Add Task"></div>
<div id="objective-task-add-branch-${this.id}" class="objective-task-button fa-solid fa-code-fork fa-2x" title="Branch Task"></div>
</div><br>
`;
// Add the filled out template
$('#objective-tasks').append(template);
this.completedCheckbox = $(`#objective-task-complete-${this.id}`);
this.descriptionSpan = $(`#objective-task-description-${this.id}`);
this.addButton = $(`#objective-task-add-${this.id}`);
this.deleteButton = $(`#objective-task-delete-${this.id}`);
this.taskHtml = $(`#objective-task-label-${this.id}`);
this.branchButton = $(`#objective-task-add-branch-${this.id}`);
// Handle sub-task forking style
if (this.children.length > 0) {
this.branchButton.css({ 'color': '#33cc33' });
} else {
this.branchButton.css({ 'color': '' });
}
// Add event listeners and set properties
$(`#objective-task-complete-${this.id}`).prop('checked', this.completed);
$(`#objective-task-complete-${this.id}`).on('click', () => (this.onCompleteClick()));
$(`#objective-task-description-${this.id}`).on('keyup', () => (this.onDescriptionUpdate()));
$(`#objective-task-description-${this.id}`).on('focusout', () => (this.onDescriptionFocusout()));
$(`#objective-task-delete-${this.id}`).on('click', () => (this.onDeleteClick()));
$(`#objective-task-add-${this.id}`).on('click', () => (this.onAddClick()));
this.branchButton.on('click', () => (this.onBranchClick()));
}
onBranchClick() {
currentObjective = this;
updateUiTaskList();
setCurrentTask();
}
onCompleteClick() {
this.completed = this.completedCheckbox.prop('checked');
this.checkParentComplete();
setCurrentTask();
}
onDescriptionUpdate() {
this.description = this.descriptionSpan.text();
}
onDescriptionFocusout() {
setCurrentTask();
}
onDeleteClick() {
const index = this.getIndex();
const parent = getTaskById(this.parentId);
parent.children.splice(index, 1);
updateUiTaskList();
setCurrentTask();
}
onAddClick() {
const index = this.getIndex();
const parent = getTaskById(this.parentId);
parent.addTask('', index + 1);
updateUiTaskList();
setCurrentTask();
}
toSaveStateRecurse() {
let children = [];
if (this.children.length > 0) {
for (const child of this.children) {
children.push(child.toSaveStateRecurse());
}
}
return {
'id': this.id,
'description': this.description,
'completed': this.completed,
'parentId': this.parentId,
'children': children,
};
}
}
//###############################//
//# Custom Prompts #//
//###############################//
function onEditPromptClick() {
let popupText = '';
popupText += `
<div class="objective_prompt_modal">
<small>Edit prompts used by Objective for this session. You can use {{objective}} or {{task}} plus any other standard template variables. Save template to persist changes.</small>
<br>
<div>
<label for="objective-prompt-generate">Generation Prompt</label>
<textarea id="objective-prompt-generate" type="text" class="text_pole textarea_compact" rows="8"></textarea>
<label for="objective-prompt-check">Completion Check Prompt</label>
<textarea id="objective-prompt-check" type="text" class="text_pole textarea_compact" rows="8"></textarea>
<label for="objective-prompt-extension-prompt">Injected Prompt</label>
<textarea id="objective-prompt-extension-prompt" type="text" class="text_pole textarea_compact" rows="8"></textarea>
</div>
<div class="objective_prompt_block">
<label for="objective-custom-prompt-select">Custom Prompt Select</label>
<select id="objective-custom-prompt-select"><select>
</div>
<div class="objective_prompt_block">
<input id="objective-custom-prompt-new" class="menu_button" type="submit" value="New Prompt" />
<input id="objective-custom-prompt-save" class="menu_button" type="submit" value="Save Prompt" />
<input id="objective-custom-prompt-delete" class="menu_button" type="submit" value="Delete Prompt" />
</div>
</div>`;
callPopup(popupText, 'text');
populateCustomPrompts();
// Set current values
$('#objective-prompt-generate').val(objectivePrompts.createTask);
$('#objective-prompt-check').val(objectivePrompts.checkTaskCompleted);
$('#objective-prompt-extension-prompt').val(objectivePrompts.currentTask);
// Handle value updates
$('#objective-prompt-generate').on('input', () => {
objectivePrompts.createTask = $('#objective-prompt-generate').val();
});
$('#objective-prompt-check').on('input', () => {
objectivePrompts.checkTaskCompleted = $('#objective-prompt-check').val();
});
$('#objective-prompt-extension-prompt').on('input', () => {
objectivePrompts.currentTask = $('#objective-prompt-extension-prompt').val();
});
// Handle new
$('#objective-custom-prompt-new').on('click', () => {
newCustomPrompt();
});
// Handle save
$('#objective-custom-prompt-save').on('click', () => {
saveCustomPrompt();
});
// Handle delete
$('#objective-custom-prompt-delete').on('click', () => {
deleteCustomPrompt();
});
// Handle load
$('#objective-custom-prompt-select').on('change', loadCustomPrompt);
}
async function newCustomPrompt() {
const customPromptName = await callPopup('<h3>Custom Prompt name:</h3>', 'input');
if (customPromptName == '') {
toastr.warning('Please set custom prompt name to save.');
return;
}
if (customPromptName == 'default') {
toastr.error('Cannot save over default prompt');
return;
}
extension_settings.objective.customPrompts[customPromptName] = {};
Object.assign(extension_settings.objective.customPrompts[customPromptName], objectivePrompts);
saveSettingsDebounced();
populateCustomPrompts();
}
function saveCustomPrompt() {
const customPromptName = $('#objective-custom-prompt-select').find(':selected').val();
if (customPromptName == 'default') {
toastr.error('Cannot save over default prompt');
return;
}
Object.assign(extension_settings.objective.customPrompts[customPromptName], objectivePrompts);
saveSettingsDebounced();
populateCustomPrompts();
}
function deleteCustomPrompt() {
const customPromptName = $('#objective-custom-prompt-select').find(':selected').val();
if (customPromptName == 'default') {
toastr.error('Cannot delete default prompt');
return;
}
delete extension_settings.objective.customPrompts[customPromptName];
saveSettingsDebounced();
populateCustomPrompts();
loadCustomPrompt();
}
function loadCustomPrompt() {
const optionSelected = $('#objective-custom-prompt-select').find(':selected').val();
Object.assign(objectivePrompts, extension_settings.objective.customPrompts[optionSelected]);
$('#objective-prompt-generate').val(objectivePrompts.createTask);
$('#objective-prompt-check').val(objectivePrompts.checkTaskCompleted);
$('#objective-prompt-extension-prompt').val(objectivePrompts.currentTask);
}
function populateCustomPrompts() {
// Populate saved prompts
$('#objective-custom-prompt-select').empty();
for (const customPromptName in extension_settings.objective.customPrompts) {
const option = document.createElement('option');
option.innerText = customPromptName;
option.value = customPromptName;
option.selected = customPromptName;
$('#objective-custom-prompt-select').append(option);
}
}
//###############################//
//# UI AND Settings #//
//###############################//
const defaultSettings = {
currentObjectiveId: null,
taskTree: null,
chatDepth: 2,
checkFrequency: 3,
hideTasks: false,
prompts: defaultPrompts,
};
// Convenient single call. Not much at the moment.
function resetState() {
lastMessageWasSwipe = false;
loadSettings();
}
//
function saveState() {
const context = getContext();
if (currentChatId == '') {
currentChatId = context.chatId;
}
chat_metadata['objective'] = {
currentObjectiveId: currentObjective.id,
taskTree: taskTree.toSaveStateRecurse(),
checkFrequency: $('#objective-check-frequency').val(),
chatDepth: $('#objective-chat-depth').val(),
hideTasks: $('#objective-hide-tasks').prop('checked'),
prompts: objectivePrompts,
};
saveMetadataDebounced();
}
// Dump core state
function debugObjectiveExtension() {
console.log(JSON.stringify({
'currentTask': currentTask,
'currentObjective': currentObjective,
'taskTree': taskTree.toSaveStateRecurse(),
'chat_metadata': chat_metadata['objective'],
'extension_settings': extension_settings['objective'],
'prompts': objectivePrompts,
}, null, 2));
}
window.debugObjectiveExtension = debugObjectiveExtension;
// Populate UI task list
function updateUiTaskList() {
$('#objective-tasks').empty();
// Show button to navigate back to parent objective if parent exists
if (currentObjective) {
if (currentObjective.parentId !== '') {
$('#objective-parent').show();
} else {
$('#objective-parent').hide();
}
}
$('#objective-text').val(currentObjective.description);
if (currentObjective.children.length > 0) {
// Show tasks if there are any to show
for (const task of currentObjective.children) {
task.addUiElement();
}
} else {
// Show button to add tasks if there are none
$('#objective-tasks').append(`
<input id="objective-task-add-first" type="button" class="menu_button" value="Add Task">
`);
$('#objective-task-add-first').on('click', () => {
currentObjective.addTask('');
setCurrentTask();
updateUiTaskList();
});
}
}
function onParentClick() {
currentObjective = getTaskById(currentObjective.parentId);
updateUiTaskList();
setCurrentTask();
}
// Trigger creation of new tasks with given objective.
async function onGenerateObjectiveClick() {
await generateTasks();
saveState();
}
//Write to file
async function Write1(text) {
const apiResult1 = await fetch('/api/extra/objective1', {
method: 'POST',
headers: getRequestHeaders(),
body: JSON.stringify({ text: text }),
});
}
// Update extension prompts
function onChatDepthInput() {
saveState();
setCurrentTask(); // Ensure extension prompt is updated
}
function onObjectiveTextFocusOut() {
if (currentObjective) {
currentObjective.description = $('#objective-text').val();
saveState();
}
}
// Update how often we check for task completion
function onCheckFrequencyInput() {
checkCounter = $('#objective-check-frequency').val();
$('#objective-counter').text(checkCounter);
saveState();
}
function onHideTasksInput() {
$('#objective-tasks').prop('hidden', $('#objective-hide-tasks').prop('checked'));
saveState();
}
function loadTaskChildrenRecurse(savedTask) {
let tempTaskTree = new ObjectiveTask({
id: savedTask.id,
description: savedTask.description,
completed: savedTask.completed,
parentId: savedTask.parentId,
});
for (const task of savedTask.children) {
const childTask = loadTaskChildrenRecurse(task);
tempTaskTree.children.push(childTask);
}
return tempTaskTree;
}
function loadSettings() {
// Load/Init settings for chatId
currentChatId = getContext().chatId;
// Reset Objectives and Tasks in memory
taskTree = null;
currentObjective = null;
// Init extension settings
if (Object.keys(extension_settings.objective).length === 0) {
Object.assign(extension_settings.objective, { 'customPrompts': { 'default': defaultPrompts } });
}
// Bail on home screen
if (currentChatId == undefined) {
return;
}
// Migrate existing settings
if (currentChatId in extension_settings.objective) {
// TODO: Remove this soon
chat_metadata['objective'] = extension_settings.objective[currentChatId];
delete extension_settings.objective[currentChatId];
}
if (!('objective' in chat_metadata)) {
Object.assign(chat_metadata, { objective: defaultSettings });
}
// Migrate legacy flat objective to new objectiveTree and currentObjective
if ('objective' in chat_metadata.objective) {
// Create root objective from legacy objective
taskTree = new ObjectiveTask({ id: 0, description: chat_metadata.objective.objective });
currentObjective = taskTree;
// Populate root objective tree from legacy tasks
if ('tasks' in chat_metadata.objective) {
let idIncrement = 0;
taskTree.children = chat_metadata.objective.tasks.map(task => {
idIncrement += 1;
return new ObjectiveTask({
id: idIncrement,
description: task.description,
completed: task.completed,
parentId: taskTree.id,
});
});
}
saveState();
delete chat_metadata.objective.objective;
delete chat_metadata.objective.tasks;
} else {
// Load Objectives and Tasks (Normal path)
if (chat_metadata.objective.taskTree) {
taskTree = loadTaskChildrenRecurse(chat_metadata.objective.taskTree);
}
}
// Make sure there's a root task
if (!taskTree) {
taskTree = new ObjectiveTask({ id: 0, description: $('#objective-text').val() });
}
currentObjective = taskTree;
checkCounter = chat_metadata['objective'].checkFrequency;
// Update UI elements
$('#objective-counter').text(checkCounter);
$('#objective-text').text(taskTree.description);
updateUiTaskList();
$('#objective-chat-depth').val(chat_metadata['objective'].chatDepth);
$('#objective-check-frequency').val(chat_metadata['objective'].checkFrequency);
$('#objective-hide-tasks').prop('checked', chat_metadata['objective'].hideTasks);
$('#objective-tasks').prop('hidden', $('#objective-hide-tasks').prop('checked'));
setCurrentTask(null, true);
}
function addManualTaskCheckUi() {
$('#extensionsMenu').prepend(`
<div id="objective-task-manual-check-menu-item" class="list-group-item flex-container flexGap5">
<div id="objective-task-manual-check" class="extensionsMenuExtensionButton fa-regular fa-square-check"/></div>
Manual Task Check
</div>`);
$('#objective-task-manual-check-menu-item').attr('title', 'Trigger AI check of completed tasks').on('click', checkTaskCompleted);
}
jQuery(() => {
const settingsHtml = `
<div class="objective-settings">
<div class="inline-drawer">
<div class="inline-drawer-toggle inline-drawer-header">
<b>Objective</b>
<div class="inline-drawer-icon fa-solid fa-circle-chevron-down down"></div>
</div>
<div class="inline-drawer-content">
<label for="objective-text"><small>Enter an objective and generate tasks. The AI will attempt to complete tasks autonomously</small></label>
<textarea id="objective-text" type="text" class="text_pole textarea_compact" rows="4"></textarea>
<div class="objective_block flex-container">
<input id="objective-generate" class="menu_button" type="submit" value="Auto-Generate Tasks" />
<label class="checkbox_label"><input id="objective-hide-tasks" type="checkbox"> Hide Tasks</label>
</div>
<div id="objective-parent" class="objective_block flex-container">
<i class="objective-task-button fa-solid fa-circle-left fa-2x" title="Go to Parent"></i>
<small>Go to parent task</small>
</div>
<div id="objective-tasks"> </div>
<div class="objective_block margin-bot-10px">
<div class="objective_block objective_block_control flex1 flexFlowColumn">
<label for="objective-chat-depth">Position in Chat</label>
<input id="objective-chat-depth" class="text_pole widthUnset" type="number" min="0" max="99" />
</div>
<br>
<div class="objective_block objective_block_control flex1">
<label for="objective-check-frequency">Task Check Frequency</label>
<input id="objective-check-frequency" class="text_pole widthUnset" type="number" min="0" max="99" />
<small>(0 = disabled)</small>
</div>
</div>
<span> Messages until next AI task completion check <span id="objective-counter">0</span></span>
<div class="objective_block flex-container">
<input id="objective_prompt_edit" class="menu_button" type="submit" value="Edit Prompts" />
</div>
<hr class="sysHR">
</div>
</div>
</div>
`;
addManualTaskCheckUi();
$('#extensions_settings').append(settingsHtml);
$('#objective-generate').on('click', onGenerateObjectiveClick);
$('#objective-chat-depth').on('input', onChatDepthInput);
$('#objective-check-frequency').on('input', onCheckFrequencyInput);
$('#objective-hide-tasks').on('click', onHideTasksInput);
$('#objective_prompt_edit').on('click', onEditPromptClick);
$('#objective-parent').hide();
$('#objective-parent').on('click', onParentClick);
$('#objective-text').on('focusout', onObjectiveTextFocusOut);
loadSettings();
eventSource.on(event_types.CHAT_CHANGED, () => {
resetState();
});
eventSource.on(event_types.MESSAGE_SWIPED, () => {
lastMessageWasSwipe = true;
});
eventSource.on(event_types.MESSAGE_RECEIVED, () => {
if (currentChatId == undefined || jQuery.isEmptyObject(currentTask) || lastMessageWasSwipe) {
lastMessageWasSwipe = false;
return;
}
if ($('#objective-check-frequency').val() > 0) {
// Check only at specified interval
if (checkCounter <= 0) {
checkTaskCompleted();
}
checkCounter -= 1;
}
setCurrentTask();
$('#objective-counter').text(checkCounter);
});
registerSlashCommand('taskcheck', checkTaskCompleted, [], '– checks if the current task is completed', true, true);
});
index.js
Displaying index.js.In the VaM script we add reading a file with the result of the Objective module and a scene transformation operation for each of the possible options, for example:
C#:
if (expr1 == "Upgrade your equipment at the Resistance camp") {
myAction = containingAtom.GetStorableByID("plugin#" + 4 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}In this case, if the current task changes to “Upgrade your equipment at the Resistance camp”, then the trigger of the plugin attached to the Person atom (the MacGruber.Relay plugin from the Logic Bricks set) and all the actions attached to it are launched in the scene.
See the script classify_3.cs for VaM, modified to take into account the change in the scene based on the output of the Objective file below:
C#:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using UnityEngine;
using UnityEngine.UI;
//using System.Collections;
//using SimpleJSON;
//using MacGruber;
namespace Test
{
class Test01: MVRScript
{
protected UIDynamicButton myButton;
public DAZMorph FaceMorph1;
protected String path1;
protected String path2;
protected String temp2;
JSONStorableAction myAction;
public override void Init()
{
pluginLabelJSON.val = "";
temp2 = "";
path1 = @"C:\...\VaM\Custom\Scripts\output.txt";
path2 = @"C:\...\VAM\Custom\Scripts\output33.txt";
// Add a button and a click handler
myButton = CreateButton("Reset Expression", false);
myButton.height = 100;
myButton.label = "Reset Expression";
myButton.button.onClick.AddListener(delegate ()
{
SuperController.singleton.SaveStringIntoFile(path1, "neutral");
});
}
// Runs once when plugin loads - after Init()
protected void Start()
{
// show a message
SuperController.LogMessage(pluginLabelJSON.val + " Loaded");
}
// A Unity thing - runs every physics cycle
public void FixedUpdate()
{
// put code here
}
// Unity thing - runs every rendered frame
public void Update()
{
SwitchTr1();
SwitchPose1();
SuperController.LogMessage("running");
}
void SwitchPose1()
{
//path for the outpur file with expression
string expr1 = SuperController.singleton.ReadFileIntoString(path2);
//revome quote symbols if any
expr1 = expr1.Replace("\"", string.Empty);
if (temp2 != expr1)
{
//put a pose on Person according to Objective output
if (expr1 == "Take the last human survivor to the Resistance camp") {
myAction = containingAtom.GetStorableByID("plugin#" + 1 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
if (expr1 == "Upgrade your equipment at the Resistance camp") {
myAction = containingAtom.GetStorableByID("plugin#" + 4 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
if (expr1 == "Repel the attack of machine lifeforms on the resistance camp") {
myAction = containingAtom.GetStorableByID("plugin#" + 5 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
}
temp2 = expr1;
}
void SwitchTr1()
{
JSONStorable geo = containingAtom.GetStorableByID("geometry");
DAZCharacterSelector character = geo as DAZCharacterSelector;
GenerateDAZMorphsControlUI morphControl = character.morphsControlUI;
//reset morphs
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Excitement"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Angry"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Happy"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Confused"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Desire"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Contempt"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Fear"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 2"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - For Me"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 3"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("AA Taste Surprise"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("Curious1"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Bedroom Smile"); FaceMorph1.morphValue = 0f;
FaceMorph1 = morphControl.GetMorphByDisplayName("AAeyesquintbrowworry2"); FaceMorph1.morphValue = 0f;
//path for the outpur file with expression
string expr1 = SuperController.singleton.ReadFileIntoString(path1);
//revome quote symbols if any
expr1 = expr1.Replace("\"", string.Empty);
SuperController.LogMessage(expr1);
//put a morph on Person according to Classify output
if (expr1 == "admiration") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "amusement") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Excitement");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "anger") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Angry");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "annoyance") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "approval") {
//FaceMorph1 = morphControl.GetMorphByDisplayName("Happy");
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - For Me");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "caring") {
//FaceMorph1 = morphControl.GetMorphByDisplayName("AAeyesquintbrowworry2");
FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 2");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "confusion") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Confused");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "curiosity") {
//FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
FaceMorph1 = morphControl.GetMorphByDisplayName("Curious1");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "desire") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Desire");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disappointment") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disapproval") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Contempt");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "disgust") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Disgust");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "embarrassment") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "grief") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "gratitude") {
//FaceMorph1 = morphControl.GetMorphByDisplayName("AA Cute 3");
FaceMorph1 = morphControl.GetMorphByDisplayName("AA Taste Surprise");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "joy") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "love") {
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile");
FaceMorph1.morphValue = 0.6f;
}
if (expr1 == "nervousness") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Fear");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "neutral") {
//
}
if (expr1 == "optimism") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "pride") {
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Regal Smile");
FaceMorph1.morphValue = 0.6f;
}
if (expr1 == "realization") {
FaceMorph1 = morphControl.GetMorphByDisplayName("asco - Bedroom Smile");
FaceMorph1.morphValue = 0.4f;
}
if (expr1 == "relief") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Smile Open Full Face");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "remorse") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "sadness") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Sad");
FaceMorph1.morphValue = 0.3f;
}
if (expr1 == "surprise") {
FaceMorph1 = morphControl.GetMorphByDisplayName("Surprise");
FaceMorph1.morphValue = 0.3f;
}
}
}
}
classify_3.cs
Displaying classify_3.cs.The video at the link below shows an example of the result with lip&expression synch, as well as the scene change caused by the output of the Objective module:
This video shows the following:
- AI plays the role of an android, which must guide the player to the Resistance camp in order to improve equipment there and replenish supplies, which is necessary to complete the global goal;
- At first, the dialogue with the AI is in the "Take the last human survivor to the Resistance camp" state. The application is configured so that the AI will check the completion of the current task after every 3 requests;
- 1:18 After responding to the player’s next request, the AI begins checking the completion of the current goal “Take the last human survivor to the Resistance camp” (the notification “Checking for task completion” pops up);
- 1:28 The check completes with an affirmative result (true), the current AI goal changes to "Upgrade your equipment at the Resistance camp" and the application changes the scene for this dialogue state. In this case, the Person's pose simply changes, and an object [box] is added to the scene, indicating that this part of the story takes place in the Resistance camp. But you can, of course, come up with and implement something more complex and interactive.
The character look is 2B by Xspada. The idle character animation is obtained by using the Alive plugin by SPQR.
The following video shows another demo in which not only the Person's pose, but also the background image switches depending on the state of the task tree. The current task is indicated in an UIText and the current character expression is indicated in the bottom left:
4. Function calling
Let's look at how you can make the avatar (person atom) in VaM perform some action that was requested by the player from AI. For example, actions such as sitting on a chair or taking a book from a shelf. To do this, we will define in advance a set of actions (functions) available to the character, write down their names and descriptions in the SillyTavern script, and give the AI instructions for selecting one of these actions based on the content of the last phrases of the dialogue. The idea was adopted from Action List Manager (ALM) plugin for Voxta by vaan20.
In order to skip implementing a new extension for SillyTavern when testing this functionality, let’s modify the existing Objective module. To do this, we will edit the index.js script in the ‘..\SillyTavern\public\scripts\extensions\third-party\Extension-Objective’ directory.
Please note that the selection and configuration of the LLM in this case is carried out in the same way as when using the original Objective module. I got the best results using the aphrodite/derceto-labs/Psycho-Crystal-16B model. Although smaller models, such as koboldcpp/LLaMA2-13B-TiefighterLR or koboldcpp/Silicon-Maid-7B, also showed acceptable results.
We will process the response of the role-playing AI to select one of the functions using the GenerateQuietPrompt() procedure. When using this feature, SillyTavern makes a request to the LLM engine, but does not record the result in the chat history, which is useful for performing AI-assisted internal functions.
We will use the following addition to the main prompt:
JavaScript:
var prompt = `Pause your roleplay. Given the last reply from {{char}}, determine the best available function from the list that {{char}} should execute. If none of the available functions match, choose 'fn.noop()'. Your response should only contain the name of the function and nothing else.
### Input: The list of functions available to {{char}}:
- 'fn.noop()': None of the other functions matching the reply
- 'fn.equip_sword()': when {{char}} decides to equip the sword
- 'fn.equip_bow()': when {{char}} decides to equip the bow
- 'fn.equip_spear()': when {{char}} decides to equip the spear
- 'fn.unequip_all()': when {{char}} is asked to remove her current weapon
### Response:`;The ### Input tag contains all the functions available to the character that will be used in the scene. In this example, the functions are responsible for equipping various weapons (sword/bow/spear/nothing).
Next, change {{char}} in the prompt to the current character name, request an AI response via the GenerateQuietPrompt() function and save the response to a file.
JavaScript:
prompt = prompt.replace(/{{char}}/gi, name2);
//run the task completion check
const taskResponse = (await generateQuietPrompt(prompt)).toLowerCase();
Write1(taskResponse,'C:/.../VAM/Custom/Scripts/output4.txt');See below script index.js from the Objective directory modified this way:
JavaScript:
import { chat_metadata, callPopup, saveSettingsDebounced, is_send_press, getRequestHeaders, name2} from '../../../../script.js';
import { getContext, extension_settings, saveMetadataDebounced } from '../../../extensions.js';
import {
substituteParams,
eventSource,
event_types,
generateQuietPrompt,
} from '../../../../script.js';
import { registerSlashCommand } from '../../../slash-commands.js';
import { waitUntilCondition } from '../../../utils.js';
import { is_group_generating, selected_group } from '../../../group-chats.js';
const MODULE_NAME = 'Objective';
let taskTree = null;
let globalTasks = [];
let currentChatId = '';
let currentObjective = null;
let currentTask = null;
let checkCounter = 0;
let lastMessageWasSwipe = false;
let apiResult2 = null;
const defaultPrompts = {
'createTask': 'Pause your roleplay. Please generate a numbered list of plain text tasks to complete an objective. The objective that you must make a numbered task list for is: "{{objective}}". The tasks created should take into account the character traits of {{char}}. These tasks may or may not involve {{user}} directly. Include the objective as the final task.',
'checkTaskCompleted': 'Pause your roleplay. Determine if this task is completed: [{{task}}]. To do this, examine the most recent messages. Your response must only contain either true or false, and nothing else. Example output: true',
'currentTask': 'Your current task is [{{task}}]. Balance existing roleplay with completing this task.',
};
let objectivePrompts = defaultPrompts;
//###############################//
//# Task Management #//
//###############################//
// Return the task and index or throw an error
function getTaskById(taskId) {
if (taskId == null) {
throw 'Null task id';
}
return getTaskByIdRecurse(taskId, taskTree);
}
function getTaskByIdRecurse(taskId, task) {
if (task.id == taskId) {
return task;
}
for (const childTask of task.children) {
const foundTask = getTaskByIdRecurse(taskId, childTask);
if (foundTask != null) {
return foundTask;
}
}
return null;
}
function substituteParamsPrompts(content, substituteGlobal) {
content = content.replace(/{{objective}}/gi, currentObjective.description);
content = content.replace(/{{task}}/gi, currentTask.description);
if (currentTask.parent) {
content = content.replace(/{{parent}}/gi, currentTask.parent.description);
}
if (substituteGlobal) {
content = substituteParams(content);
}
return content;
}
// Call Quiet Generate to create task list using character context, then convert to tasks. Should not be called much.
async function generateTasks() {
const prompt = substituteParamsPrompts(objectivePrompts.createTask, false);
console.log('Generating tasks for objective with prompt');
toastr.info('Generating tasks for objective', 'Please wait...');
const taskResponse = await generateQuietPrompt(prompt);
// Clear all existing objective tasks when generating
currentObjective.children = [];
const numberedListPattern = /^\d+\./;
// Create tasks from generated task list
for (const task of taskResponse.split('\n').map(x => x.trim())) {
if (task.match(numberedListPattern) != null) {
currentObjective.addTask(task.replace(numberedListPattern, '').trim());
}
}
updateUiTaskList();
setCurrentTask();
console.info(`Response for Objective: '${currentObjective.description}' was \n'${taskResponse}', \nwhich created tasks \n${JSON.stringify(currentObjective.children.map(v => { return v.toSaveState(); }), null, 2)} `);
toastr.success(`Generated ${currentObjective.children.length} tasks`, 'Done!');
}
// Call Quiet Generate to check if a task is completed
async function checkTaskCompleted() {
// Make sure there are tasks
if (jQuery.isEmptyObject(currentTask)) {
return;
}
try {
// Wait for group to finish generating
if (selected_group) {
await waitUntilCondition(() => is_group_generating === false, 1000, 10);
}
// Another extension might be doing something with the chat, so wait for it to finish
await waitUntilCondition(() => is_send_press === false, 30000, 10);
} catch {
console.debug('Failed to wait for group to finish generating');
return;
}
checkCounter = $('#objective-check-frequency').val();
toastr.info('Checking for task completion.');
//insert the name of the current task into the defolt prompt
//const prompt = substituteParamsPrompts(objectivePrompts.checkTaskCompleted, false);
var prompt = `Pause your roleplay. Given the last reply from {{char}}, determine the best available function from the list that {{char}} should execute. If none of the available functions match, choose 'fn.noop()'. Your response should only contain the name of the function and nothing else.
### Input: The list of functions available to {{char}}:
- 'fn.noop()': None of the other functions matching the reply
- 'fn.equip_sword()': when {{char}} decides to equip the sword
- 'fn.equip_bow()': when {{char}} decides to equip the bow
- 'fn.equip_spear()': when {{char}} decides to equip the spear
- 'fn.unequip_all()': when {{char}} is asked to remove her current weapon
### Response:`;
prompt = prompt.replace(/{{char}}/gi, name2);
//run the task completion check
var taskResponse = (await generateQuietPrompt(prompt)).toLowerCase();
console.log('taskResponse0: ', taskResponse);
if (taskResponse.includes('noop')) taskResponse = 'fn.noop()';
else if (taskResponse.includes('equip_sword')) taskResponse = 'fn.equip_sword()';
else if (taskResponse.includes('equip_bow')) taskResponse = 'fn.equip_bow()';
else if (taskResponse.includes('equip_spear')) taskResponse = 'fn.equip_spear()';
else if (taskResponse.includes('unequip_all')) taskResponse = 'fn.unequip_all()';
else taskResponse = 'fn.noop()';
Write1(taskResponse,'C:/.../VAM/Custom/Scripts/output4.txt');
console.log('prompt1: ', prompt);
console.log('taskResponse1: ', taskResponse);
console.log('length1: ', taskResponse.length);
// Check response if task complete
if (taskResponse.includes('true') && taskResponse.includes('false') === false && taskResponse.length <= 25)
{
console.info(`Character determined task '${currentTask.description} is completed.`);
currentTask.completeTask();
} else if (!(taskResponse.includes('false'))) {
console.warn(`checkTaskCompleted response did not contain true or false. taskResponse: ${taskResponse}`);
} else {
console.debug(`Checked task completion. taskResponse: ${taskResponse}`);
}
}
function getNextIncompleteTaskRecurse(task) {
if (task.completed === false // Return task if incomplete
&& task.children.length === 0 // Ensure task has no children, it's subtasks will determine completeness
&& task.parentId !== '' // Must have parent id. Only root task will be missing this and we dont want that
) {
return task;
}
for (const childTask of task.children) {
if (childTask.completed === true) { // Don't recurse into completed tasks
continue;
}
const foundTask = getNextIncompleteTaskRecurse(childTask);
if (foundTask != null) {
return foundTask;
}
}
return null;
}
// Set a task in extensionPrompt context. Defaults to first incomplete
function setCurrentTask(taskId = null, skipSave = false) {
const context = getContext();
//console.log('test1');
// TODO: Should probably null this rather than set empty object
currentTask = {};
// Find the task, either next incomplete, or by provided taskId
if (taskId === null) {
currentTask = getNextIncompleteTaskRecurse(taskTree) || {};
} else {
currentTask = getTaskById(taskId);
}
// Don't just check for a current task, check if it has data
const description = currentTask.description || null;
if (description) {
const extensionPromptText = substituteParamsPrompts(objectivePrompts.currentTask, true);
// Remove highlights
$('.objective-task').css({ 'border-color': '', 'border-width': '' });
// Highlight current task
let highlightTask = currentTask;
while (highlightTask.parentId !== '') {
if (highlightTask.descriptionSpan) {
highlightTask.descriptionSpan.css({ 'border-color': 'yellow', 'border-width': '2px' });
}
const parent = getTaskById(highlightTask.parentId);
highlightTask = parent;
}
// Update the extension prompt
context.setExtensionPrompt(MODULE_NAME, extensionPromptText, 1, $('#objective-chat-depth').val());
console.info(`Current task in context.extensionPrompts.Objective is ${JSON.stringify(context.extensionPrompts.Objective)}`);
var arr3 = Object.values(currentTask);
Write1(JSON.stringify(arr3[1]),'C:/.../VAM/Custom/Scripts/output33.txt');
console.log('apiResult: ', apiResult2);
} else {
context.setExtensionPrompt(MODULE_NAME, '');
console.info('No current task');
}
// Save state if not skipping
if (!skipSave) {
saveState();
}
}
function getHighestTaskIdRecurse(task) {
let nextId = task.id;
for (const childTask of task.children) {
const childId = getHighestTaskIdRecurse(childTask);
if (childId > nextId) {
nextId = childId;
}
}
return nextId;
}
//###############################//
//# Task Class #//
//###############################//
class ObjectiveTask {
id;
description;
completed;
parentId;
children;
// UI Elements
taskHtml;
descriptionSpan;
completedCheckbox;
deleteTaskButton;
addTaskButton;
constructor({ id = undefined, description, completed = false, parentId = '' }) {
this.description = description;
this.parentId = parentId;
this.children = [];
this.completed = completed;
// Generate a new ID if none specified
if (id == undefined) {
this.id = getHighestTaskIdRecurse(taskTree) + 1;
} else {
this.id = id;
}
}
// Accepts optional index. Defaults to adding to end of list.
addTask(description, index = null) {
index = index != null ? index : index = this.children.length;
this.children.splice(index, 0, new ObjectiveTask(
{ description: description, parentId: this.id },
));
saveState();
}
getIndex() {
if (this.parentId !== null) {
const parent = getTaskById(this.parentId);
const index = parent.children.findIndex(task => task.id === this.id);
if (index === -1) {
throw `getIndex failed: Task '${this.description}' not found in parent task '${parent.description}'`;
}
return index;
} else {
throw `getIndex failed: Task '${this.description}' has no parent`;
}
}
// Used to set parent to complete when all child tasks are completed
checkParentComplete() {
let all_completed = true;
if (this.parentId !== '') {
const parent = getTaskById(this.parentId);
for (const child of parent.children) {
if (!child.completed) {
all_completed = false;
break;
}
}
if (all_completed) {
parent.completed = true;
console.info(`Parent task '${parent.description}' completed after all child tasks complated.`);
} else {
parent.completed = false;
}
}
}
// Complete the current task, setting next task to next incomplete task
completeTask() {
this.completed = true;
console.info(`Task successfully completed: ${JSON.stringify(this.description)}`);
this.checkParentComplete();
setCurrentTask();
updateUiTaskList();
}
// Add a single task to the UI and attach event listeners for user edits
addUiElement() {
const template = `
<div id="objective-task-label-${this.id}" class="flex1 checkbox_label">
<input id="objective-task-complete-${this.id}" type="checkbox">
<span class="text_pole objective-task" style="display: block" id="objective-task-description-${this.id}" contenteditable>${this.description}</span>
<div id="objective-task-delete-${this.id}" class="objective-task-button fa-solid fa-xmark fa-2x" title="Delete Task"></div>
<div id="objective-task-add-${this.id}" class="objective-task-button fa-solid fa-plus fa-2x" title="Add Task"></div>
<div id="objective-task-add-branch-${this.id}" class="objective-task-button fa-solid fa-code-fork fa-2x" title="Branch Task"></div>
</div><br>
`;
// Add the filled out template
$('#objective-tasks').append(template);
this.completedCheckbox = $(`#objective-task-complete-${this.id}`);
this.descriptionSpan = $(`#objective-task-description-${this.id}`);
this.addButton = $(`#objective-task-add-${this.id}`);
this.deleteButton = $(`#objective-task-delete-${this.id}`);
this.taskHtml = $(`#objective-task-label-${this.id}`);
this.branchButton = $(`#objective-task-add-branch-${this.id}`);
// Handle sub-task forking style
if (this.children.length > 0) {
this.branchButton.css({ 'color': '#33cc33' });
} else {
this.branchButton.css({ 'color': '' });
}
// Add event listeners and set properties
$(`#objective-task-complete-${this.id}`).prop('checked', this.completed);
$(`#objective-task-complete-${this.id}`).on('click', () => (this.onCompleteClick()));
$(`#objective-task-description-${this.id}`).on('keyup', () => (this.onDescriptionUpdate()));
$(`#objective-task-description-${this.id}`).on('focusout', () => (this.onDescriptionFocusout()));
$(`#objective-task-delete-${this.id}`).on('click', () => (this.onDeleteClick()));
$(`#objective-task-add-${this.id}`).on('click', () => (this.onAddClick()));
this.branchButton.on('click', () => (this.onBranchClick()));
}
onBranchClick() {
currentObjective = this;
updateUiTaskList();
setCurrentTask();
}
onCompleteClick() {
this.completed = this.completedCheckbox.prop('checked');
this.checkParentComplete();
setCurrentTask();
}
onDescriptionUpdate() {
this.description = this.descriptionSpan.text();
}
onDescriptionFocusout() {
setCurrentTask();
}
onDeleteClick() {
const index = this.getIndex();
const parent = getTaskById(this.parentId);
parent.children.splice(index, 1);
updateUiTaskList();
setCurrentTask();
}
onAddClick() {
const index = this.getIndex();
const parent = getTaskById(this.parentId);
parent.addTask('', index + 1);
updateUiTaskList();
setCurrentTask();
}
toSaveStateRecurse() {
let children = [];
if (this.children.length > 0) {
for (const child of this.children) {
children.push(child.toSaveStateRecurse());
}
}
return {
'id': this.id,
'description': this.description,
'completed': this.completed,
'parentId': this.parentId,
'children': children,
};
}
}
//###############################//
//# Custom Prompts #//
//###############################//
function onEditPromptClick() {
let popupText = '';
popupText += `
<div class="objective_prompt_modal">
<small>Edit prompts used by Objective for this session. You can use {{objective}} or {{task}} plus any other standard template variables. Save template to persist changes.</small>
<br>
<div>
<label for="objective-prompt-generate">Generation Prompt</label>
<textarea id="objective-prompt-generate" type="text" class="text_pole textarea_compact" rows="8"></textarea>
<label for="objective-prompt-check">Completion Check Prompt</label>
<textarea id="objective-prompt-check" type="text" class="text_pole textarea_compact" rows="8"></textarea>
<label for="objective-prompt-extension-prompt">Injected Prompt</label>
<textarea id="objective-prompt-extension-prompt" type="text" class="text_pole textarea_compact" rows="8"></textarea>
</div>
<div class="objective_prompt_block">
<label for="objective-custom-prompt-select">Custom Prompt Select</label>
<select id="objective-custom-prompt-select"><select>
</div>
<div class="objective_prompt_block">
<input id="objective-custom-prompt-new" class="menu_button" type="submit" value="New Prompt" />
<input id="objective-custom-prompt-save" class="menu_button" type="submit" value="Save Prompt" />
<input id="objective-custom-prompt-delete" class="menu_button" type="submit" value="Delete Prompt" />
</div>
</div>`;
callPopup(popupText, 'text');
populateCustomPrompts();
// Set current values
$('#objective-prompt-generate').val(objectivePrompts.createTask);
$('#objective-prompt-check').val(objectivePrompts.checkTaskCompleted);
$('#objective-prompt-extension-prompt').val(objectivePrompts.currentTask);
// Handle value updates
$('#objective-prompt-generate').on('input', () => {
objectivePrompts.createTask = $('#objective-prompt-generate').val();
});
$('#objective-prompt-check').on('input', () => {
objectivePrompts.checkTaskCompleted = $('#objective-prompt-check').val();
});
$('#objective-prompt-extension-prompt').on('input', () => {
objectivePrompts.currentTask = $('#objective-prompt-extension-prompt').val();
});
// Handle new
$('#objective-custom-prompt-new').on('click', () => {
newCustomPrompt();
});
// Handle save
$('#objective-custom-prompt-save').on('click', () => {
saveCustomPrompt();
});
// Handle delete
$('#objective-custom-prompt-delete').on('click', () => {
deleteCustomPrompt();
});
// Handle load
$('#objective-custom-prompt-select').on('change', loadCustomPrompt);
}
async function newCustomPrompt() {
const customPromptName = await callPopup('<h3>Custom Prompt name:</h3>', 'input');
if (customPromptName == '') {
toastr.warning('Please set custom prompt name to save.');
return;
}
if (customPromptName == 'default') {
toastr.error('Cannot save over default prompt');
return;
}
extension_settings.objective.customPrompts[customPromptName] = {};
Object.assign(extension_settings.objective.customPrompts[customPromptName], objectivePrompts);
saveSettingsDebounced();
populateCustomPrompts();
}
function saveCustomPrompt() {
const customPromptName = $('#objective-custom-prompt-select').find(':selected').val();
if (customPromptName == 'default') {
toastr.error('Cannot save over default prompt');
return;
}
Object.assign(extension_settings.objective.customPrompts[customPromptName], objectivePrompts);
saveSettingsDebounced();
populateCustomPrompts();
}
function deleteCustomPrompt() {
const customPromptName = $('#objective-custom-prompt-select').find(':selected').val();
if (customPromptName == 'default') {
toastr.error('Cannot delete default prompt');
return;
}
delete extension_settings.objective.customPrompts[customPromptName];
saveSettingsDebounced();
populateCustomPrompts();
loadCustomPrompt();
}
function loadCustomPrompt() {
const optionSelected = $('#objective-custom-prompt-select').find(':selected').val();
Object.assign(objectivePrompts, extension_settings.objective.customPrompts[optionSelected]);
$('#objective-prompt-generate').val(objectivePrompts.createTask);
$('#objective-prompt-check').val(objectivePrompts.checkTaskCompleted);
$('#objective-prompt-extension-prompt').val(objectivePrompts.currentTask);
}
function populateCustomPrompts() {
// Populate saved prompts
$('#objective-custom-prompt-select').empty();
for (const customPromptName in extension_settings.objective.customPrompts) {
const option = document.createElement('option');
option.innerText = customPromptName;
option.value = customPromptName;
option.selected = customPromptName;
$('#objective-custom-prompt-select').append(option);
}
}
//###############################//
//# UI AND Settings #//
//###############################//
const defaultSettings = {
currentObjectiveId: null,
taskTree: null,
chatDepth: 2,
checkFrequency: 3,
hideTasks: false,
prompts: defaultPrompts,
};
// Convenient single call. Not much at the moment.
function resetState() {
lastMessageWasSwipe = false;
loadSettings();
}
//
function saveState() {
const context = getContext();
if (currentChatId == '') {
currentChatId = context.chatId;
}
chat_metadata['objective'] = {
currentObjectiveId: currentObjective.id,
taskTree: taskTree.toSaveStateRecurse(),
checkFrequency: $('#objective-check-frequency').val(),
chatDepth: $('#objective-chat-depth').val(),
hideTasks: $('#objective-hide-tasks').prop('checked'),
prompts: objectivePrompts,
};
saveMetadataDebounced();
}
// Dump core state
function debugObjectiveExtension() {
console.log(JSON.stringify({
'currentTask': currentTask,
'currentObjective': currentObjective,
'taskTree': taskTree.toSaveStateRecurse(),
'chat_metadata': chat_metadata['objective'],
'extension_settings': extension_settings['objective'],
'prompts': objectivePrompts,
}, null, 2));
}
window.debugObjectiveExtension = debugObjectiveExtension;
// Populate UI task list
function updateUiTaskList() {
$('#objective-tasks').empty();
// Show button to navigate back to parent objective if parent exists
if (currentObjective) {
if (currentObjective.parentId !== '') {
$('#objective-parent').show();
} else {
$('#objective-parent').hide();
}
}
$('#objective-text').val(currentObjective.description);
if (currentObjective.children.length > 0) {
// Show tasks if there are any to show
for (const task of currentObjective.children) {
task.addUiElement();
}
} else {
// Show button to add tasks if there are none
$('#objective-tasks').append(`
<input id="objective-task-add-first" type="button" class="menu_button" value="Add Task">
`);
$('#objective-task-add-first').on('click', () => {
currentObjective.addTask('');
setCurrentTask();
updateUiTaskList();
});
}
}
function onParentClick() {
currentObjective = getTaskById(currentObjective.parentId);
updateUiTaskList();
setCurrentTask();
}
// Trigger creation of new tasks with given objective.
async function onGenerateObjectiveClick() {
await generateTasks();
saveState();
}
//Write to file
async function Write1(text,path) {
apiResult2 = await fetch('/api/extra/objective1', {
method: 'POST',
headers: getRequestHeaders(),
body: JSON.stringify({ text: text, path: path }),
});
//console.log('apitest');
//console.log('apiResult', apiResult1);
}
// Update extension prompts
function onChatDepthInput() {
saveState();
setCurrentTask(); // Ensure extension prompt is updated
}
function onObjectiveTextFocusOut() {
if (currentObjective) {
currentObjective.description = $('#objective-text').val();
saveState();
}
}
// Update how often we check for task completion
function onCheckFrequencyInput() {
checkCounter = $('#objective-check-frequency').val();
$('#objective-counter').text(checkCounter);
saveState();
}
function onHideTasksInput() {
$('#objective-tasks').prop('hidden', $('#objective-hide-tasks').prop('checked'));
saveState();
}
function loadTaskChildrenRecurse(savedTask) {
let tempTaskTree = new ObjectiveTask({
id: savedTask.id,
description: savedTask.description,
completed: savedTask.completed,
parentId: savedTask.parentId,
});
for (const task of savedTask.children) {
const childTask = loadTaskChildrenRecurse(task);
tempTaskTree.children.push(childTask);
}
return tempTaskTree;
}
function loadSettings() {
// Load/Init settings for chatId
currentChatId = getContext().chatId;
// Reset Objectives and Tasks in memory
taskTree = null;
currentObjective = null;
// Init extension settings
if (Object.keys(extension_settings.objective).length === 0) {
Object.assign(extension_settings.objective, { 'customPrompts': { 'default': defaultPrompts } });
}
// Bail on home screen
if (currentChatId == undefined) {
return;
}
// Migrate existing settings
if (currentChatId in extension_settings.objective) {
// TODO: Remove this soon
chat_metadata['objective'] = extension_settings.objective[currentChatId];
delete extension_settings.objective[currentChatId];
}
if (!('objective' in chat_metadata)) {
Object.assign(chat_metadata, { objective: defaultSettings });
}
// Migrate legacy flat objective to new objectiveTree and currentObjective
if ('objective' in chat_metadata.objective) {
// Create root objective from legacy objective
taskTree = new ObjectiveTask({ id: 0, description: chat_metadata.objective.objective });
currentObjective = taskTree;
// Populate root objective tree from legacy tasks
if ('tasks' in chat_metadata.objective) {
let idIncrement = 0;
taskTree.children = chat_metadata.objective.tasks.map(task => {
idIncrement += 1;
return new ObjectiveTask({
id: idIncrement,
description: task.description,
completed: task.completed,
parentId: taskTree.id,
});
});
}
saveState();
delete chat_metadata.objective.objective;
delete chat_metadata.objective.tasks;
} else {
// Load Objectives and Tasks (Normal path)
if (chat_metadata.objective.taskTree) {
taskTree = loadTaskChildrenRecurse(chat_metadata.objective.taskTree);
}
}
// Make sure there's a root task
if (!taskTree) {
taskTree = new ObjectiveTask({ id: 0, description: $('#objective-text').val() });
}
currentObjective = taskTree;
checkCounter = chat_metadata['objective'].checkFrequency;
// Update UI elements
$('#objective-counter').text(checkCounter);
$('#objective-text').text(taskTree.description);
updateUiTaskList();
$('#objective-chat-depth').val(chat_metadata['objective'].chatDepth);
$('#objective-check-frequency').val(chat_metadata['objective'].checkFrequency);
$('#objective-hide-tasks').prop('checked', chat_metadata['objective'].hideTasks);
$('#objective-tasks').prop('hidden', $('#objective-hide-tasks').prop('checked'));
setCurrentTask(null, true);
}
function addManualTaskCheckUi() {
$('#extensionsMenu').prepend(`
<div id="objective-task-manual-check-menu-item" class="list-group-item flex-container flexGap5">
<div id="objective-task-manual-check" class="extensionsMenuExtensionButton fa-regular fa-square-check"/></div>
Manual Task Check
</div>`);
$('#objective-task-manual-check-menu-item').attr('title', 'Trigger AI check of completed tasks').on('click', checkTaskCompleted);
}
jQuery(() => {
const settingsHtml = `
<div class="objective-settings">
<div class="inline-drawer">
<div class="inline-drawer-toggle inline-drawer-header">
<b>Objective</b>
<div class="inline-drawer-icon fa-solid fa-circle-chevron-down down"></div>
</div>
<div class="inline-drawer-content">
<label for="objective-text"><small>Enter an objective and generate tasks. The AI will attempt to complete tasks autonomously</small></label>
<textarea id="objective-text" type="text" class="text_pole textarea_compact" rows="4"></textarea>
<div class="objective_block flex-container">
<input id="objective-generate" class="menu_button" type="submit" value="Auto-Generate Tasks" />
<label class="checkbox_label"><input id="objective-hide-tasks" type="checkbox"> Hide Tasks</label>
</div>
<div id="objective-parent" class="objective_block flex-container">
<i class="objective-task-button fa-solid fa-circle-left fa-2x" title="Go to Parent"></i>
<small>Go to parent task</small>
</div>
<div id="objective-tasks"> </div>
<div class="objective_block margin-bot-10px">
<div class="objective_block objective_block_control flex1 flexFlowColumn">
<label for="objective-chat-depth">Position in Chat</label>
<input id="objective-chat-depth" class="text_pole widthUnset" type="number" min="0" max="99" />
</div>
<br>
<div class="objective_block objective_block_control flex1">
<label for="objective-check-frequency">Task Check Frequency</label>
<input id="objective-check-frequency" class="text_pole widthUnset" type="number" min="0" max="99" />
<small>(0 = disabled)</small>
</div>
</div>
<span> Messages until next AI task completion check <span id="objective-counter">0</span></span>
<div class="objective_block flex-container">
<input id="objective_prompt_edit" class="menu_button" type="submit" value="Edit Prompts" />
</div>
<hr class="sysHR">
</div>
</div>
</div>
`;
addManualTaskCheckUi();
$('#extensions_settings').append(settingsHtml);
$('#objective-generate').on('click', onGenerateObjectiveClick);
$('#objective-chat-depth').on('input', onChatDepthInput);
$('#objective-check-frequency').on('input', onCheckFrequencyInput);
$('#objective-hide-tasks').on('click', onHideTasksInput);
$('#objective_prompt_edit').on('click', onEditPromptClick);
$('#objective-parent').hide();
$('#objective-parent').on('click', onParentClick);
$('#objective-text').on('focusout', onObjectiveTextFocusOut);
loadSettings();
eventSource.on(event_types.CHAT_CHANGED, () => {
resetState();
});
eventSource.on(event_types.MESSAGE_SWIPED, () => {
lastMessageWasSwipe = true;
});
eventSource.on(event_types.MESSAGE_RECEIVED, () => {
if (currentChatId == undefined || jQuery.isEmptyObject(currentTask) || lastMessageWasSwipe) {
lastMessageWasSwipe = false;
return;
}
checkTaskCompleted();
if ($('#objective-check-frequency').val() > 0) {
// Check only at specified interval
if (checkCounter <= 0) {
//checkTaskCompleted();
}
checkCounter -= 1;
}
setCurrentTask();
$('#objective-counter').text(checkCounter);
});
registerSlashCommand('taskcheck', checkTaskCompleted, [], '– checks if the current task is completed', true, true);
});If everything works correctly, the AI response should contain only a name of one of the specified functions, as shown for example in the pictures below:
An example of Function calling working correctly with the aphrodite/derceto-labs/Psycho-Crystal-16B model
An example of Function calling working correctly with the koboldcpp/Silicon-Maid-7B model
In the script for processing SillyTavern responses in VaM, we will add the procedure SwitchWeapon1(), which reads the name of the function from the file “path3” (a variable containing the path to the file) and launches the corresponding trigger in VaM:
C#:
void SwitchWeapon1()
{
//path for the outpur file with expression
wnp1 = SuperController.singleton.ReadFileIntoString(path3);
//remove quote symbols if any
wnp1 = wnp1.Replace("\"", string.Empty);
wnp1 = wnp1.Replace("'", string.Empty);
if (temp2 != wnp1)
{
//put a weapon on Person according to Objective output
if (wnp1 == "fn.noop()") {
//
}
if (wnp1 == "fn.equip_sword()") {
myAction = containingAtom.GetStorableByID("plugin#" + 6 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
if (wnp1 == "fn.equip_bow()") {
myAction = containingAtom.GetStorableByID("plugin#" + 7 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
if (wnp1 == "fn.equip_spear()") {
myAction = containingAtom.GetStorableByID("plugin#" + 8 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
if (wnp1 == "fn.unequip_all()") {
myAction = containingAtom.GetStorableByID("plugin#" + 9 + "_MacGruber.Relay").GetAction("Trigger");
if (myAction != null) myAction.actionCallback();
}
}
temp2 = wnp1;
SuperController.LogMessage(wnp1);
}Add a call to function SwitchWeapon1() to the Update() function.
The video below shows an example of Function Calling in VaM based on requests to AI in SillyTavern:
In this video example, the player first asks the character to change her weapons to a bow, then remove all weapons, then equip a spear. The AI correctly calls the appropriate functions in accordance with the player's requests. The weapon assets are made by Torvald.
Another example shows the implementation of Function Calling in a scene with a nurse in a hospital. The functions are specified for situations where a nurse is interviewing a patient, conducting a physical examination, or performing treatment:
Prompt is similar to the previous example, but the following functions are specified:
### Input: The list of functions available to {{char}}:
- 'fn.noop()': None of the other functions matching the reply
- 'fn.interview_user()': when {{char}} asks about {{user}}'s health condition
- 'fn.checkup_user()': when {{char}} decides to perform a medical examination on {{user}}
- 'fn.treat_user()': when {{char}} decides to provide a medical treatment to {{user}}Model used in this example is aphrodite/jebcarter/psyonic-cetacean-20B:
jebcarter/psyonic-cetacean-20B · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
