This simple plugin allows you to send requests from VaM to a locally running Koboldcpp, and display and voice the responses using game audio sources. There are basic settings for AI roleplay. Manual and voice text input is available. The plugin also supports launching Kobold via in-game triggers (for example, when changing poses or clothing). This allows the AI to comment on the player's actions in the scene. It also supports running game triggers after AI response (such as changing facial expressions or starting animations).
The plugin can be added to either a Person atom or an AudioSource atom.
It is assumed that Koboldccp runs in Instruct mode. If koboldccp is running on the same computer as VaM, then in the KoboldAI URL field you need to enter the address “http://localhost:5001/api/v1/generate”. If the program is running on another computer in the local network, you need to enter the address of that local computer, for example, "http://192.168.0.107:5001/api/v1/generate".
The character's memory and author's note can be changed by editing the corresponding text files in the Custom/PluginData/koboldlink/ directory. Symbol {{char}} changes to character name, {{user}} changes to player name (specified in pref.json file).
Some plugin parameters are not available in the plugin interface window, but can be changed in the file "Custom/PluginData/koboldlink/pref.json" (for example, the path to the audio file that the TextAudioTool creates).
Include "$AI_text" term in an UIText atom name (the term can be changed in the pref.json file) and press "Set AI driving buttons" to it to make it a text field for the player to enter a message.
Include "$AI_resp" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the AI response.
Include "$AI_STT" term in an UIText atom name (the term can be changed in the pref.json file) to it to make it a text field for displaying the STT result (useful for spell checking before sending it to Kobold).
Turn on "SentimentHadlingOn" toggle to use the Sentiment (emotion) classification option. This will work only with the following fix for the TextAudioTool (replace the original files with the fixed ones):
github.com
Include "$AI_senti" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the sentiment. Face morphs for each sentiment can be chosen by editing "sentiments.json" file from the Custom/PluginData/koboldlink directory. The ID of a specific morph can be found by pressing the "Copy Uid" button in the "Male morphs"/"Female morphs" menu.
Turn on "CustomTriggerOn" toggle to use the Custom trigger option. To use this, add the MacGruber.Relay plugin (from LogicBricks) to the person atom and set there the actions that should be performed after receiving Kobold's response. Also enter the MacGruber.Relay plugin ID in the "Trigger Plugin ID" text field available in the KoboldLink plugin interface window.
Use "Send message (external trigger)" to trigger Kobold from a custom in-game trigger (for example, from another plugin). In this case the "External trigger prompt" variable will be send as a prompt.
Tips and known bugs and restrictions:
- You can use Plugin UI Undocker by Stopper to free up screen space when using this plugin's UI menu for text input;
- You may need to disable the "Auto Jaw Mouth Morphs" in the Auto Behaviours/Auto Systems menu to get the correct lip-synch animation;
- Also disable the "Enable auto-jaw animation" option for the VAMMoan plugin if it is used in the Person atom;
- You can change the pitch of the voice using the Pitch slider on the Head Audio tab;
- Before installing this updated version, please, delete the configuration files of the old version from the Custom/PluginData/koboldlink directory, otherwise it may cause an error;
- Tested with TheBloke/Kunoichi-7B-GGUF model and KoboldCpp v1.66.1;
- Avoid using special characters and emojis in your requests and appearing them in AI responses. This may cause errors. I added "Don't use any emoji" instruction to the author's note, which reduces the probability of emoji appearing in AI responses;
-If your PC or file system is slow (e.g. high system load or slow hard drive), it is possible that the audio file with the new AI response will not be able to load in time and the audio file with the previous response will be played instead. In this case, it is recommended to increase the playback delay set by the slider "Audio playback delay, s";
- If glitches and crashes continue despite the maximum audio playback delay being set, turn off the "SpeechOn" toggle. In this case, the audio file will not be played immediately after loading, which should improve the stability of the plugin. In this case, to listen the voicing of the last response, you can click the "Repeat AI response" button after the loading of the new audio file is apparently complete;
- I found that when using STT, it is best to wait 2-3 seconds after pressing "Microphone Record (STT)" button before speaking to get more accurate results;
- Connects to a local Koboldcpp API by itself. Connecting to AI Horde's API is possible using an additional proxy (please check https://github.com/Musa-step/koboldlink-tools for details).
The plugin can be added to either a Person atom or an AudioSource atom.
Example of using
Another example of using
The 110 Animated Sex Loops Beach scene by sxs4 is used in this example.
Another example of using
In this video, AI requests are sent both from the plugin interface and by pressing various UI buttons in the scene. The example in the video uses the SPQR Alive plugin to make the character follow the player. The character look is 2B by Xspada. The environment is from NieR and MvC ENV pack by neci.
Another example of using
Another example of using
In this video, AI requests are sent both from the plugin interface and by pressing various UI buttons in the scene. The example in the video uses the SPQR Alive plugin to make the character follow the player. The character look is 2B by Xspada. The environment is from NieR and MvC ENV pack by neci.
Koboldcpp is one of the most popular and easy-to-use programs for communicating with text generating artificial intelligence (large language model, LLM):
https://github.com/LostRuins/koboldcpp
koboldcpp Quickstart guide:
https://github.com/LostRuins/koboldcpp
koboldcpp Quickstart guide:
It is assumed that Koboldccp runs in Instruct mode. If koboldccp is running on the same computer as VaM, then in the KoboldAI URL field you need to enter the address “http://localhost:5001/api/v1/generate”. If the program is running on another computer in the local network, you need to enter the address of that local computer, for example, "http://192.168.0.107:5001/api/v1/generate".
From a performance point of view, it is better to run Kobold and VaM on different PCs in the same local network (then these two resource intensive applications will not slow each other down). You can connect two PCs to a local network using a cable, a Wi-Fi router or a mobile phone with a Wi-Fi router mode. When connecting both computers to a phone or a router, you need to make sure that they see each other (the names of both PCs must be displayed in the "Network" windows menu; in Windows 10, make the appropriate settings in the Network and Sharing Center). Then, to send a request to the Koboldcpp server on another PC, you need to enter the IP-address of that local computer to the "KoboldAI URL" instead of "localhost", for example, "http://192.168.0.107:5001/api/v1/generate". You can check the IP-address of a PC using the "ipconfig" command in the Command Prompt (cmd).
If the computer with Koboldcpp cannot be connected to the computer with VaM via a local network, you can use Cloudflare tunnel. In this case, when setting up Koboldcpp, click the Remote Tunnel checkbox. You can also use Cloudflare application directly.
If the link to the tunnel looks like, for example, "https://carlo-intimate-beaver-burden.trycloudflare.com", then you need to enter the following in the "KoboldAI URL" field: "http://carlo-intimate-beaver-burden.trycloudflare.com/api/v1/generate".
If the link to the tunnel looks like, for example, "https://carlo-intimate-beaver-burden.trycloudflare.com", then you need to enter the following in the "KoboldAI URL" field: "http://carlo-intimate-beaver-burden.trycloudflare.com/api/v1/generate".
Option 1: To voice AI responses with SAPI voices, you need to install SPQR TextAudioTool in the directory with VaM (tested on version 0.4, the TextAudioTool-main folder is saved in the root directory of VaM). The tool can be downloaded from the following link:
drive.google.com
Unzip the archive with the TextAudioTool directly into the VaM folder. The path that the plugin uses to access the audio file created by the TextAudioTool can be edited in the pref.json file. For Text-to-Speech (TTS) to work, you need to run the file "SPQR.TextAudioTool - (standalone).bat" (before or after starting VaM).
The voices available for the TextAudioTool can be checked by entering the request "http://127.0.0.1:7069/voices" into the browser search bar;
Option 2: Silero TTS is also available. For this option install and run SillyTavern Extras (module "silero-tts" must be enabled):
https://github.com/SillyTavern/SillyTavern-Extras
Tested with SillyTavern Extras folder saved in VaM root directory.
Also set the "TTS Provider" menu on the "TTS Settings" page to "SillyTavern Extras". Enter the voice name (e.g. "en_12") in the "Voice Name" text box.
If you want to slow down the speech rate when using Silero, you can change the tempo of the corresponding wav file sample (they are saved in SillyTavernExtras\tts_samples directory) using audio editing software (such as Audacity).
TextAudioTool-0.4.zip
drive.google.com
The voices available for the TextAudioTool can be checked by entering the request "http://127.0.0.1:7069/voices" into the browser search bar;
Option 2: Silero TTS is also available. For this option install and run SillyTavern Extras (module "silero-tts" must be enabled):
https://github.com/SillyTavern/SillyTavern-Extras
Tested with SillyTavern Extras folder saved in VaM root directory.
Also set the "TTS Provider" menu on the "TTS Settings" page to "SillyTavern Extras". Enter the voice name (e.g. "en_12") in the "Voice Name" text box.
If you want to slow down the speech rate when using Silero, you can change the tempo of the corresponding wav file sample (they are saved in SillyTavernExtras\tts_samples directory) using audio editing software (such as Audacity).
The character's memory and author's note can be changed by editing the corresponding text files in the Custom/PluginData/koboldlink/ directory. Symbol {{char}} changes to character name, {{user}} changes to player name (specified in pref.json file).
Some plugin parameters are not available in the plugin interface window, but can be changed in the file "Custom/PluginData/koboldlink/pref.json" (for example, the path to the audio file that the TextAudioTool creates).
Include "$AI_text" term in an UIText atom name (the term can be changed in the pref.json file) and press "Set AI driving buttons" to it to make it a text field for the player to enter a message.
Include "$AI_resp" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the AI response.
Include "$AI_STT" term in an UIText atom name (the term can be changed in the pref.json file) to it to make it a text field for displaying the STT result (useful for spell checking before sending it to Kobold).
Turn on "SentimentHadlingOn" toggle to use the Sentiment (emotion) classification option. This will work only with the following fix for the TextAudioTool (replace the original files with the fixed ones):
GitHub - Musa-step/koboldlink-tools: KoboldLinkTools project repository.
KoboldLinkTools project repository. Contribute to Musa-step/koboldlink-tools development by creating an account on GitHub.
github.com
Include "$AI_senti" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the sentiment. Face morphs for each sentiment can be chosen by editing "sentiments.json" file from the Custom/PluginData/koboldlink directory. The ID of a specific morph can be found by pressing the "Copy Uid" button in the "Male morphs"/"Female morphs" menu.
Turn on "CustomTriggerOn" toggle to use the Custom trigger option. To use this, add the MacGruber.Relay plugin (from LogicBricks) to the person atom and set there the actions that should be performed after receiving Kobold's response. Also enter the MacGruber.Relay plugin ID in the "Trigger Plugin ID" text field available in the KoboldLink plugin interface window.
Use "Send message (external trigger)" to trigger Kobold from a custom in-game trigger (for example, from another plugin). In this case the "External trigger prompt" variable will be send as a prompt.
Tips and known bugs and restrictions:
- You can use Plugin UI Undocker by Stopper to free up screen space when using this plugin's UI menu for text input;
- You may need to disable the "Auto Jaw Mouth Morphs" in the Auto Behaviours/Auto Systems menu to get the correct lip-synch animation;
- Also disable the "Enable auto-jaw animation" option for the VAMMoan plugin if it is used in the Person atom;
- You can change the pitch of the voice using the Pitch slider on the Head Audio tab;
- Before installing this updated version, please, delete the configuration files of the old version from the Custom/PluginData/koboldlink directory, otherwise it may cause an error;
- Tested with TheBloke/Kunoichi-7B-GGUF model and KoboldCpp v1.66.1;
TheBloke/Kunoichi-7B-GGUF · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
-If your PC or file system is slow (e.g. high system load or slow hard drive), it is possible that the audio file with the new AI response will not be able to load in time and the audio file with the previous response will be played instead. In this case, it is recommended to increase the playback delay set by the slider "Audio playback delay, s";
- If glitches and crashes continue despite the maximum audio playback delay being set, turn off the "SpeechOn" toggle. In this case, the audio file will not be played immediately after loading, which should improve the stability of the plugin. In this case, to listen the voicing of the last response, you can click the "Repeat AI response" button after the loading of the new audio file is apparently complete;
- I found that when using STT, it is best to wait 2-3 seconds after pressing "Microphone Record (STT)" button before speaking to get more accurate results;
- Connects to a local Koboldcpp API by itself. Connecting to AI Horde's API is possible using an additional proxy (please check https://github.com/Musa-step/koboldlink-tools for details).
Thank you for reviewing and for your recommendations! I will check your links.
No, there is no NSFW filter in koboldcpp! You can also choose your model: any model if you are running koboldcpp locally, or from a list if you are using AI Horde.
I'm not trying to replace Voxta. Voxta is a high quality project with a user-friendly interface. KoboldLink is just an experimental, glitchy plugin to try new ideas :D
About multimodal input, yes, great idea. I hope to add this feature someday.