This plugin allows you to send requests from VaM to a locally running Koboldcpp, as well as display and voice responses using in-game audio sources (there are several Text-to-Speech/TTS options, from simple to more hardware-intensive, but higher quality). Basic settings for AI roleplaying are available. Manual and voice input are supported. The plugin also supports launching Kobold via in-game triggers (for example, when changing poses or clothing). This allows the AI to comment on the player's actions in the scene. It also supports launching in-game triggers after AI response (such as changing facial expressions, poses or starting animations): simply upon a response, based on the mood of the response, or calling triggers from a predefined list based on a match between their descriptions and the AI's response content (aka "function calling").
The plugin can be added to either a Person atom or an AudioSource atom.
The character's memory and author's note can be changed by editing the corresponding text files in the Custom/PluginData/koboldlink/ directory. Symbol {{char}} changes to character name, {{user}} changes to player name (specified in pref.json file).
Some plugin parameters are not available in the plugin interface window, but can be changed in the file "Custom/PluginData/koboldlink/pref.json" (for example, the path to the audio file that the TextAudioTool creates).
Include "$AI_text" term in an UIText atom name (the term can be changed in the pref.json file) and press "Set AI driving buttons" to it to make it a text field for the player to enter a message.
Include "$AI_resp" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the AI response.
Include "$AI_STT" term in an UIText atom name (the term can be changed in the pref.json file) to it to make it a text field for displaying the STT result (useful for spell checking before sending it to Kobold).
Turn on "SentimentHadlingOn" toggle to use the Sentiment (emotion) classification option. This will work only with the following fork for the TextAudioTool:
github.com
Include "$AI_senti" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the sentiment. The facial morphs for each sentiment can be set from the plugin's UI or loaded from the "sentiments.json" file in the Custom/PluginData/koboldlink directory. The ID of a specific morph for this file can be found by pressing the "Copy Uid" button in the "Male morphs"/"Female morphs" menu.
Turn on "CustomTriggerOn" toggle to use the Custom trigger option (This trigger will work every time the character plays/repeats the response).
Use "Send message (external trigger)" to trigger Kobold from a custom in-game trigger (for example, from another plugin). In this case the "External trigger prompt" variable will be send as a prompt.
Tips and known bugs and restrictions:
- You can use Plugin UI Undocker by Stopper to free up screen space when using this plugin's UI menu for text input;
- You may need to disable the "Auto Jaw Mouth Morphs" in the Auto Behaviours/Auto Systems menu to get the correct lip-synch animation;
- Also disable the "Enable auto-jaw animation" option for the VAMMoan plugin if it is used in the Person atom;
- You can change the pitch of the voice using the Pitch slider on the Head Audio tab;
- Before installing this updated version, please, delete the configuration files of the old version from the Custom/PluginData/koboldlink directory, otherwise it may cause an error;
- Avoid using any brackets, special characters and emojis in your requests and appearing them in AI responses. This may cause errors. I added "Don't use any emoji" instruction to the author's note, which reduces the probability of emoji appearing in AI responses;
-If your PC or file system is slow (e.g. high system load or slow hard drive), it is possible that the audio file with the new AI response will not be able to load in time and the audio file with the previous response will be played instead. In this case, it is recommended to increase the playback delay set by the slider "Audio playback delay, s";
- I found that when using STT, it is best to wait 2-3 seconds after pressing "Microphone Record (STT)" button before speaking to get more accurate results.
The plugin can be added to either a Person atom or an AudioSource atom.
Example of using
Another example of using
Another example of using
Another example of using
Another example of using
Koboldcpp is one of the most popular and easy-to-use programs for communicating with text generating artificial intelligence (large language model, LLM):
https://github.com/LostRuins/koboldcpp
koboldcpp Quickstart guide:
The plugin sends requests to Koboldcpp in instruct mode. Therefore, instruction models are expected to work more correctly. However, chat models can also work well with the plugin if provided with a sample dialogue in instruct mode style.
This plugin was tested with koboldcpp-1.101.1 and the following models:
L3-8B-Stheno-v3.2's responses are found to be more creative and explicit, while Kunoichi-7B follows instructions more accurately.
https://github.com/LostRuins/koboldcpp
koboldcpp Quickstart guide:
The plugin sends requests to Koboldcpp in instruct mode. Therefore, instruction models are expected to work more correctly. However, chat models can also work well with the plugin if provided with a sample dialogue in instruct mode style.
This plugin was tested with koboldcpp-1.101.1 and the following models:
TheBloke/Kunoichi-7B-GGUF · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
bartowski/L3-8B-Stheno-v3.2-GGUF · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
Option 1: Local host. If koboldccp is running on the same computer as VaM, then in the KoboldAI URL field you need to enter the address “http://localhost:5001/api/v1/generate”.
However, from a performance point of view, it is better to run Kobold and VaM on different PCs (then these two resource intensive applications will not slow each other down).
Option 2: Local network connection. You can connect two PCs to a local network using a cable, a Wi-Fi router or a mobile phone with a Wi-Fi router mode. When connecting both computers to a phone or a router, you need to make sure that they see each other (the names of both PCs must be displayed in the "Network" windows menu; in Windows 10, make the appropriate settings in the Network and Sharing Center). Then, to send a request to the Koboldcpp server on another PC, you need to enter the IP-address of that local computer to the "KoboldAI URL" instead of "localhost", for example, "http://192.168.0.107:5001/api/v1/generate". You can check the IP-address of a PC using the "ipconfig" command in the Command Prompt (cmd).
Option 3: Cloudflare tunnel. If the computer with Koboldcpp cannot be connected to the computer with VaM via a local network, you can use Cloudflare tunnel. In this case, when setting up Koboldcpp, click the Remote Tunnel checkbox. You can also use Cloudflare application directly. If the link to the tunnel looks like, for example, "https://carlo-intimate-beaver-burden.trycloudflare.com", then you need to enter the following in the "KoboldAI URL" field: "http://carlo-intimate-beaver-burden.trycloudflare.com/api/v1/generate".
Option 4: AI Horde. Connecting to AI Horde's API is possible using an additional proxy (please check https://github.com/Musa-step/koboldlink-tools for details).
Option 5: Run on RunPod. KoboldCpp can be used on RunPod cloud GPUs. In this case, the plugin can be connected to it via an http address.
However, from a performance point of view, it is better to run Kobold and VaM on different PCs (then these two resource intensive applications will not slow each other down).
Option 2: Local network connection. You can connect two PCs to a local network using a cable, a Wi-Fi router or a mobile phone with a Wi-Fi router mode. When connecting both computers to a phone or a router, you need to make sure that they see each other (the names of both PCs must be displayed in the "Network" windows menu; in Windows 10, make the appropriate settings in the Network and Sharing Center). Then, to send a request to the Koboldcpp server on another PC, you need to enter the IP-address of that local computer to the "KoboldAI URL" instead of "localhost", for example, "http://192.168.0.107:5001/api/v1/generate". You can check the IP-address of a PC using the "ipconfig" command in the Command Prompt (cmd).
Option 3: Cloudflare tunnel. If the computer with Koboldcpp cannot be connected to the computer with VaM via a local network, you can use Cloudflare tunnel. In this case, when setting up Koboldcpp, click the Remote Tunnel checkbox. You can also use Cloudflare application directly. If the link to the tunnel looks like, for example, "https://carlo-intimate-beaver-burden.trycloudflare.com", then you need to enter the following in the "KoboldAI URL" field: "http://carlo-intimate-beaver-burden.trycloudflare.com/api/v1/generate".
Option 4: AI Horde. Connecting to AI Horde's API is possible using an additional proxy (please check https://github.com/Musa-step/koboldlink-tools for details).
Option 5: Run on RunPod. KoboldCpp can be used on RunPod cloud GPUs. In this case, the plugin can be connected to it via an http address.
Option 1: To voice AI responses with SAPI voices, you need to install SPQR TextAudioTool in the directory with VaM (tested on version 0.4, the TextAudioTool-main folder is saved in the root directory of VaM). The tool can be downloaded from the following link:
drive.google.com
Unzip the archive with the TextAudioTool directly into the VaM folder. The path that the plugin uses to access the audio file created by the TextAudioTool can be edited in the pref.json file. For Text-to-Speech (TTS) to work, you need to run the file "SPQR.TextAudioTool - (standalone).bat" (before or after starting VaM).
The voices available for the TextAudioTool can be checked by entering the request "http://127.0.0.1:7069/voices" into the browser search bar;
Option 2: Silero TTS is also available. For this option install and run SillyTavern Extras (module "silero-tts" must be enabled):
https://github.com/SillyTavern/SillyTavern-Extras
Tested with SillyTavern-Extras folder saved in VaM root directory.
Also set the "TTS Provider" menu on the "TTS Options" page to "SillyTavern Extras". Enter the voice name (e.g. "en_12") in the "Voice Name" text box.
If you want to slow down the speech rate when using Silero, you can change the tempo of the corresponding wav file sample (they are saved in SillyTavernExtras\tts_samples directory) using audio editing software (such as Audacity).
Please note that SillyTavern-Extras is written in an outdated version of Python and is currently discontinued, so issues with its installation are possible. If you get errors like "ModuleNotFoundError: No module named xxx" when trying to run it, you will need to install the missing Python modules manually via the command prompt. The list of required modules can be found at the beginning of the server.py file in the root folder of SillyTavern-Extras. If an error occurs like "AttributeError: module 'torchaudio' has no attribute 'set_audio_backend'", then you need to manually delete the line "torchaudio.set_audio_backend("soundfile")" from the file "...\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\silero_api_server\tts.py".
Option 3: Built-in Kobold TTS is also available.
Set the "TTS Provider" menu on the "TTS Options" page to "Koboldcpp". Enter the voice name (e.g. "cheery") in the "Voice Name" text box.
TextAudioTool-0.4.zip
drive.google.com
The voices available for the TextAudioTool can be checked by entering the request "http://127.0.0.1:7069/voices" into the browser search bar;
Option 2: Silero TTS is also available. For this option install and run SillyTavern Extras (module "silero-tts" must be enabled):
https://github.com/SillyTavern/SillyTavern-Extras
Tested with SillyTavern-Extras folder saved in VaM root directory.
Also set the "TTS Provider" menu on the "TTS Options" page to "SillyTavern Extras". Enter the voice name (e.g. "en_12") in the "Voice Name" text box.
If you want to slow down the speech rate when using Silero, you can change the tempo of the corresponding wav file sample (they are saved in SillyTavernExtras\tts_samples directory) using audio editing software (such as Audacity).
Please note that SillyTavern-Extras is written in an outdated version of Python and is currently discontinued, so issues with its installation are possible. If you get errors like "ModuleNotFoundError: No module named xxx" when trying to run it, you will need to install the missing Python modules manually via the command prompt. The list of required modules can be found at the beginning of the server.py file in the root folder of SillyTavern-Extras. If an error occurs like "AttributeError: module 'torchaudio' has no attribute 'set_audio_backend'", then you need to manually delete the line "torchaudio.set_audio_backend("soundfile")" from the file "...\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\silero_api_server\tts.py".
Option 3: Built-in Kobold TTS is also available.
Set the "TTS Provider" menu on the "TTS Options" page to "Koboldcpp". Enter the voice name (e.g. "cheery") in the "Voice Name" text box.
The character's memory and author's note can be changed by editing the corresponding text files in the Custom/PluginData/koboldlink/ directory. Symbol {{char}} changes to character name, {{user}} changes to player name (specified in pref.json file).
Some plugin parameters are not available in the plugin interface window, but can be changed in the file "Custom/PluginData/koboldlink/pref.json" (for example, the path to the audio file that the TextAudioTool creates).
Include "$AI_text" term in an UIText atom name (the term can be changed in the pref.json file) and press "Set AI driving buttons" to it to make it a text field for the player to enter a message.
Include "$AI_resp" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the AI response.
Include "$AI_STT" term in an UIText atom name (the term can be changed in the pref.json file) to it to make it a text field for displaying the STT result (useful for spell checking before sending it to Kobold).
Turn on "SentimentHadlingOn" toggle to use the Sentiment (emotion) classification option. This will work only with the following fork for the TextAudioTool:
GitHub - Musa-step/koboldlink-tools: KoboldLinkTools project repository.
KoboldLinkTools project repository. Contribute to Musa-step/koboldlink-tools development by creating an account on GitHub.
github.com
Include "$AI_senti" term in an UIText atom name (the term can be changed in the pref.json file) to make it a text field for displaying the sentiment. The facial morphs for each sentiment can be set from the plugin's UI or loaded from the "sentiments.json" file in the Custom/PluginData/koboldlink directory. The ID of a specific morph for this file can be found by pressing the "Copy Uid" button in the "Male morphs"/"Female morphs" menu.
Turn on "CustomTriggerOn" toggle to use the Custom trigger option (This trigger will work every time the character plays/repeats the response).
Use "Send message (external trigger)" to trigger Kobold from a custom in-game trigger (for example, from another plugin). In this case the "External trigger prompt" variable will be send as a prompt.
AI can run triggers in the scene assigned to UIbutton atoms based on an analysis of last requests and responses (also known as "function calling"). This feature can be used to issue commands to the AI that the character will perform in the scene (move somewhere, change pose, change clothes, etc.)
Turn on “the FunctionCallingOn” toggle (page "AI triggered actions") to enable function calling. After responding to the primary chat request, the AI will be prompted to select the function that best matches the last request-response pair, and that function will be launched. To add a new function to the list of functions available to AI, use a separate UIbutton atom. The name must be like “fn.xxx()”, where “xxx” part is arbitrary. Set the name of the new function as the button name. Add a description of the function to the button caption. If you want to use only part of the button caption text, enclose that part in square brackets, like [...]. Parameter “functioncalling chatdepth” determines how many request-response pairs will be used in the prompt.
Please check this video guide for more information:
Turn on “the FunctionCallingOn” toggle (page "AI triggered actions") to enable function calling. After responding to the primary chat request, the AI will be prompted to select the function that best matches the last request-response pair, and that function will be launched. To add a new function to the list of functions available to AI, use a separate UIbutton atom. The name must be like “fn.xxx()”, where “xxx” part is arbitrary. Set the name of the new function as the button name. Add a description of the function to the button caption. If you want to use only part of the button caption text, enclose that part in square brackets, like [...]. Parameter “functioncalling chatdepth” determines how many request-response pairs will be used in the prompt.
Please check this video guide for more information:
Tips and known bugs and restrictions:
- You can use Plugin UI Undocker by Stopper to free up screen space when using this plugin's UI menu for text input;
- You may need to disable the "Auto Jaw Mouth Morphs" in the Auto Behaviours/Auto Systems menu to get the correct lip-synch animation;
- Also disable the "Enable auto-jaw animation" option for the VAMMoan plugin if it is used in the Person atom;
- You can change the pitch of the voice using the Pitch slider on the Head Audio tab;
- Before installing this updated version, please, delete the configuration files of the old version from the Custom/PluginData/koboldlink directory, otherwise it may cause an error;
- Avoid using any brackets, special characters and emojis in your requests and appearing them in AI responses. This may cause errors. I added "Don't use any emoji" instruction to the author's note, which reduces the probability of emoji appearing in AI responses;
-If your PC or file system is slow (e.g. high system load or slow hard drive), it is possible that the audio file with the new AI response will not be able to load in time and the audio file with the previous response will be played instead. In this case, it is recommended to increase the playback delay set by the slider "Audio playback delay, s";
- I found that when using STT, it is best to wait 2-3 seconds after pressing "Microphone Record (STT)" button before speaking to get more accurate results.
Thank you for reviewing and for your recommendations! I will check your links.
No, there is no NSFW filter in koboldcpp! You can also choose your model: any model if you are running koboldcpp locally, or from a list if you are using AI Horde.
I'm not trying to replace Voxta. Voxta is a high quality project with a user-friendly interface. KoboldLink is just an experimental, glitchy plugin to try new ideas :D
About multimodal input, yes, great idea. I hope to add this feature someday.