Question How to Attach a Script in VAM

[Hedgepig]

To keep my old brain alive I've decided to take a course in Windows Speech Recognition.


Could someone kindly walk me through the correct process of attaching a VS script to a character in VAM? If you can, this will save me the trouble of spending hours and hours trying figure out how to do this and I can use that time to create a free plugin for VAM.

Please keep your explanation as simple as you can. In the past, I've asked Unity asset devs to help, and sometimes I've had no idea what they trying to tell me. This is not their fault, I don't speak fluent high-level code.

Can I use Visual Studio to write my code?
What type of code does VAM use?
I need to work in C# and XML, will this be a problem in VAM?
Do I need to make a special file for the code, or does one already exist?

 

[MacGruber]

Can I use Visual Studio to write my code?

You can, you don't have to. I'm just using Notepad++ because I'm too lazy to setup a project for just a few files.
There is a plugin template for VSCode (which is not Visual Studio) here: https://github.com/acidbubbles/vam-plugin-template

What type of code does VAM use?

The compiler used for plugins is "DynamicCSharp", which is apparently a wrapper for some version of the Roslyn compiler. So, it's C#, exact version is unclear.

I need to work in C# and XML, will this be a problem in VAM?

Plugins are C#, but you got to work with the restrictions of the Unity engine as well as additional security restrictions by VaM. For example everthing that has to do with Reflection is blocked, as are file access utilities. You can however access files through VaM's wrappers, which restrict you to certain folders.
Not sure you got access to an XML parser, and VaM does not recognize the file type, so handling dependencies with VAR packages is not automatic.

Do I need to make a special file for the code, or does one already exist?

In the end you need a *.cs file containing your source code. It needs a class that derives from MVRScript, which again is a Unity MonoBehaviour. If you want more than one file, you can create a *.cslist file, which is just a list of your *.cs files, plain and simple.
For your first steps I recommend simply looking at existing plugins by other creators. Start simple, work your way up.

There is no proper VaM documentation worth mentioning. However, you can use ILSpy decompiler to open VaM_Data\Managed\Assembly-CSharp.dll, it contains VaM's source code. Source code is always the best documentation, because it does not lie. https://github.com/icsharpcode/ILSpy/releases
When using VisualStudio (or similar IDE) you would want to include some DLLs of that directory in the project references, so the IDE can help you properly with compiler errors, auto completion and so on.



Regarding Windows SpeechRecognition, note that there is a Unity wrapper for that you can use. There are also already several VaM plugins around using that. For example my own, which also has a solution for the XML problem:

SpeechRecognition - Plugins -

Recognize simple keyword lists as well as more complex grammar using Windows 10 and Unity's speech recognition features. When starting with this project the assumption was I can wrap the simple looking API in a plugin and build a simple demo...

hub.virtamate.com

 

[Hedgepig]

Hi MG,

Thank you so much for the detailed reply! Hmm, looks daunting. Like you say, start simple and add a layer of difficulty at a time.

I have already tried your plugin, and VRifter's; they're both great assets. And, thank you for making them. I wish I had learned to code when I was younger!

So, I'm going to follow a standard chatbot writing practice and map out a simple branching narrative and see what I can do with the plugins at first, before attempting to write my own code. The plugins might be sufficient for what I want to do. By the way, the writers the for big players in the chatbot field (can't name them) use Whimsical for visualizing and mapping the conversation flows. It's perfect for the job. Two free files before you need to pay btw.

I know you can only emulate a chatbot with the VAM plugins, but having extensively tested IBM Watson Assistant in IBM Cloud and in Unity, its not all the IBM marking hype suggests. They had to stop calling it a 'cognitive' machine, because many users point out that this is misleading about it's actual capabilities. Great for selling pizza or taking a user's bank details, but not so good in simulated everyday conversation, even combined with Discovery. I don't (in conversation) think its NLP is any more accurate than keywords and grammar, its just a wider semantic funnel, a bigger spider's web to catch the user's intents. It's very easy to trip up a $3000.00 per month-IBM driven commercial bot in directed conversation and get it hit default answers. So, I'm wagering that for small scale interaction, your plugins might have 99% of the high end commercial bots' capabilities, as long as the users have a list of keywords and don't expect too much. As I wrote in a previous post, an in-game, cloud-based bot will take perhaps 15 seconds and more to process a longer question and return a response, which repeatedly breaks the player's immersion in the game. Which is one explanation as to why the great AI conversational revolution never happened in games when IBM launched its Unity SDK a few years back, and , why it's now deprecated from the asset store. The other reason is, bot's can't cope with, or they break, when responding to multilingual, regional accent voice input. All the expensive commercial bots will try to shift a user into texting if they are have voice comprehension difficulties. No good in an intimate scene , is it?

So be proud, because what we have here in VAM combined with other plugins can do most of what a commercial bot can do in-game when its running a model. It's a question of figuring out how to pull it all together in VAM.

I'll just be content to be repeatedly humbled by what people, like yourself, who know their code, can achieve!

Fantastic talking to you guys, and thanks for the valuable reply. Off to map out a timeline for the speech recognition plugin.

 

[MacGruber]

Yes, I suspect you can get relatively far just with LogicBricks, SpeechRecognition and some pre-recorded speech lines.
If you haven't already, I recommend to check out the two LogicBricks StateMachine tutorials (Basics + Advanced):

Guides - MacGruber's Video Tutorials

This page will gather my hopefully growing collection of video tutorials about various VaM topics. "Pay after watching, IF you liked it!" Producing quality video tutorials consumes a LOT of time. Just producing 10min of video takes about 6-8...

hub.virtamate.com

As well as my CyberNight scene:

CyberNight - Scenes -

Separate demo scene showing off new features that came with LogicBricks 7, which help with building scenes with branching multiple-choice dialogs. Please excuse the poor dialog, I'm a coder, not a Hollywood writer. Anyway, you got a couple of...

hub.virtamate.com

Instead of buttons for input you could just use the SpeechRecognition plugin and instead of text in speech bubbles, you play pre-recorded audio lines.
One potential problem is that I could not yet figure out how to make the GrammarRecognizer detect "something was said, but it did not match any rule I know" so you could have some kind of generic response.

 

[Hedgepig]

"something was said, but it did not match any rule I know"

This is generally called a 'default answer'. In a conversational flow, defaults are always the buffers at the end of the track, the responses of last resort. It's best to have a maybe fifteen and randomize them. " Sorry, I didn't get that." " WTF dude, is this the Turing Test or something !?" etc.

From default answer you can return the flow to the user via distraction, ( standard chatbot trick) Bot: "Huh? I didn't understand your question." Bot: " So, er, you were telling me you studied media studies, that's just... so interesting..." Trigger yawn and eye-glaze.

There is a way to create default answers and distractions in AIML, which I have promptly forgotten, but I think the basic default answer looks something like this :

<category>
<pattern>*</pattern>
<template>I have no answer for that.</template>
</category>

Is there something like a default answer in C#?

****

The great thing about AIML is that it is so well-documented. There's a FREE 3hr course! Free Artificial Intelligence Tutorial - Artificial Intelligence Markup Language (AIML) | Udemy - The course narrator created Mitsuku.

 

[MacGruber]

Is there something like a default answer in C#?

This isn't a C# issue. Or LogicBricks issue. The problem is how a rule with the grammar XML for the Windows SpeechReecognition needs to written to account for this. There is a special rule with the keyword "GARBAGE" you can define, which is possible intended for this kind of thing. However, in my experiments it did not work as expected, it basically seems to make the system ignore any other rule.

 

[Hedgepig]

One final thought, I have no idea how this works but the Chatbot for Unity uses something called 'Jurassic' to get the AIML to work in Unity. Which is great because I have some custom code to make RTVoice to speak the bot's text responses, then I have SALSA to lipsync that RTV spoken response. You probably cant do Text-to-Speech in VAM, yet. So that a non-starter. However, the AIML bot can respond with rich media, but, bangs head on desk, as far as I can see, only images GIF and video, not MP3! This is always the same, building bots is like walking around in a maze, you get so far and hit a dead end.

 

[Hedgepig]

Hmm, no default, that's so weird.

I'm going to do a Windows Speech Recognition course. I've just double-checked , it has a section called 'Repeats and Wildcards', which should cover default answers. Every functional interactive system has to have an 'is everything else' response. I'd be surprised if Windows didn't. Like everything else about bots, it's not going to be obvious.

In that course there's also a very interesting section called 'Choices', so it looks like we can do branching and possibly complex narratives.

There was no way I was going to do the course, and pay for it, unless I knew could use the bot in VAM.

Let me do the course and I'll see what I can come up with in a few weeks.

BTW, what kind of response times have you been getting from WSR?

 

[MacGruber]

BTW, what kind of response times have you been getting from WSR?

Like a second or so after rule match.

 

[Hedgepig]

Excellent!

IBM Watson-Unity-lipsynced: two-three short sentences, up to 15-20 seconds response time.

IBM Watson cost- $80.00+, per month. For paid IBM Cloud account incl. text-to-speech and voices.

Your plugin: Responds in a couple of seconds.

Free to use.

No cloud. Less things to break.

All private on desktop.

Win,
win,
win.

As I said before, human nature, put a group of highly paid developers together and they'll string out the job for years.

Put a bunch of misfits, like myself, together in a sex simulator and they'll crack the problem in months.

 

[Hedgepig]

This isn't a C# issue. Or LogicBricks issue. The problem is how a rule with the grammar XML for the Windows SpeechReecognition needs to written to account for this. There is a special rule with the keyword "GARBAGE" you can define, which is possible intended for this kind of thing. However, in my experiments it did not work as expected, it basically seems to make the system ignore any other rule.

I sat through an incredibly tedious hour of a 'how to build a speech recognition engine', training vid.

The default answer is an:

if else " I have no answer for that question". (Or variation of that response)- the key point is the if else.

No idea where it goes in engine, because most unhelpfully, the video training instructor doesn't show numbers to indicate on which lines the code is meant to go, and, by this time he's already written about fifty lines of the SR engine, and has started adding keywords and grammars..

I can send you a screen shot or vid if you like. It will probably make clear and perfect make sense to you.