Plugins SallySoundFX

[FighterCNash]

SALLY I LOVE YOU THANKS FOR LET ME KNOW!!!! I WILL DOWNLOAD AND TEST IT NOW!

 

[MonsterShinkai]

@MonsterShinkai @SPQR @FighterCNash @C&G-STUDIO

Releasing SallySoundFX 2 quietly for now on my (free and public) Patreon.
Just in case I F'ed up something I will wait before I upload this to the Hub.
This will hopefully be the foundation for many cool Visualizations.

SallySoundFX 2.00 Release

I've tested long enough. Only visual bugs remain and performance is good. Everything "should" work.
Feedback welcome of course! And please be critical! I can take it.

Hope this does not disappoint people because it still looks 'basic'. The potential of all the crazy stuff you can do with this counts. If I would have VR hardware I'd try to make a beat saber Plugin ... that's complicated, but possible. However it would never be 100% the same, since whatever object you spawn to slash is generated from a 'live' signal. Just some thoughts ...

Todo-list: more visualizations *doh*, custom shaders, reflective materials, learn about advanced mesh rendering for even more performance

This was the first "working" version a few days ago. Later I found out it leaked memory like crazy. (fixed now ofc)

Something I found. I have a scene with 6 AVSLAVE plugins. When I change texture on 1, sometimes texture is removed from another.

 

[Sally Whitemane]

Something I found. I have a scene with 6 AVSLAVE plugins. When I change texture on 1, sometimes texture is removed from another.

Yep, that is not suppose to happen. Fortunately easy to reproduce. If the "Load Texture"-Button gets a new valid Texture back from the VaM-Filebrowser I destroy the previous one in an attempt to keep memory clean. I expected it it be instanced from the other Plugins and so I was very aggressive about keeping memory free.

Concerning what you (MonsterShinkai) wrote on Discord DM

One thing that came to mind, it would be nice if each tile was scalable.

That should be no problem to add. Was thinking about a drop-down menu similar to the Color Logic.
From there you can pick from various "Geometry Logic" behaviors. Scaling the tiles would then be one of them.

And also if opacity was able to be controlled.

That's probably something to look into when I investigate Shaders and Materials more. Currently the Visualizations use the "Sprites/Default" Shader. Unlike other typical VaM Materials the main reason to use this one is - it uses Vertex-Colors. So each geometry point has a color assigned. It is always 'unlit' - no shadows. And it's faster to render compared to the "Standard" Shader where colors come from the the material-color. Vertex-Colors are nice for Visualizations because the color "fine control" for each point.

What would happen if I changed the Shader to "Standard"?
It would need light to become visible, it would have shadows, but no color (or material color only).
The Visualizations really need a good custom Shader.
That would open up the floodgates for all kind of new customization options.

I've experimented with a simple custom one that I created with a Shader Editor named Amplify. At one point I had a version where I loaded this one ...

... from an AssetBundle. It worked, but to be honest I absolutely hate AssetBundles. I removed that code part again. I get why they are useful for game-development, but I just find the workflow to create them super annoying, they add so many additional potential points of failure and they are more or less black box once packed. If I can avoid them I will avoid them. Not sure whether that will be possible with Shaders to load them "directly" into VaM.

I thought of that beat saber thing too and if it helps i think the easiest way to do it is to have a 'learn song' functionality. The user would just let the song play once, you record the data in a json and then it would be used in the game, like at 0:30s in the song you would start popping in atoms based on the data for 0:25s or something like that

Hmm that reminds me, I think the VaM integrated RythymTool does something similar when a audio file is loaded.
Could try to "steal" the required data from that one too. I think that should be 100% the same data every time it loads - as long as the audio file is the same.

Meanwhile more experiments ... back to cave now.

 

[MonsterShinkai]

Thanks for the detailed explanation and for your consideration of the suggestions @Sally Whitemane. Also that AVGrid is wow!

 

[FighterCNash]

I love this plugin Sally! Used 2 of your VA in a single scene already this is so good!

 

[FighterCNash]

Yep, that is not suppose to happen. Fortunately easy to reproduce. If the "Load Texture"-Button gets a new valid Texture back from the VaM-Filebrowser I destroy the previous one in an attempt to keep memory clean. I expected it it be instanced from the other Plugins and so I was very aggressive about keeping memory free.

Concerning what you (MonsterShinkai) wrote on Discord DM

That should be no problem to add. Was thinking about a drop-down menu similar to the Color Logic.
From there you can pick from various "Geometry Logic" behaviors. Scaling the tiles would then be one of them.


That's probably something to look into when I investigate Shaders and Materials more. Currently the Visualizations use the "Sprites/Default" Shader. Unlike other typical VaM Materials the main reason to use this one is - it uses Vertex-Colors. So each geometry point has a color assigned. It is always 'unlit' - no shadows. And it's faster to render compared to the "Standard" Shader where colors come from the the material-color. Vertex-Colors are nice for Visualizations because the color "fine control" for each point.

What would happen if I changed the Shader to "Standard"?
It would need light to become visible, it would have shadows, but no color (or material color only).
The Visualizations really need a good custom Shader.
That would open up the floodgates for all kind of new customization options.

I've experimented with a simple custom one that I created with a Shader Editor named Amplify. At one point I had a version where I loaded this one ...
View attachment 155574
... from an AssetBundle. It worked, but to be honest I absolutely hate AssetBundles. I removed that code part again. I get why they are useful for game-development, but I just find the workflow to create them super annoying, they add so many additional potential points of failure and they are more or less black box once packed. If I can avoid them I will avoid them. Not sure whether that will be possible with Shaders to load them "directly" into VaM.


Hmm that reminds me, I think the VaM integrated RythymTool does something similar when a audio file is loaded.
Could try to "steal" the required data from that one too. I think that should be 100% the same data every time it loads - as long as the audio file is the same.

Meanwhile more experiments ... back to cave now.

OMG Cant wait for the AVGRID for VAM!! YEEEES

 

[Sally Whitemane]

Sally Whitemane updated SallySoundFX with a new update entry:

Version 2.01 (Sally.SallySoundFX.3.var)

Demo Scene in separate Upload to keep dependency small.
Initial Version 2 VaM Hub Upload so users can build scenes from it and have the required dependency on VaM Hub.
Fixed some bugs. Don't worry, I will add new ones .

Changes Version 2.01

• Replaced non square textures with ones that contain an Alpha Channel for transparency testing. Note that these Alpha Channel are more like a transparency masks, either 1 or 0. Custom Textures with values in between can behave different...

Read the rest of this update entry...

 

[Sally Whitemane]

Looking for ways to avoid AssetBundles and load a custom shader directly.
Documenting this for myself because I'm stupid and forget things.

Found this obsolete function the seems (?!) to be for loading a shader from it's sourcecode directly.

 [Obsolete("Creating materials from shader source string will be removed in the future. Use Shader assets instead.")]
 public static Material Create(string scriptContents)
 {
     return new Material(scriptContents);
 }

Do not really care about it being obsolete if it works, since VaM 1.x is a dead end anyway.

So I 'hardcoded' the shader into this string and tried load that ...

Spoiler: Shader as string

public static readonly string SSFXShader = @"Shader ""SSFX""
        {
          Properties
          {
            _MainTex(""MainTex"", 2D) = ""white"" {}
            _Smoothness(""Smoothness"", Float) = 1
            _EmissionMultiply(""EmissionMultiply"", Range( 0 , 4)) = 1
            _Metallic(""Metallic"", Float) = 1
            [HideInInspector] _texcoord( """", 2D ) = ""white"" {}
            [HideInInspector] __dirty( """", Int ) = 1
          }

            SubShader
          {
            Tags{ ""RenderType"" = ""Opaque""  ""Queue"" = ""Geometry+0"" ""IsEmissive"" = ""true""  }
        Cull Back
            CGPROGRAM
                #pragma target 3.0
        #pragma surface surf Standard keepalpha noshadow exclude_path:deferred
                struct Input
        {
            float2 uv_texcoord;
            float4 vertexColor : COLOR;
            };

        uniform sampler2D _MainTex;
        uniform float4 _MainTex_ST;
        uniform float _EmissionMultiply;
        uniform float _Metallic;
        uniform float _Smoothness;

        void surf(Input i, inout SurfaceOutputStandard o )
        {
            float2 uv_MainTex = i.uv_texcoord * _MainTex_ST.xy + _MainTex_ST.zw;
            float4 tex2DNode3 = tex2D(_MainTex, uv_MainTex);
            o.Albedo = tex2DNode3.rgb;
            o.Emission = (tex2DNode3 * i.vertexColor * _EmissionMultiply).rgb;
            o.Metallic = _Metallic;
            o.Smoothness = _Smoothness;
            o.Alpha = 1;
        }

        ENDCG
      }
      Fallback ""Sprites/Default""

        CustomEditor ""ASEMaterialInspector""
    }";

Try to load it:

material = Material.Create(AVSlave.SSFXShader);

... which just results in VaM loading a pink material with error-shader: Hidden/InternalErrorShader (UnityEngine.Material)
Must use AssetBundle I guess. I know the shader does work when loaded from an AssetBundle.
Not really sure why it's failing. Maybe it really is obsolete and not working anymore. Maybe I made a mistake in the shader string with the double-quotes or something.

 

[Sally Whitemane]

*rambling about Shaders*

In the 2.01 Release the "dancefloor" aka AVTiles used the "UI/Default" Shader which was two-sided.
So you could fly under it and could still see the "back".
If you do that when switched to "Custom/Subsurface/EmissveGlow" the back-faces are 'culled' and nothing is visible.
I'm in the process of adding custom Shaders and I need to decide how the default SSFX-Shader should behave.
Mainly the Backface-Culling (whether to render a mesh two-sided or not) makes me scratch my head.

Use this as default?

Light Model:
[ ] lit (requires light source, can have shadows)
[x] Unlit (always visible unless colored 100% black)

Culling Mode: Visible from two sides? (do not confuse with occlusion culling)
[x] one-sided (Backface culling, better performance, only front faces visible)
[ ] two-sided (No culling, slower performance, easier to use because it's always visible no matter from where you look)

Blend Mode:
[x] Opaque (fastest)
[ ] Transparent (There are so many more options here on how to blend and render transparency and/or masks)

Another reason for using one-sided by default is that it can break a scene when switching from two- to one-sided later for performances reasons later. Meshes could be facing the "wrong" way. So by using one-sided you'd have to build them correctly from the start.

On the other hand, I have no clue how high the performance impact really is. I only know that Backface-culling is just in general considered faster and it's typically used in game-engines for better performance.

 

[Sally Whitemane]

@Shadow Venom

As promised, here is an experimental SallySoundFX version with more triggers in a Slave-Plugin:
Edit: Small Update (older link removed) - https://www.patreon.com/posts/74931032

Intended for testing or private use only, because I will probably still make changes to that Slave-Plugin that can break a scene when updated later.
So publishing a scene with that version is a bad idea.

 

[Sally Whitemane]

Sally Whitemane updated SallySoundFX with a new update entry:

Version 2.02 (Sally.SallySoundFX.4.var)

• Added AVMoreTriggers-Slave Plugin that provides 8 additional Trigger, loaded via AVMoreTriggers.cslist

It is the version previously tested here. More details here (older outdated, do not use).

Read the rest of this update entry...

 

[ossa]

Let me ask you a question.
I use RenderToMovie included in ZHFX's MMDPlayerPack for video output.

When a scene using SallySoundFX is output as a video with RenderToMovie, it will not synchronize with the sound source, and the effect will not appear in the output video.

Is there a way to output a scene using RenderToMovie as a video with RenderToMovie?

 

[Sally Whitemane]

If enabled the SSFX Plugin should sync with anything playing on any Unity AudioSource.
The scene Sonic Stage 2 disables the Plugin when playing the demo. (From what I understand I'd need paid MMD plugin for that scene?)
Can't help with the paid RenderToMovie Plugin because I do not know how it plays audio and I do not own it.

The data source is the function documented here.

AudioListener.GetSpectrumData(samplesUnity, 0, FFTWindow.Blackman);

The 0 means it's using the left channel.

The VAM embedded webbrowser does not work. It plays audio in a different way.
It's noticeable on the taskbar volume control. It has a separate volume slider for that browser.

Edit:
Btw. the only reason why I enforce the SSFX Plugin must be added to an AudioSource has nothing to do with that audio playing on it.
It's just in case some user decides to use the microphone function later where that AudioSource would be required.
So the whole 'scene logic' does not have to be redone.

 

[ossa]

Thank you for your reply.
I'll study the Sonic Stage a lot.

 

[CheersMate]

It appears the plugin triggers off the actual audio volume output to the speakers/headphones/etc. Is there a way to trigger off a sound source that isn't the loudest in the scene? For example a distant source that the may not even be audible to the user.

 

[Sally Whitemane]

@CheersMate
The Unity Engine only provides the GetSpectrumData() function for the "master channel". That's all Audio Sources mixed together. I would need that function for a specific AudioSource to isolate the signal from all other sources.

Triggering from a relatively quiet source works if it's the only source. But it's kinda sketchy. Maybe you'd have to disable Auto-amplification and crank that up very high.
This reminds me of the situation in real life - when you put a headphone plug into a "line out" instead of "headphone out" - the signal on a "line out" has no amplifier and comes out very quiet - the amplifier is needed - it's works, but is not ideal.

If you need a simple separated signal from various AudioSources (assuming you have for example loud music playing on another source) it may be better to use the standard VAM "Volume Action". The volume signal coming from there is tied to what's currently playing on that AudioSource. I guess that uses the in VAM implemented RhythmTool, but I'm not sure how that works.

 

[CheersMate]

@CheersMate
The Unity Engine only provides the GetSpectrumData() function for the "master channel". That's all Audio Sources mixed together. I would need that function for a specific AudioSource to isolate the signal from all other sources.

Triggering from a relatively quiet source works if it's the only source. But it's kinda sketchy. Maybe you'd have to disable Auto-amplification and crank that up very high.
This reminds me of the situation in real life - when you put a headphone plug into a "line out" instead of "headphone out" - the signal on a "line out" has no amplifier and comes out very quiet - the amplifier is needed - it's works, but is not ideal.

If you need a simple separated signal from various AudioSources (assuming you have for example loud music playing on another source) it may be better to use the standard VAM "Volume Action". The volume signal coming from there is tied to what's currently playing on that AudioSource. I guess that uses the in VAM implemented RhythmTool, but I'm not sure how that works.

Too bad. Your plugin is 1000% better than VaM's volume action. Thanks for the quick response.

 

[VamTimbo]

I need a way for the triggers to have a hierchy... because with my project, sometimes some frequencies trigger at the same time as others - I need a way for some frequencies to take priority over others. Can you make a way for that to be possible?

 

[ossa]

The triggers of the SSFX plugin are functioning, but the on/off of individual triggers created with SSFX cannot be done. Is this a specification? If so, is there any plan for improvement in the future?

 

[Sally Whitemane]

@ossa
Are you talking about this trigger for example not disabling the cats scaling when unchecked?

I was aware this did not work. Thought nobody would need it anyway and it would introduce another potential point of failure leaving users wondering why things do not *react* to the audio.
The triggers use MacGrubers code and I am not very familiar with it - kind of want to avoid touching that.

How about this:
Fixed key shortcuts like Control & F1 to toggle Trigger 1 on/off. (Would have to use keys that are not used by other popular Plugins to avoid conflicts)
When pressed it shows a "SSFX Trigger 1 on/off" message in VAMs blue message log.

• Advantage: Trigger could be toggled from anywhere without opening the Plugins UI
• Disadvantage: SSFX UI has no visual feedback on whether the trigger is on/off

I'd have to look into how deal with disabled triggers. Currently all slaves Plugin expect to receive values for the SSFX triggers. I need to send something (or would have to break compatibility). So I'd probably have to define a 0.00 value as trigger disabled. Meaning that internally they would still be updated. That's important - in the example picture it would *freeze* the cats scale value to 0.00.
What I cannot do is add this trigger shortcuts for any VR keys - I simply do not own VR hardware and cannot test it.

 

[Jimi T]

I have a noob question! I have been learning my way in VAM for two years and love it. I have created some models, scene etc. I have some scenes that have music for the animation dances. Would love to share but I really do not want to get into any trouble with copy right laws. So I been trying to figure away to get maybe the Web panel, or web panel emissive ( YouTube videos). to use the rhythm audio source. I searched every where but I found this discussion seems like it is impossible at this time. Hopefully Mesh VR will fix that!! I read the EULA. but really do not understand or it is not clear to me. My question is, Is it illegal to post scene with a music (mp3) file of a song? Reason I ask is I have seen some other dance scenes posted with songs added but not clear on why they can up loud that and not get into trouble??. Any advice would be much appreciated!!! Thanks!!

 

[Sally Whitemane]

@Jimi T
Yes including copyrighted music with a scene is a legal problem.
Pretty sure many scenes here are not legal with their included music. Guess everybody is just 'looking away'.

All the web-related Atoms do play audio with the VAM embedded browser.
Note how the Windows Volume Mixer it has an extra Volume control for that browser. It's played in a different way.

Some scene creators include instructions to download a song from YT - with renaming & moving it to the correct VAM folder.
*meh* - that's annoying and just put's the legal problem into the users hands and everybody has to do extra work to make a scene work.

I'd try to find CC0 or CC BY licensed music and include the file. It's more reliable.
Personally I think streaming is bad:
-Users may not give permission to access web for security reasons
-VAM does not 'react' to audio from embedded browser
-Link could die after some time -> breaks scene
-Potential bad latency/connection if hosted on a server far away
-Embedded browser is picky with the audio format and isn't very reliable (browser crashes and streams randomly stop)

The SSFX demo scene uses classic music where the copyright has expired. It's public domain.
Vinheteiro on YT inspired me to use this.
Obviously it's 'difficult' to find a RAP song from 1855.
Older music is just something to keep in mind when looking for usable music.

 

[Jimi T]

@Jimi T
Yes including copyrighted music with a scene is a legal problem.
Pretty sure many scenes here are not legal with their included music. Guess everybody is just 'looking away'.

All the web-related Atoms do play audio with the VAM embedded browser.
Note how the Windows Volume Mixer it has an extra Volume control for that browser. It's played in a different way.

Some scene creators include instructions to download a song from YT - with renaming & moving it to the correct VAM folder.
*meh* - that's annoying and just put's the legal problem into the users hands and everybody has to do extra work to make a scene work.

I'd try to find CC0 or CC BY licensed music and include the file. It's more reliable.
Personally I think streaming is bad:
-Users may not give permission to access web for security reasons
-VAM does not 'react' to audio from embedded browser
-Link could die after some time -> breaks scene
-Potential bad latency/connection if hosted on a server far away
-Embedded browser is picky with the audio format and isn't very reliable (browser crashes and streams randomly stop)

The SSFX demo scene uses classic music where the copyright has expired. It's public domain.
Vinheteiro on YT inspired me to use this.
Obviously it's 'difficult' to find a RAP song from 1855.
Older music is just something to keep in mind when looking for usable music.

Thank you for clearing that up. I can just put in the embedded VAM music and the scene will still work. Also thank you for your help and the advice. Ill have to go check the SSFX demo out.

 

[qdaro]

Am I the only one with this error?

!> Compile of Sally.SallySoundFX.4:/Custom/Scripts/Sally/SallySoundFX/SallySoundFX.cslist failed. Exception: System.Exception: Path MacGruber.LogicBricks.12:\Custom\Scripts\MacGruber\LogicBricks\Internal\MacGruber_Utils.cs not found
  at MVR.FileManagement.FileManager.ReadAllText (System.String path, Boolean restrictPath) [0x00000] in <filename unknown>:0
  at MVRPluginManager.SyncPluginUrlInternal (.MVRPlugin mvrp, Boolean isFromConfirmDenyResponse) [0x00000] in <filename unknown>:0

I have MacGruber.LogicBricks.14 installed. I guess you hardcoded a dependency on version 12?

 

[Sally Whitemane]

I have MacGruber.LogicBricks.14 installed. I guess you hardcoded a dependency on version 12?

Yes, some .cslist -files are hardcoded to reference LogicBricks.12.
I would like them to use the latest version automatically.
However it does not seem to be possible from a .cslist-file. Using the .latest syntax does not work here.
Or I don't know how to do it?
I'd guess this is intentional by VAM. In theory nobody knows if future LogicBricks-changes could break my Plugin. So it's safer to reference that specific version.

If for whatever reason you need this - I can private message you a version that uses LogicBricks.14.
Or you can edit SallySoundFX.cslist, AVTiles.cslist and AVMoreTriggers.cslist yourself from LogicBricks.12 to LogicBricks.14.
I've tested this and know LogicBricks.14 is compatible.
However I'd strongly advise to NOT use this if you plan to share scenes - this could end in a scenario where the scene dependency point to the wrong LogicBricks - breaking it for other users.
I'm very very careful with updates on this in general. Some creators made insane complex scenes with SSFX - that I know - will break from just upping the .var version number. Nothing unfixable, but very annoying and time consuming - simply due to their scenes complexity. So I'd rather not make an official update only to reference a newer LogicBricks-version.

If I ever update this I will remove this dependency completely. Either by moving the code (small part of LogicBricks) into my package or by replacing it. No dependency or version shenanigans anymore.