Hey, thanks so much for your advice and tips! I'm currently learning about harnesses and have realized that, without even knowing it, I've already put together a very simple one. The AI can already trigger certain actions, but I first have to define them in the plugin and set a trigger word. When the AI types that word in *...*, the trigger is automatically activated the moment the last sentence before the trigger word is spoken. But your approach goes much further—if I understand correctly, the AI can decide for itself what to trigger without any actions being defined in advance. For example, it could automatically. trigger a timeline animation if it feels like the timing is right!? Am I understanding this correctly? Can you tell me what in-game data you send to the AI so it knows what’s happening in the game right now? Or do you do this using images?
I think so, yes. In the System Prompt, describe how you want the model to behave in the scene.
Unfortunately, it is difficult to evaluate what may be causing this without knowing what your System Prompt and Role Prompt are. It may also be caused either by the specific LLM model you are using, or by how the role or System Prompt is written.
I would recommend writing clear behavior rules in the System Prompt and checking how the model’s behavior changes. If possible, please let me know whether this helped and which model you used.
Also, to better understand your problem, you can send me your Role Prompt, System Prompt, and the model you are using in a private message, along with a more detailed description of the issue, so I can try to reproduce it.
And to answer your question: yes, my server provides the AI with information about what it can do in the scene on every request. Each time, new screenshots, the full dialogue, and the full available toolset are sent to LM Studio. This includes triggers and my scripts with controls that the AI can use.
The AI itself does not have memory. On every request, you need to show it what it can do and return information about what it has already done. Otherwise, it will forget. An LLM does not have its own memory. It is just a text generator, good or bad. That is why the full context, screenshots, tools, and in-game scripts it can use have to be sent again every time.