It depends on how heavy are your characters and scene in terms of complexity, texture, lights, etc.
So it's a mix of multiple factors that need to be investigated.
But it shouldn't be going down to 1 for no reason.
It would start with a new install of VAM to exclude any issue with the base game.