As you are experiencing first hand, the size of your AddonPackages dir (inclusive of ALL of its sub dirs) affect the loading time and stability of VAM. Technically, it is not the number of VARs that impacts performance. MeshedVR stated that the issue has to do with the total number of morphs that VAM can see. It doesn't know in advance which ones are needed so VAM loads EVERY morph including duplicates. The number of VARs in the AddonPackages directory tree roughly correlates with the number of morphs you're exposing to VAM. Which is why slowness is associated with large AddonPackages dirs. This is also why sorting VARs in AddonPackages into sub directories does NOT help your situation at all. The only way to fix the performance impact is to "offload" the VARs to another directory that is NOT recognized by VAM - and as you requested, ""load them" when needed".
To my knowledge, there are two tools that can help with large VAR collections through offloading: the first and earliest is Var Browser from sfishere. The second is the Var Management feature which was recently added by JayJayWon to his most excellent Browser Assist. Var Browser is no longer being actively developed (which is a shame because it's the one I use). Browser Assist is still being actively maintained.
Do some research/digging as there is now a lot of written material on this topic and associated tools. (When I first went down this rabbit hole - there was little to no documentation on Var Browser and JayJayWon had not yet released the Var Management feature).
On a personal note (i.e. datapoint of one): I started experiencing painfully slow loading at around 500GB. Then I experienced instability issues (heap errors) at around 800GB. Then my VAM installation would just flat out not load at around 1TB. Thanks to Var Browser (and some other tools*), I am currently running smoothly even with a VAR collection that sits at 3+TB. [The other tools that help me reduce duplicate morphs are: Pesadilla's DeDuplicator and Bill Prime's Var Manager].
HTH
