You can, but that won't be reflected inside VaM unless you use the "filtered view" or whatever is called on the left sidebar.
You can make favourites of packages and filter for those, or you can use a plugin like pluginassist by JayJayWon to organise plugins.
My way of doing things is a little different. I use a plugin called UIAssist, also by JayJayWon, where I set a few buttons that add specific plugins that I frequently use.
For less often used plugins, I have my plugins in a plugins folder in Addonpackages, and whenever I need one that is not in the UIAssist HUD I use the filtered view, go to that plugins folder, and search for the one I want.