There is a button in VaM to clean the cache. If the assets are needed again it will just take slightly longer to load the scene the next time, after that it will be cached again. Especially if you are in the habit of downloading stuff, looking at it and deleting it or never looking at it again...you should delete the cache every once in a while. However, as long as you still got disk space, its not a problem either.