Thanks for being so attentive
I looked into this more and discovered 2.5 interesting pieces of information:
1) Changing User Preferences -> Physics Update Cap from 3 to 1 tripled my FPS

(screenshot attached)
2) Inserting microsecond measurements into FrameUpdate revealed that function IsDetachedSimClothing is quite slow on my computer taking ~30ms for just two calls (screenshot attached)
2.5) It appears that FrameUpdate is part of category 'Physics' not 'Script' in Performance Monitor numbers
Maybe there is an issue with my library size since you are not seeing performance penalties anywhere close to mine. I use the BrowserAssist session plugin to "offload" (and manage) 300GB of AddonPackages which should be invisible to VaM. Maybe I am mistaken in how that works and there is still some slowness in processing clothing.