Thanks man
Almost everything is cpu driven, except maybe the morphing (mainly expressions), but this is only done on the plugin holder.
The feature that scales the most with person count is the Gaze. Each person has many targets and BL has to check their visibility every time a new target is to be choosen. By default only in respect to the plugin holders view, but the amount of computations needed rises linearly with the amount of Gaze-clients enabled. You can enhance the performance by disabling Target Occlusion in the Performance tab. The cost of Sub Target Occlusion does scale with the client number, but not with the sheer amount of other persons in the scene. Target occlusion scales with both in a quadratic fashion: 6 clients means 36 times the work. This factor is further multiplied by the amount of mirrors you have in the scene (glass & reflectives), because for each mirror there has to be checked if the image of any real target is visible to client X (virtual target). You can disable mirror functionality as well.
If you only have one client, I don't think that this could cut your fps in half. If anything, it should cause some hicups cause the computations are only performed once in a while (Target Time). If you have multiple clients it possibly could, because the Subtarget Occlusion has to be checked every physics update.
6 persons really is a lot and I wonder how you get those fps, even without any plugins. Are you running in 1080p?