The problem is most likely VRAM, not RAM. So, memory of your graphics card. In the UI the plugin shows you how much VRAM it eats, that can be up to around 6GB, depending on the settings, if I remember correctly. That VRAM usage is on top on whatever VAM needs for your scene and what other applications need. SuperShot increases image quality, but you pay for it in insane VRAM usage.
MSAA is just a plain multiplier in VRAM usage and it defaults to 8x for example. Reducing the resolution multiplier helps, too, of course.
If you don't want to reduce quality, you could try to set the resolution multiplier to 1x, but instead increase the image resolution by the same factor instead. You get a high-res image that you can downscale yourself with external tools, rather than having SuperShot do the downscaling for you. Taking a screenshot will take much longer, but it reduces VRAM usage by 1/3.