Didn't know how to properly use cloth sim, went back and improved physics. Now primarily only the loin cloth area will react to physics.
Thanks to Atani for the help. Be sure to upvote his guide if you like this.
https://hub.virtamate.com/resources/a-guide-for-sim-textures.28569/
