Missed that one (as usual, stoopid notif system)
That's how the baking process is made, and how Unity engine optimizes how a scene runs. So no I will not "fix this" : )
That enviro is so massive, and there are so little situations where you'd need to move seamlessly, that swapping enviro on and...