For VR, each scene has to be rendered twice, instead of just once. And often at a higher resolution than on desktop.
So drastically lower framerates in VR is completely normal.
You might try turning down quality to increase performance. Using the
GiveMeFPS plugin to tweak your performance settings, or go into user preferences>performance and changing them by hand.
Or buy a faster computer.