The gallery is not slowing down page load, but preloading many videos might.
If the video is below the fold
Go to video gallery settings > Miscellaneous > and set Initialize on to scroll and now the gallery will initialize when the gallery is visible
IN the shortcode make sure init_on is set to scroll
If you want to disable preloading
Go to video gallery settings > Play options > and set Cue First media to Disabled
In the player configuration you are using set preload method to none