Yes, as you’ll all of the time be considered as inside of the island, you’ll never be able to load any other interior minimap.
A workaround that I found can be found here : [Releases] [Free] Cayo Perico Improvements Freeroam 4.2.2 - #506 by AvaN0x
Detailed version here : [How-To] Load CayoPerico minimap all of the time