Fixed it by just keeping the map always on