I had asked about this back in '18 when developing a queue system and ended up going with a solution like you are describing where an endpoint check is made for players in queue and if that is null then I done the deferral on that event and let it close.
C#
if (source?.EndPoint == null)
{
deferrals.done($"{Configuration.messages["Canceled"]}");
I believe I was having problems with the same player being able to queue up again after they canceled if the original player connecting event was still in progress in the queue when they connected again.