the technical reason why this happens is because when you submit your deadline to the pool. the pool then has to go through the entire routine to verify that the nonce is valid,
this means it has to shabal256 your accountid and nonce 4096 times then run the calculation to find the deadline
you could refer to the below to see the full routine
whilst this routine is run in different threads it still takes a while to execute this command for a single request. the time it takes ranges from 0.25 seconds to a full second.
so the pool is effectively plotting as its receiving requests then submitting them to the network.
now add that with the speed your uploading/current network traffic/time it takes over distance. and the delay could take longer than your miner will wait for the response.
i hope this answers some questions