IPSEC is only the security blanket used for the VPN that you are travelling through. VOIP packets (yes UDP on RTP) are never resent. It would not make sense to resend voice packets as the delay could be unreal.
It occurs to me that you may be experiencing a nasty amount of jitter

Dave M