What do you think about using a monotonic clock like the one in System.Clock instead of getCurrentTime? System clocks occasionally move backwards in order to correct clock skew, which makes a mess of latency statistics.
(To be clear, I haven't yet seen this issue with this particular library, but it's a known challenge.)
Happy to follow up with a PR if you like the idea.