Commit retries block main thread: Drop commits instead? #415
Description
our default commit retry policy is retrying the commit up to 3 times with 1 second delay. during those 1-3 seconds we are blocking the main thread, potentially creating backlogs (not just in terms of consumer lag, which is unavoidable since we can't commit, but also application-level ingestion latency). it would be more appropriate to either drop the commit (if we know another commit will come in anyway), or defer committing to a background thread.
related: #414