e2e tests #34
e2e tests #34
Conversation
|
Dumb question: Sometimes ingest delay is NOT two hours, right. Sometimes it's a few minutes (does this vary by account?) could we implement something akin to retry logic that backs off each time? 5 minutes > 20 minutes > 2 hours > 8 hours Maybe that would be fighting the tools too much, but I think it could work if we could cancel a scheduled job upon receiving our desired outcome. |
|
@daniloc I'm down to try this. I'm (probably) gonna ship this for now with a 24 hour delay, and do a separate job with this back off later. I think if it's not overly complicated, this is the right way to approach it, but weary of how much we'd benefit from knowing an example is broken ~20 hours earlier. For now, I don't think it's worth it if doing back off retry proves difficult. Can't know until I try! |
|
fair enough! |
|
Aight this :should: work, did a bunch of tests. But we have alerts, so we'll find out tonight |
Because ingestion takes time and the query apis are cached, e2e runs every night, but checks if yesterdays' events were properly captured.
We'll set up alerting. knowing 1 day later is better than never. We could also just run these ~2 hours after integration nightly on the same day, but is flakier.
how these work
integration generates dated ID -> identifies -> captures events -> 24 hours later -> e2e test queries yesterday's captures to make sure prev. events made it through ingestion.