Streaming Result for Query/QueryRange HTTP API

[bwplotka]

In Thanos we are moving forward with query sharding, parallelization and pushdown ideas 💪🏽 Recently, we pushed a change that is capable of safely pushdown certain aggregations to Prometheus (thanos-io/thanos#4917).

What we do in sidecar now, for certain "safe" functions like min, max, group, min_over_time, and max_over_time instead of fetching all series through remote read, we change to HTTP Query API. We build PromQL from select hints we get from root Querier and perform query. Then we transform the Query result to Thanos GRPC StoreAPI.

Now, we have efficiency problem with Query API, because it's not streamed (something I was moaning about for long time 🙈). This means we unnecessary wait for full response payload, instead of streaming each resulted series by series like we do with remote read. AFAIK for most cases PromQL performs series by series calculations, so first series is calculated before next one, so that fits server part of this API too.

Now, to solve our use case, there are two options we could do to improve Prometheus for these use cases and actually win in other fields too.

A) Introduce streaming (per series) for query and query range Prometheus HTTP APIs. We could do this exactly as we did with remote read. Don't stream by default and opt-in using header negotiation for streamed JSON using chunked JSON logic we use with profobuf in new remote read. It's not straightforward as our current JSON result with this data format was never attempted to be streamed.
B) Allow remote-read to perform PromQL using hints. It's not hard to implement, but quite weird as there is strong overlap now between Query + Query range and remote write that can do PromQL. Plus it's easy right now to make mistake and pass wrong hints (aggregations/function that are NOT safe to be pushed down).

I would vote for A. I believe streaming response on query would help with many other use cases (e.g Prometheus -> Grafana interaction and Querier -> Query frontend in Thanos).

If we are ok to pursue A I could try to work with community on proposing exact format of JSON streamed API.

cc @tomwilkie @roidelapluie @juliusv @fpetkovski @brian-brazil @moadz @RichiH

[brian-brazil]

AFAIK for most cases PromQL performs series by series calculations,

That's not the case. This is true for range vector functions internally, however basically everything else is step by step including the functions you list.
So there's not much to be streamed, and you'd have to rewrite a large chunk of PromQL to make it happen - plus ensure that failure semantics are dealt with correctly, and somehow ensure that resource usage isn't adversely affected given you're now doing more work and keeping more intermediate results in memory.

[juliusv]

Yeah, making the PromQL computation itself streamable on a per-series basis sounds hard, but the final JSON result could of course be streamed quite easily. Maybe that's still worth it for large payloads in some use cases?

[brian-brazil]

Once we have all the result samples, streaming from there makes sense. I think I ended up not doing it when making the json iterator changes as it ended up worse performance wise, but that could have changed.

[bwplotka]

Thanks for the input. Looks like we could descope it to just an API and check if it helps with anything (it helps for Thanos sidecar case for sure).

On separate note then...

and you'd have to rewrite a large chunk of PromQL to make it happen - plus ensure that failure semantics are dealt with correctly, and somehow ensure that resource usage isn't adversely affected given you're now doing more work and keeping more intermediate results in memory.

@brian-brazil you made me curious now! So do you think there is room to make PromQL computing as much series by series internally, even if it's not currently? 🤔

[brian-brazil]

It's already doing as much as it can as clearly makes sense. Beyond that would require a big redesign, and it's unclear that it'd result in any performance gains. Keeping maximum memory down is tricky.

…

On Fri 17 Dec 2021, 21:59 Bartlomiej Plotka, ***@***.***> wrote: Thanks for the input. Looks like we could descope it to just an API and check if it helps with anything (it helps for Thanos sidecar case for sure). On separate note then... and you'd have to rewrite a large chunk of PromQL to make it happen - plus ensure that failure semantics are dealt with correctly, and somehow ensure that resource usage isn't adversely affected given you're now doing more work and keeping more intermediate results in memory. @brian-brazil <https://github.com/brian-brazil> you made me curious now! So do you think there is room to make PromQL computing as much series by series internally, even if it's not currently? 🤔 — Reply to this email directly, view it on GitHub <#10040 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWJG5TZIVQ5ROL6K63QZYTUROXDZANCNFSM5KIRQ6YQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[beorn7]

Hello from the bug scrub!

In the meantime, a lot has happened in the area of "yet another PromQL engine". Thanos has one, Mimir is working on one, but they are all meant for their specific use case and not necessarily to just plug that new engine into a vanilla Prometheus.

Having said that, the topic of creating a new PromQL engine for vanilla Prometheus is still coming up quite often, and with a new engine, a streaming as proposed here might be feasible. We'll keep this issue open as a reminder.

[ringerc]

While the ability to stream-out responses does not solve peak memory use issues, it would be a really helpful step forwards. Especially where Prometheus is one link in a chain of components, such that it might be expected to send large interim responses that would not be typical for returning directly as user-facing results.

There are AFAICS two related but independent things required:

1. A client query interface using a protocol that can stream samples and series (HTTP/2, HTTP/3 over QUIC, or gRPC), preferably with a more efficient response serialisation than json-text representation [this ticket]
2. A Prometheus engine Go level interface that can yield a stream of samples and series, rather than accumulating a vector in-memory before returning it [raised as Add iterator-returning variant of promql.Query engine interface #17276]

The former (1) is useful without the latter. I've tested Thanos's gRPC PrometheusQuery as a client interface for Thanos, and there I find that even though Thanos still buffers the whole PrometheusResult in memory I still see a significant decrease in peak memory use. The improvement comes from avoiding building a large json serialisation of the entire result in memory concurrently with keeping the result vector in memory; the response can be streamed out over gRPC instead.

It might make sense for Prometheus to adopt the Thanos gRPC interface PrometheusRequest or an adaption of it to expose a streaming query interface. Thoughts @beorn7 , @MichaHoffmann ?

Then (2) can be addressed separately to get rid of materialisation of the final result vector in-memory by allowing the engine to yield an iterator. I'll raise a separate issue for that. (2) would benefit users of /api/v1/range as well as the new streaming API by avoiding storing the whole matrix in memory as well as its json-serialisation.

Once those are done, Prometheus will still be loading whole series into memory in the executor before it works on them. But that can be tackled progressively, as-needed, once the basis for streaming out results is in place, starting with the simplest cases, then moving on to streaming of more complex funcs, joins etc.

In conjunction with #10040, this could also allow a Thanos Query in Distributed mode to more efficiently push-down some PromQL execution into Prometheus instead of using remote-read for series fetches. That could help reduce the impact of the external labels sorting problem and consequent Thanos sidecar buffering and re-sorting by allowing Thanos to use Query rather than Series requests most of the time. Thanos Query could also entirely bypass Sidecar in this case, reducing the layers of multiple-handling for data.