[Feature]: SageMaker Endpoint Streaming via invoke_endpoint_with_response_stream

[nicolasj92]

The Feature

True (not faked) streaming of SageMaker Endpoint model responses via the invoke_endpoint_with_response_stream API in boto3.

Motivation, pitch

I would be willing to make a proposal / PR for the implementation of this feature but I wanted to check if you are already working on this or if there are any strong blockers. I found this code snippet that seems to explore this functionality within the repository:

litellm/cookbook/misc/sagmaker_streaming.py

Line 48 in 0288ed3

response = client.invoke_endpoint_with_response_stream(

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

[nicolasj92]

@ishaan-jaff let me know if it makes sense for me to contribute here or if there are other plans.