-
-
Notifications
You must be signed in to change notification settings - Fork 5k
Closed as not planned
Closed as not planned
Copy link
Labels
Description
What happened?
Description
We’ve observed unexpected behavior when setting weight: 0 for a deployment in one model group.
Expected behavior
Setting weight: 0 should only remove that deployment from load balancing within that model group.
It should not impact other groups or direct calls to that same provider model elsewhere.
Actual behavior
- When a deployment is configured with
weight: 0(e.g. Bedrock Sonnet 4), it not only triggered a division by zero crash (now patched by #13624) but also appeared to affect calls in other model groups that referenced the same provider model. - This makes direct calls to
bedrock/converse/...unreliable if anyweight: 0entries exist in other groups. - In practice, config for one group unexpectedly influences behavior of other groups.
Example Config
- model_name: -multi-provider-claude-sonnet-4-20250514
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
weight: 85
- model_name:-multi-provider-claude-sonnet-4-20250514
litellm_params:
model: bedrock/converse/us.anthropic.claude-sonnet-4-20250514-v1:0
weight: 0 # problematic entry
- model_name: -multi-provider-claude-sonnet-4-20250514
litellm_params:
model: vertex_ai/claude-sonnet-4@20250514
vertex_project: os.environ/VERTEX_PROJECT
vertex_location: us-east5
vertex_credentials: os.environ/VERTEX_SERVICE_ACCOUNT
weight: 15
### Relevant log output
```shell
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
v1.75
Twitter / LinkedIn details
No response