Documented request limits don't seem to match up with observed values

[davegreen]

When making a request to list groups (or users), it seems like the request limits are lower than the documented values (5 requests per minute for groups, more for other use cases).

When making a request to return all groups, I can do one single page request before running into HTTP 429 (Too Many Requests). It also seems like the retry value we are being given when trying again is approximately 40 minutes in the future, rather than a more reasonable sub-minute value. I'm not sure I understand why this would be the case?

Our organisation contains 17 groups and 153 users, so all well contained within one page of returned information, so it doesn't seem like we would have any cause to run into these limits?

Groups

Get-AdobeGroup -Verbose -Debug
DEBUG: {
  "ContentType": "application/json",
  "Method": "Get",
  "Headers": {
    "X-Api-Key": "APIKEY",
    "Authorization": "Bearer BEARER"
  },
  "ErrorAction": "Stop",
  "UseBasicParsing": true,
  "Uri": "https://usermanagement.adobe.io/v2/usermanagement/groups/ORGANIZATION@AdobeOrg/0"
}
VERBOSE: HTTP/1.1 GET with 0-byte payload
VERBOSE: received 3250-byte response of content type application/json

Users

Get-AdobeUser -Verbose -Debug
DEBUG: {
  "ContentType": "application/json",
  "Method": "Get",
  "Headers": {
    "X-Api-Key": "APIKEY",
    "Authorization": "Bearer BEARER"
  },
  "ErrorAction": "Stop",
  "UseBasicParsing": true,
  "Uri": "https://usermanagement.adobe.io/v2/usermanagement/users/ORGANIZATION@AdobeOrg/0"
}
VERBOSE: HTTP/1.1 GET with 0-byte payload
VERBOSE: received -byte response of content type 

[molldk]

Same issues here, only very few requests are allowed and very far off from the limits in documentation.

[davegreen]

It would be good to know if this is expected behaviour on the server end, as this behaviour makes the API impossible to use for any normal use case, even to enumerate users and groups at the same time.

I can provide more to reproduce the problem if there are any questions about the implementation of the client code?

[lesavage]

There are many reasons why a request could be throttled so it is difficult to document this. We always encourage clients to sync at unusual times of the day so they are not attempting to share resources with other clients which would result in the global limit being reached.

[molldk]

@davegreen I had a ticket open with Adobe support who told me there is a global rate-limit override. If you are calling endpoint to fetch page 0 of all users more than once an hour, the global rate-limit kick in, blocking ALL endpoints until time is expired.

So the way I get around this is by fetching users by group once every 2 hour maximum. If you stay clear of the global override, the rate-limits should be as defined in documentation.

This is a bit confusing from the API documentation, but was what I ran into, so might be the same for you.

I was told Adobe would update their documentation to make this more clear, however I believe this is the section that mentions it:
"Since October 2021, Adobe has added controls to check the running frequency of each UMAPI client instance to prevent running syncs more frequently than the recommended timing of no more than once every 2 hours. Running the calls more frequently can start a new session prior to the completion of the previous session, resulting in syncing delays. When the access limit is reached, further calls fail with the error message ‘429 Too Many Requests’ and a Retry-After header containing the delay required before the next call can be made. Please refer to the Throttling section of each API to determine its limitations."
Src: https://adobe-apiplatform.github.io/umapi-documentation/en/

[davegreen]

@molldk This is really interesting, thanks! This seems like it makes it much more difficult to use with a small org, as we only have a single page of users and groups, I suspect the global override is the main problem area I'm running into.

I would guess in this sense it would be very useful for the pages to strengthen the language around more frequently than the recommended timing of no more than once every 2 hours. Using the work recommended doesn't seem appropriate if it's demonstrably impossible to do anything more often.

I'll have to see what I can do without calling the user or groups lists as much as possible I guess. It's suboptimal, as I didn't want to have to build out a caching solution.

[bhusler027]

I read somewhere that adding the 'user-agent' to the header allows it work. Added it myself and haven't run into 'too many requests' error since.

[davegreen]

That's really interesting and helpful, thanks! I'll try this and see what happens. Are you adding a specific user agent value, or a blank one?

…

On Tue, 26 Mar 2024, 21:24 bhusler027, ***@***.***> wrote: I read somewhere that adding the 'user-agent' to the header allows it work. Added it myself and haven't run into 'too many requests' error since. — Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFZMYOSIPWXRG3DMMKBJFLY2HRPRAVCNFSM6AAAAAAXULGNSKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGQ4TQNJUGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>