[Feature] Add support for token-based rate-limiting in RemoteInferenceEngine

[jgreer013]

Feature request

Add support for token-based rate-limiting for both input and output tokens when using RemoteInferenceEngine.

Motivation / references

Many API providers in addition to request-based limits also have token-based limits for both input and output tokens:

https://console.anthropic.com/settings/limits
https://platform.openai.com/settings/organization/limits

If we don't allow users to configure these limits, their requests end up failing after a certain point.

Your contribution

1. Update remote_params.py to add input and output token limits. Also modify it to use RPM rather than a custom politeness policy:

   oumi/src/oumi/core/configs/params/remote_params.py

   Line 45 in eee596f

   politeness_policy: float = 0.0

2. Update RemoteInferenceEngine to use "requests per minute" rather than a custom politeness policy
3. Update RemoteInferenceEngine to sleep either when the RPM or the TPM limit is reached (whichever comes first):

   oumi/src/oumi/inference/remote_inference_engine.py

   Line 427 in eee596f

   await asyncio.sleep(remote_params.politeness_policy)

Most APIs specify in a successful API response how many input and output tokens were processed for that request.

[shivam096]

Hey @jgreer013 ,

If this is available, then I would like to take this issue up. I am new to the repository, hence wanted to start with small issues :)

[nikg7]

here you go @shivam096

[shivam096]

Thank you @nikg4

Started working on it .

A follow-up question: What should be the default values for the input and output limits? Are there any defined values ?

[nikg7]

I don't have an exact number in mind. You can start with any reasonable value (e.g., as used by common providers). Let's revisit exact numbers during code review.

[shivam096]

Hi @nikg4 ,

I wanted to check one thing - I am keeping the politeness_policy as well since a lot of the existing inference engines use this. If I go with removing this, it would result in updating a lot of files, which I believe would not be an ideal update.

Also, I just wanted to check how I can test this. Is there a recipe for this that I could run, or is there a specific execution doc I could refer to?

On a separate note, most of the token limits are per minute, which means that I would need to keep a rolling track of per-minute-based limits and wanted to confirm if these are the expected limitations that are to be implemented.

[nikg7]

@shivav096 Please first review the available Python libraries for rate limiting (ratelimiter , ratelimit , https://asynciolimiter.readthedocs.io/en/latest/, and others…) . I recommend using one of them instead of building from scratch.

[AnandVishesh1301]

Hi @nikg4 , I Can pick this up if its still available! Looking to get familiar with the codebase!