Rate limiting is one of the most common challenges developers face when integrating AI APIs. Whether you're using OpenAI, Claude, DeepSeek, or GLM, every provider imposes rate limits that can cause unexpected failures in production. This guide breaks down everything you need to know about AI API rate limiting and how to handle it effectively.
Rate limiting is a mechanism that AI API providers use to control the number of requests a client can make within a specific time window. It serves three purposes: preventing server overload, ensuring fair resource allocation among all users, and protecting against abuse or DDoS attacks.
When you hit a rate limit, the API responds with an HTTP 429 "Too Many Requests" status code. This tells your application to slow down and retry later. Understanding how rate limits work across different providers is essential for building reliable AI-powered applications.
RPM limits the number of API calls you can make in a 60-second window. For example, if your plan allows 60 RPM, you can make one request per second on average. Bursting beyond this limit triggers a 429 error. This is the most straightforward rate limit type and applies to all AI API providers.
TPM limits the total number of tokens (input + output) processed per minute across all your requests. This is particularly important for large prompts or long completions. A single request with a 10,000-token prompt and a 2,000-token completion consumes 12,000 tokens from your TPM budget.
Some providers also limit the number of simultaneous in-flight requests. If your limit is 10 concurrent requests and you already have 10 pending, new requests will be rejected until some complete. This is common with streaming endpoints and long-running completions.
In addition to per-minute limits, many providers enforce daily or monthly quotas. These are typically based on your billing tier and can be increased by upgrading your plan or contacting sales. Daily quotas reset at midnight UTC for most providers.
| Provider/Model | Free/Tier 1 RPM | Paid RPM | TPM |
|---|---|---|---|
| GPT-4o | 500 RPM | 10,000 RPM | 30K-10M |
| Claude 3.5 Sonnet | 50 RPM | 1,000-4,000 RPM | 40K-400K |
| DeepSeek-V3 | 60 RPM | Unlimited* | No hard limit |
| GLM-4-Plus | 30 RPM | 300-2,000 RPM | 50K-500K |
*DeepSeek's "unlimited" applies to paid tiers but may still throttle under high load.
The HTTP 429 status code is your signal to slow down. Here's a robust implementation using exponential backoff with jitter:
import time
import random
from openai import OpenAI
client = OpenAI(
api_key="your-token1-api-key",
base_url="https://discount-token.com/v1"
)
def call_with_retry(messages, max_retries=5):
for attempt in range(max_retries):
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=1000
)
return response
except Exception as e:
if "429" in str(e) or "rate_limit" in str(e).lower():
# Exponential backoff with jitter
wait_time = min(2 ** attempt + random.uniform(0, 1), 60)
print(f"Rate limited. Retrying in {wait_time:.1f}s...")
time.sleep(wait_time)
else:
raise
raise Exception(f"Failed after {max_retries} retries")
One of the most effective strategies for avoiding rate limits is using a unified API gateway like Token1.US. Instead of being locked into a single provider's rate limits, a gateway lets you:
Here's how to implement automatic model failover with Token1.US:
# Automatic failover: try primary, fall back to cheaper alternatives
fallback_chain = [
("deepseek-v3", "cheapest, rarely rate-limited"),
("glm-4-plus", "good alternative for most tasks"),
("gpt-4o", "premium, may have stricter limits"),
("claude-3-5-sonnet", "high quality, moderate limits")
]
for model, description in fallback_chain:
try:
response = client.chat.completions.create(
model=model,
messages=messages
)
return response
except rate_limit_error:
print(f"{model} rate-limited, trying next: {description}")
continue
raise Exception("All models exhausted")
Don't wait for 429 errors to find out you're hitting limits. Proactively monitor your rate limit usage using response headers:
| Header | Meaning |
|---|---|
X-RateLimit-Limit | Your total rate limit per window |
X-RateLimit-Remaining | Requests remaining in current window |
X-RateLimit-Reset | Unix timestamp when the window resets |
Retry-After | Seconds to wait before retrying (on 429) |
While parallel requests improve throughput, sending 100 simultaneous requests when your limit is 60 RPM guarantees 40 failures. Use a semaphore or queue to control concurrency.
The Retry-After header tells you exactly how long to wait. Ignoring it and retrying immediately wastes requests and can lead to longer lockout periods.
If 100 clients all hit a rate limit simultaneously and all retry at the same time after the same delay, they'll all hit the limit again. Jitter prevents this thundering herd problem.
Single-provider dependencies create a single point of failure. If that provider rate-limits you, your entire application stalls. A multi-model gateway eliminates this risk.
The API returns an HTTP 429 error with a Retry-After header indicating how many seconds to wait. Your request is not processed — you must retry it after the cooldown period.
Yes. Most providers increase rate limits based on usage history and payment tier. With Token1.US, you can also aggregate limits across multiple models, effectively multiplying your throughput without needing tier upgrades.
Yes. Streaming requests consume the same RPM and TPM budget as non-streaming requests. However, streaming can help you stay under TPM limits by processing tokens incrementally rather than buffering a large response.
Token1.US aggregates rate limits across all supported models. When one model hits its limit, requests automatically failover to alternative models. This gives you effectively unlimited throughput as long as at least one model is available.
Token1.US provides a unified, OpenAI-compatible API gateway with automatic rate limit management across GPT-4o, Claude 3.5 Sonnet, DeepSeek-V3, and GLM-4-Plus. Get started in 5 minutes.