When building AI applications, one of the first architectural decisions is choosing between streaming and batch API modes. Both have distinct advantages, and the right choice depends on your use case, user experience requirements, and cost considerations. This guide covers everything you need to know to make the right call.
Streaming (Server-Sent Events or SSE) sends the AI model's response incrementally — token by token as the model generates it. Instead of waiting 10 seconds for a complete 500-word response, the user sees the first word appear in under a second, and text flows in continuously like someone typing in real-time.
This creates a dramatically better user experience. Studies show that perceived latency matters more than actual latency — users are willing to wait longer if they see progress immediately. Streaming exploits this psychological principle perfectly.
Batch API processes requests without streaming. The client sends a request, waits for the model to generate the complete response, and receives it all at once. Batch can also refer to submitting multiple requests in a single API call for asynchronous processing.
Batch is ideal for background processing, bulk operations, and scenarios where the full response is needed before any action can be taken. OpenAI's batch API offers a 50% discount for requests processed within 24 hours, making it highly cost-effective for non-urgent workloads.
| Aspect | Streaming | Batch |
|---|---|---|
| First token latency | 0.3-0.8s | 5-30s (full response) |
| User experience | Real-time typing effect | Loading spinner → result |
| Per-token cost | Standard | Standard (or 50% off with batch API) |
| Implementation complexity | Medium | Simple |
| Best for | Chatbots, interactive UIs | Bulk processing, background jobs |
| Error recovery | Complex (partial responses) | Simple (retry entire request) |
| Connection requirements | Long-lived connection | Short request-response |
Here's how to implement streaming with Token1.US:
from openai import OpenAI
client = OpenAI(api_key="your-key", base_url="https://discount-token.com/v1")
# Streaming response - tokens arrive incrementally
stream = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a short poem about APIs"}],
stream=True, # Enable streaming
max_tokens=200
)
full_text = ""
for chunk in stream:
if chunk.choices[0].delta.content is not None:
token = chunk.choices[0].delta.content
print(token, end="", flush=True) # Print as it arrives
full_text += token
print(f"\n\nComplete response: {full_text}")
const response = await fetch('https://discount-token.com/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer your-api-key'
},
body: JSON.stringify({
model: 'gpt-4o',
messages: [{role: 'user', content: 'Hello!'}],
stream: true
})
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const {done, value} = await reader.read();
if (done) break;
const text = decoder.decode(value);
// Parse SSE data lines
text.split('\n').forEach(line => {
if (line.startsWith('data: ') && line !== 'data: [DONE]') {
const data = JSON.parse(line.slice(6));
const token = data.choices[0]?.delta?.content;
if (token) {
document.getElementById('output').textContent += token;
}
}
});
}
# Standard batch request (non-streaming)
response = client.chat.completions.create(
model="deepseek-v3", # Use cheaper model for bulk tasks
messages=[{"role": "user", "content": "Classify this text: ..."}],
stream=False, # Default - wait for full response
max_tokens=100
)
result = response.choices[0].message.content
# Processing multiple requests efficiently
import concurrent.futures
texts = ["text 1", "text 2", "text 3", ...] # 1000+ items
def classify(text):
response = client.chat.completions.create(
model="deepseek-v3", # $0.14/$0.28 - cheapest option
messages=[{"role": "user", "content": f"Classify: {text}"}],
max_tokens=50
)
return response.choices[0].message.content
# Process in parallel (respect rate limits!)
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
results = list(executor.map(classify, texts))
Users expect real-time responses in chat interfaces. Streaming makes your bot feel responsive and human. The typing effect also masks generation time for long responses.
AI writing assistants, code generators, and content tools benefit from streaming because users can start reading (or copying) the beginning of the response while the rest is still being generated.
When AI generates summaries or insights for dashboards, streaming lets users see results immediately rather than staring at a loading spinner.
For text-to-speech integration, streaming lets you start synthesizing audio from the first tokens before the full response is ready, reducing end-to-end latency significantly.
Classifying 10,000 support tickets, summarizing 500 articles, or extracting entities from a large dataset — these don't need real-time responses. Use DeepSeek-V3 for maximum cost efficiency.
Monthly reports, analytics summaries, and automated content pipelines run in the background. Batch mode is simpler to implement and debug.
When you need JSON, tables, or structured data from the API, batch mode ensures you receive the complete, well-formed response. Partial streaming responses may be malformed.
Email classification, content moderation queues, and batch translations can be processed asynchronously. Take advantage of batch API discounts for 50% cost savings.
The per-token pricing is identical for streaming and non-streaming requests. However, batch processing can be cheaper in practice:
| Strategy | Cost (1M requests) | Notes |
|---|---|---|
| Streaming + GPT-4o | $3,250 | Real-time UX, premium model |
| Batch + GPT-4o | $1,625 | 50% batch discount |
| Batch + DeepSeek-V3 | $91 | Cheapest option for bulk |
| Streaming + DeepSeek-V3 | $182 | Best balance: real-time + cheap |
Pro tip: For most applications, the best strategy is streaming with DeepSeek-V3 for user-facing interactions and batch with DeepSeek-V3 for background processing. This gives you real-time UX at the lowest possible cost.
Streaming introduces a unique challenge: what happens when the connection drops mid-stream? You've already received partial content, but the response is incomplete. Here's how to handle it:
def stream_with_recovery(messages, model="gpt-4o", max_retries=3):
for attempt in range(max_retries):
collected = ""
try:
stream = client.chat.completions.create(
model=model,
messages=messages,
stream=True
)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
collected += token
yield token # Stream to your UI
return # Success
except Exception as e:
if attempt < max_retries - 1:
# Append what we have and continue with remaining prompt
messages = messages + [{"role": "assistant", "content": collected}]
messages = messages + [{"role": "user", "content": "Continue from where you left off."}]
time.sleep(2 ** attempt)
else:
raise
No. The token count is identical regardless of streaming mode. The model generates the same response; only the delivery mechanism differs.
Yes. Just change the stream parameter in your request. Token1.US supports both modes across all models with no configuration changes.
Yes, but function call arguments are also streamed incrementally. You need to accumulate the partial JSON and parse it only after the stream completes. Most SDKs handle this automatically.
Token1.US supports streaming and batch across all models. Start building today.