Stage 7 · Master
LLM Foundations for Operators
Cost, Latency & Data Privacy
Token budgets, self-hosted vs API models, and keeping secrets out of prompts.
Understanding Token Costs
LLM API providers charge per token. Pricing varies by model and whether tokens are input or output. Output tokens cost 2 to 4 times more than input tokens, which means shorter responses save money.
| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|---|---|---|
| GPT-3.5 Turbo | $0.50 | $1.50 |
| GPT-4 | $30.00 | $60.00 |
| GPT-4 Turbo | $10.00 | $30.00 |
| Claude 3 Haiku | $0.25 | $1.25 |
| Claude 3 Sonnet | $3.00 | $15.00 |
A single GPT-4 Turbo request with a 10K token context window and a 1K token response costs roughly $0.13. At 1000 requests per day, that is $130 per day or about $4,000 per month.
Latency Budgets
LLM response latency varies dramatically by model, context size, and output length. Plan your latency budgets around the user experience you need.
| Model | Typical Latency | Streaming First Token | Best For |
|---|---|---|---|
| GPT-3.5 Turbo | 1 to 3 seconds | ~200ms | Classification, quick answers |
| GPT-4 Turbo | 3 to 10 seconds | ~500ms | Complex analysis, large context |
| GPT-4 | 5 to 15 seconds | ~800ms | Deep reasoning, code generation |
| Self-hosted Llama | 1 to 5 seconds | ~100ms | Low-latency, high-volume tasks |
Streaming responses show the first token in milliseconds and let users read as the answer generates. It improves perceived latency by 5 to 10 times even if total response time is identical.
Privacy Boundaries
When you send data to an LLM API, that data leaves your infrastructure. Understand what is sent, where it goes, and what the provider does with it.
- OpenAI API data is not used for training by default. It is retained for 30 days for abuse monitoring.
- AWS Bedrock keeps data within your AWS VPC. No data leaves your account.
- Self-hosted models keep everything on your infrastructure.
- API key credentials must never appear in prompts or logs.
- PII in prompts may be transmitted to third-party providers.
Never include API keys, passwords, tokens, or PII in LLM prompts. Use secret managers and redaction libraries to scrub sensitive data before it reaches the model.
Self-Hosted vs API Models
| Factor | API Models | Self-Hosted |
|---|---|---|
| Setup | None | GPU infrastructure required |
| Cost model | Per token | Fixed infrastructure cost |
| Privacy | Data leaves your VPC | Data stays on-premises |
| Quality | State of the art | Depends on model choice |
| Scaling | Provider handles it | You handle it |
Practical Cost Controls
class TokenBudget:
def __init__(self, max_input_tokens=4000, max_output_tokens=1000):
self.max_input = max_input_tokens
self.max_output = max_output_tokens
def truncate_context(self, context_chunks, query):
"""Keep only the most relevant chunks within budget."""
selected = []
total = len(query) // 4 # Rough token estimate
for chunk in context_chunks:
chunk_tokens = len(chunk) // 4
if total + chunk_tokens > self.max_input:
break
selected.append(chunk)
total += chunk_tokens
return selected
def estimate_cost(self, model, input_tokens, output_tokens):
"""Estimate cost in USD."""
pricing = {
"gpt-4-turbo": (0.00001, 0.00003),
"gpt-3.5-turbo": (0.0000005, 0.0000015),
}
input_cost, output_cost = pricing.get(model, (0, 0))
return input_tokens * input_cost + output_tokens * output_costImplement token budgets in your application layer. Truncate context to stay within limits and estimate cost before making API calls.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.