By the end of this article, you will have a working Python microservices skeleton that handles failures gracefully, retries intelligently, and never takes your whole platform down because of one bad service.
What Is a Resilient Microservices Architecture?
A microservice is a small, independently deployable service that does one thing well like handling payments, managing users, or sending emails. Instead of one giant application (a monolith), you split your system run into new online casinos not using GamStop many small services that talk to each other.
Resilience means the system keeps working maybe in a degraded state even when parts of it fail.
Think of it like a power grid: if one neighborhood loses power, the rest of the city stays lit. That’s what we’re building.
Prerequisites
- Python 3.10+
- Basic knowledge of REST APIs
pipand a virtual environment- Docker (optional, for running services locally)
Install the required libraries:
pip install fastapi uvicorn httpx tenacity pybreaker structlog prometheus-clientProject Structure
resilient-microservices/
│
├── services/
│ ├── user_service/
│ │ ├── main.py
│ │ └── health.py
│ ├── order_service/
│ │ ├── main.py
│ │ ├── circuit_breaker.py
│ │ └── retry.py
│ └── notification_service/
│ └── main.py
│
├── gateway/
│ └── main.py
│
├── shared/
│ ├── logger.py
│ └── models.py
│
└── docker-compose.ymlBuild the Base Services with FastAPI
Each service is a small, independent FastAPI application.
User Service (services/user_service/main.py)
from fastapi import FastAPI
from pydantic import BaseModel
import structlog
log = structlog.get_logger()
app = FastAPI(title="User Service")
class User(BaseModel):
id: int
name: str
email: str
# Fake in-memory database
USERS = {
1: User(id=1, name="Alice", email="alice@example.com"),
2: User(id=2, name="Bob", email="bob@example.com"),
}
@app.get("/users/{user_id}", response_model=User)
async def get_user(user_id: int):
log.info("Fetching user", user_id=user_id)
if user_id not in USERS:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail="User not found")
return USERS[user_id]
@app.get("/health/live")
async def liveness():
return {"status": "alive"}
@app.get("/health/ready")
async def readiness():
# Check dependencies here (DB, cache, etc.)
return {"status": "ready"}
Order Service (services/order_service/main.py)
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx
import structlog
log = structlog.get_logger()
app = FastAPI(title="Order Service")
USER_SERVICE_URL = "http://localhost:8001"
class Order(BaseModel):
order_id: int
user_id: int
item: str
total: float
@app.post("/orders", response_model=dict)
async def create_order(order: Order):
log.info("Creating order", order_id=order.order_id, user_id=order.user_id)
# Call User Service to validate user exists
async with httpx.AsyncClient(timeout=3.0) as client:
try:
resp = await client.get(f"{USER_SERVICE_URL}/users/{order.user_id}")
resp.raise_for_status()
except httpx.HTTPStatusError as e:
raise HTTPException(status_code=404, detail="User not found")
except httpx.RequestError as e:
raise HTTPException(status_code=503, detail="User service unavailable")
return {"message": "Order created", "order_id": order.order_id}
@app.get("/health/live")
async def liveness():
return {"status": "alive"}
@app.get("/health/ready")
async def readiness():
return {"status": "ready"}
Add Retry with Exponential Backoff
Transient failures (network blips, brief service restarts) should be retried automatically but with increasing delays so you don’t hammer a recovering service.
services/order_service/retry.py
import httpx
from tenacity import (
retry,
stop_after_attempt,
wait_exponential,
retry_if_exception_type,
before_sleep_log,
)
import structlog
import logging
log = structlog.get_logger()
# Retry up to 3 times with exponential backoff: 1s → 2s → 4s
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=8),
retry=retry_if_exception_type((httpx.RequestError, httpx.TimeoutException)),
before_sleep=before_sleep_log(logging.getLogger("tenacity"), logging.WARNING),
)
async def fetch_user_with_retry(client: httpx.AsyncClient, user_service_url: str, user_id: int):
"""
Fetches a user from the User Service with automatic retry on transient errors.
Raises immediately on 4xx (non-transient) errors.
"""
resp = await client.get(f"{user_service_url}/users/{user_id}", timeout=3.0)
if resp.status_code == 404:
# Don't retry — user genuinely doesn't exist
resp.raise_for_status()
if resp.status_code >= 500:
# Server error — raise so tenacity can retry
raise httpx.RequestError(f"Server error: {resp.status_code}")
return resp.json()Usage in Order Service:
from retry import fetch_user_with_retry
async with httpx.AsyncClient() as client:
user = await fetch_user_with_retry(client, USER_SERVICE_URL, order.user_id)Implement the Circuit Breaker
A Circuit Breaker monitors failures. When they exceed a threshold, it “trips” blocking further calls for a cooldown period so a struggling service isn’t overwhelmed.
[CLOSED] ──(failures > 5)──▶ [OPEN] ──(30s timeout)──▶ [HALF-OPEN]
▲ │
└───────────(success)─────────────────────────────────────┘
services/order_service/circuit_breaker.py
import pybreaker
import httpx
import structlog
log = structlog.get_logger()
class CircuitBreakerListener(pybreaker.CircuitBreakerListener):
def state_change(self, cb, old_state, new_state):
log.warning(
"Circuit breaker state changed",
name=cb.name,
old=str(old_state),
new=str(new_state),
)
# Trip after 5 failures; stay open for 30 seconds
user_service_breaker = pybreaker.CircuitBreaker(
fail_max=5,
reset_timeout=30,
name="UserService",
listeners=[CircuitBreakerListener()],
)
@user_service_breaker
async def call_user_service(client: httpx.AsyncClient, url: str, user_id: int):
"""
Wrapped call to User Service. The circuit breaker monitors this function.
If it fails 5 times, the breaker opens and immediately raises
CircuitBreakerError for 30 seconds — no actual HTTP calls are made.
"""
resp = await client.get(f"{url}/users/{user_id}", timeout=3.0)
resp.raise_for_status()
return resp.json()
Using it in Order Service with a fallback:
import pybreaker
from circuit_breaker import call_user_service
async with httpx.AsyncClient() as client:
try:
user = await call_user_service(client, USER_SERVICE_URL, order.user_id)
except pybreaker.CircuitBreakerError:
# Fallback: return cached/default response instead of crashing
log.error("Circuit open — using fallback for user lookup")
user = {"id": order.user_id, "name": "Unknown", "email": ""}
except httpx.HTTPStatusError:
raise HTTPException(status_code=404, detail="User not found")Structure Logging with Correlation IDs
When a request touches 5 services, you need a single correlation ID to trace it through all of them. Structured logging makes this searchable in tools like Grafana Loki or ELK.
shared/logger.py
import structlog
import uuid
from fastapi import Request
from starlette.middleware.base import BaseHTTPMiddleware
# Configure structlog for JSON output
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.processors.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer(),
]
)
class CorrelationIDMiddleware(BaseHTTPMiddleware):
"""
Injects a correlation ID into every request.
Reads X-Correlation-ID header if provided, otherwise generates one.
Passes it downstream in the response header too.
"""
async def dispatch(self, request: Request, call_next):
correlation_id = request.headers.get("X-Correlation-ID", str(uuid.uuid4()))
structlog.contextvars.clear_contextvars()
structlog.contextvars.bind_contextvars(
correlation_id=correlation_id,
service=request.app.title,
path=request.url.path,
method=request.method,
)
response = await call_next(request)
response.headers["X-Correlation-ID"] = correlation_id
return responseAttach it to any FastAPI app:
from shared.logger import CorrelationIDMiddleware
app.add_middleware(CorrelationIDMiddleware)Sample log output (JSON):
{
"event": "Creating order",
"order_id": 42,
"user_id": 1,
"correlation_id": "e3f1c2a9-...",
"service": "Order Service",
"level": "info",
"timestamp": "2026-04-16T10:22:01Z"
}Expose Prometheus Metrics
Add observability by exposing metrics that Prometheus can scrape specifically the RED metrics: Rate, Errors, Duration.
# Add to any service's main.py
from prometheus_client import Counter, Histogram, make_asgi_app
from starlette.routing import Mount
REQUEST_COUNT = Counter(
"http_requests_total",
"Total HTTP requests",
["service", "method", "endpoint", "status"]
)
REQUEST_LATENCY = Histogram(
"http_request_duration_seconds",
"HTTP request latency",
["service", "endpoint"]
)
CIRCUIT_BREAKER_TRIPS = Counter(
"circuit_breaker_trips_total",
"Number of times circuit breaker tripped",
["service", "dependency"]
)
# Mount /metrics endpoint for Prometheus scraping
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)API Gateway
The gateway is the single entry point. It handles routing, auth headers, and injects correlation IDs before forwarding to services.
gateway/main.py
from fastapi import FastAPI, Request, HTTPException
import httpx
import uuid
app = FastAPI(title="API Gateway")
ROUTES = {
"/users": "http://localhost:8001",
"/orders": "http://localhost:8002",
}
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE"])
async def gateway(path: str, request: Request):
# Match route prefix
target_base = None
for prefix, url in ROUTES.items():
if f"/{path}".startswith(prefix):
target_base = url
break
if not target_base:
raise HTTPException(status_code=404, detail="Route not found")
# Forward request with correlation ID
headers = dict(request.headers)
headers["X-Correlation-ID"] = headers.get("X-Correlation-ID", str(uuid.uuid4()))
async with httpx.AsyncClient(timeout=5.0) as client:
try:
resp = await client.request(
method=request.method,
url=f"{target_base}/{path}",
headers=headers,
content=await request.body(),
)
return resp.json()
except httpx.RequestError:
raise HTTPException(status_code=503, detail="Upstream service unavailable")
Docker Compose (Run Everything Locally)
docker-compose.yml
version: "3.9"
services:
user-service:
build: ./services/user_service
ports:
- "8001:8000"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health/live"]
interval: 10s
timeout: 5s
retries: 3
order-service:
build: ./services/order_service
ports:
- "8002:8000"
depends_on:
user-service:
condition: service_healthy
environment:
- USER_SERVICE_URL=http://user-service:8000
gateway:
build: ./gateway
ports:
- "8000:8000"
depends_on:
- user-service
- order-service
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
grafana:
image: grafana/grafana
ports:
- "3000:3000"
Start everything:
docker-compose up --buildTesting Resilience
Simulate a service failure:
# Kill the user service
docker-compose stop user-service
# Try to create an order should get fallback, not 500 crash
curl -X POST http://localhost:8000/orders \
-H "Content-Type: application/json" \
-d '{"order_id": 1, "user_id": 1, "item": "Laptop", "total": 999.99}'Watch the circuit breaker trip:
# Send 6 requests rapidly while user-service is down
for i in {1..6}; do curl -s http://localhost:8000/orders -X POST \
-H "Content-Type: application/json" \
-d '{"order_id": '$i', "user_id": 1, "item": "Test", "total": 10}'; doneAfter 5 failures, the 6th request gets rejected instantly no HTTP call made and the log shows Circuit breaker state changed: CLOSED → OPEN.
The golden rule of resilient microservices: design for failure, not for success. Every external call will eventually fail. Your job is to decide gracefully what happens when it does.