Stage 4 · Provision
Design Case Studies
Design a File Storage Service
Multipart uploads, object metadata, checksum validation, lifecycle policies, and CDN delivery.
Requirements
A file storage service allows users to upload, download, and manage files. It must support large files (up to 5GB), provide durable storage (99.999999999% durability), enable CDN delivery, and manage file lifecycle. Upload speed and download speed are critical.
Storage Architecture
Client
│
├── Upload ──► API Gateway ──► Upload Service ──► S3
│ ├── Multipart upload
│ ├── Checksum validation
│ └── Metadata store (DynamoDB)
│
├── Download ──► CDN (CloudFront) ──► S3 Origin
│ └── Edge cache
│
└── Manage ──► API Gateway ──► File Service ──► DynamoDB (metadata)
└── S3 (data)S3 stores the actual file data. DynamoDB stores metadata (name, size, type, checksum). CloudFront caches and delivers files from edge locations.
Multipart Uploads
Multipart uploads split large files into parts, upload them in parallel, and reassemble on the server. This provides resume-on-failure (only retry failed parts), parallel upload for speed, and support for files larger than 5GB.
import boto3
s3 = boto3.client('s3')
# 1. Initiate multipart upload
response = s3.create_multipart_upload(Bucket='files', Key='video.mp4')
upload_id = response['UploadId']
# 2. Upload parts in parallel
parts = []
for i, chunk in enumerate(file_chunks(file_path, chunk_size=10*1024*1024)):
part = s3.upload_part(
Bucket='files', Key='video.mp4',
UploadId=upload_id, PartNumber=i+1,
Body=chunk
)
parts.append({'PartNumber': i+1, 'ETag': part['ETag']})
# 3. Complete multipart upload
s3.complete_multipart_upload(
Bucket='files', Key='video.mp4',
UploadId=upload_id,
MultipartUpload={'Parts': parts}
)Multipart uploads split files into 5MB-5GB parts. Parts are uploaded in parallel for speed. If a part fails, only that part is retried. This is essential for large file uploads.
Object Metadata
file_metadata:
id: "file_abc123"
user_id: "user_456"
name: "presentation.pdf"
size: 5242880 # 5MB
content_type: "application/pdf"
checksum_sha256: "a1b2c3..."
storage_path: "s3://files/user_456/abc123/presentation.pdf"
created_at: "2025-01-15T10:30:00Z"
updated_at: "2025-01-15T10:30:00Z"
tags: ["work", "presentation"]
status: "active" # active, archived, deletedMetadata is stored in DynamoDB for fast lookups. The actual file is in S3. Checksums validate file integrity on download.
Lifecycle Policies
- Transition to IA — Move infrequently accessed files to S3 IA after 30 days.
- Transition to Glacier — Archive files older than 90 days.
- Expiration — Delete temporary files after 7 days.
- Multipart cleanup — Abort incomplete multipart uploads after 7 days.
- Version cleanup — Remove old versions after 30 days.
CDN Delivery
Serve file downloads through CloudFront. Set Cache-Control headers based on file type. Images and static assets get long TTLs. User-uploaded content gets shorter TTLs. Use signed URLs for private files.
Always validate checksums on upload and download. File corruption happens at every layer: network, disk, CDN. SHA-256 checksums catch corruption and prevent serving damaged files to users.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.