1. The Anatomy of Chunked File Uploads
Instead of streaming a 5GB file in a single HTTP request, the client splits the file into uniform 5MB binary chunks using the browser's File.slice() API. Each chunk is dispatched independently with metadata including chunk index, total chunks, and a unique upload session identifier. Failed chunks are automatically retried up to three times with exponential backoff without having to restart the entire upload.
# Django view handling atomic chunk ingestion
class ChunkedUploadView(APIView):
def post(self, request):
upload_id = request.data.get('upload_id')
chunk_index = int(request.data.get('chunk_index'))
total_chunks = int(request.data.get('total_chunks'))
file_obj = request.FILES.get('file')
temp_dir = os.path.join(settings.TEMP_UPLOAD_DIR, upload_id)
os.makedirs(temp_dir, exist_ok=True)
chunk_path = os.path.join(temp_dir, f"chunk_{chunk_index:04d}")
with open(chunk_path, 'wb+') as destination:
for chunk in file_obj.chunks():
destination.write(chunk)
if len(os.listdir(temp_dir)) == total_chunks:
# Trigger asynchronous background assembly task
assemble_chunks_task.delay(upload_id, total_chunks)
return Response({'status': 'assembling'})
return Response({'status': 'chunk_received', 'index': chunk_index})Key Implementation Takeaways:
- ✓Chunk size of 5MB-10MB offers optimal balance between network overhead and retry efficiency.
- ✓Use temporary local scratch directories with deterministic zero-padded filenames.
- ✓Delegate final file assembly to Celery/background workers to keep HTTP responses instantaneous.
2. Streaming Assembly and Checksum Verification
When all chunks arrive, assembling them must not load the entire multi-gigabyte file into memory. We stream chunks sequentially into the final destination file in 64KB buffers while computing a running SHA-256 checksum. If the assembled checksum matches the client-provided checksum, the file is uploaded directly to cloud object storage (AWS S3 or GCP Cloud Storage) and local temporary artifacts are purged.
import hashlib
def assemble_file(upload_id: str, total_chunks: int, output_path: str):
hasher = hashlib.sha256()
with open(output_path, 'wb') as outfile:
for idx in range(total_chunks):
chunk_file = os.path.join(settings.TEMP_UPLOAD_DIR, upload_id, f"chunk_{idx:04d}")
with open(chunk_file, 'rb') as infile:
while chunk := infile.read(65536):
outfile.write(chunk)
hasher.update(chunk)
os.remove(chunk_file) # Clean up eagerly
return hasher.hexdigest()Key Implementation Takeaways:
- ✓Read and write in fixed 64KB buffers to keep process RAM usage under 30MB regardless of file size.
- ✓Clean up chunk files eagerly during assembly to prevent disk space exhaustion.
- ✓Verify cryptographic checksums before promoting files to permanent cloud storage.
3. Memory-Safe PDF Generation in Constrained Docker Containers
Generating comprehensive certification and reporting PDFs with hundreds of pages can quickly exceed the 512MB RAM ceiling of cloud container instances. By utilizing Python generators and streaming HTTP responses rather than accumulating huge strings in memory, we maintained container memory usage below 180MB at all times.
Key Implementation Takeaways:
- ✓Avoid loading entire document trees into memory when rendering large PDFs.
- ✓Use temporary disk-backed spooling for image-heavy document rendering.
- ✓Set strict memory limits in Docker compose and monitor with Prometheus.
Summary & Final Thoughts
Resilient systems anticipate network volatility. Implementing chunked uploads and buffer-based streaming pipelines completely eliminated file upload timeouts, reduced server memory footprints, and created an uninterrupted user experience even on unstable mobile connections.