Week 06 — Async Systems | SoftBricks Academy

What You'll Learn

Real AI systems don't run synchronously. You'll learn how to queue long-running agent tasks, manage state across workers, handle failures gracefully, and give users real-time status on background jobs.

Session Schedule

Day	Time	Focus
Saturday	8:00 - 11:00 PM WAT	Async Architecture & Celery
Sunday	8:00 - 11:00 PM WAT	Pipeline Build

Pre-Requisites

Weeks 01-05 completed
Docker running (Redis)
Basic understanding of REST APIs

Topics Covered

Celery Task Workers

Task definition, worker processes, task routing, concurrency settings. The backbone of async processing in Python.

Celery Workers Task Routing

Redis as Broker & Result Backend

Message queue patterns, result storage, TTL, pub/sub. How Redis ties everything together as the central nervous system.

Redis Message Queue Pub/Sub

DynamoDB for Job State

Job records, status tracking, GSI for queries, TTL cleanup. Persistent state for long-running async workflows.

DynamoDB Job State Status Tracking

Retry Strategies & Dead-Letter Queues

Exponential backoff, max retries, DLQ for failed tasks, alerting. Build systems that fail gracefully and recover automatically.

Retries Backoff Dead-Letter Queue

Webhook Callbacks

POST callbacks on completion, retry delivery, signature verification. Notify external systems when async work finishes.

Webhooks Callbacks Signatures

Weekly Build: Async Document Pipeline

Build an async pipeline: user uploads document → Celery queues processing → chunks/embeds/indexes → webhook notification on completion.

Architecture

POST /documents (upload)
    |
    v
FastAPI → Celery Task Queue (Redis broker)
    |
    v
Worker: Document Processing
    ├── Step 1: Parse (PDF/DOCX → text)
    ├── Step 2: Chunk (semantic splitting)
    ├── Step 3: Embed (OpenAI embeddings)
    ├── Step 4: Index (PGVector upsert)
    └── Step 5: Update job status
    |
    v
GET /jobs/{id} → Status polling
POST webhook → Completion notification

Key Files

File	Purpose
`api/routes.py`	FastAPI endpoints
`tasks/document_pipeline.py`	Celery task
`tasks/celery_config.py`	Broker config
`models/job.py`	Job state model
`webhooks/notify.py`	Callback sender

Resources

Required Reading

Celery Official Documentation — Getting Started
Redis Pub/Sub Documentation
AWS DynamoDB Developer Guide — Working with Items

Code Repository

Clone the bootcamp repo and switch to the week-06 branch:

git clone https://github.com/softbricks-academy/agentic-ai-bootcamp.git
cd agentic-ai-bootcamp
git checkout week-06

Session Recording

Recording will be available within 24 hours after the live session. Check the WhatsApp group for the link.

Homework

Due before Week 7 live session.

Complete the async document pipeline — push your code to the bootcamp repo
Add retry logic with exponential backoff — handle transient failures in the embedding step
Implement a dead-letter queue — capture and log permanently failed tasks
Build a status dashboard — simple HTML page that polls the job status endpoint

Week 06: Async Systems — Celery, Redis & Task Queues