Orchestration Engine
Client project: Tenable & ServiceNow (Crest Data Client Engagement)
Technical Deep Dive
Built a high-throughput event pipeline with a React operator dashboard, Redis idempotency keys, exponential backoff, and dead-letter queue isolation, while owning cross-timezone on-call for the Tenable and ServiceNow integrations.
Client Context
Tenable and ServiceNow integrations hit uncoordinated API rate limits, duplicate vulnerability ingestion, and worker starvation during peak sync events. On-call was cross-timezone, and most of the effort went into gathering context before a fix could start.
Execution
Built a multi-queue Celery + RabbitMQ architecture with queue isolation per vendor, Redis atomic idempotency keys, exponential backoff with full jitter, and dead-letter queues for poisoned messages. When a downstream rate limit set off a retry storm, drained poisoned messages into DLQ quarantine, added per-queue concurrency ceilings and prefetch limits, and wrote runbooks and alerting thresholds.
Outcome
Sustained 1,000 events/sec, eliminated 30 hrs/week of manual operator intervention (72% reduction), and resolved 25+ P0–P3 escalations while cutting mean time to resolution by 35% through structured fault isolation and automated retry policies.
Why It Mattered
Behind every escalation was an operator doing the same manual triage week after week, and giving those hours back was the real result. The retry storm was a lesson in reading a production failure as a design signal: isolate failure domains and defend against backpressure, or one bad upstream becomes everyone's outage.
“Designed a Python/Celery/RabbitMQ event-processing pipeline sustaining 1,000 events/sec that eliminated 30 hrs/week of manual operator intervention (72% reduction), and cut mean time to resolution by 35% across 25+ P0–P3 escalations.”
Core Stack
Metrics
throughput
1,000 events/sec
manual_work_cut
30 hrs/week
overhead_drop
72%
mttr_gain
35%
escalations
25+