WakeOps
Incident response platform that turns Grafana alerts into durable call, email and Slack escalation workflows.
- Next.js
- Temporal
- PostgreSQL
- Twilio
- RabbitMQ
The problem
An alert is not an incident response. Grafana can say that a service is failing, but it does not know which engineer owns that service, whether they answered the call, when to try again, or when to escalate. Handling those steps by hand works until the alert arrives at the wrong hour and every missed minute matters.
I wanted WakeOps to own that gap: accept one alert, map it to the right people and infrastructure, keep the response running outside the original HTTP request, and leave behind one clear incident record instead of a trail spread across provider dashboards.
What I built
WakeOps gives each organization a secured Grafana webhook and a setup flow for hosts, services, environments, and on-call and senior engineers. A firing alert is verified, normalized, mapped to the saved deployment, fingerprinted to stop duplicate active incidents, and persisted before the API returns. The dashboard then shows the incident, its status, call attempts, and audit timeline as they change.
Each new incident starts a Temporal workflow. The worker calls the on-call engineer through Twilio, waits for the result, retries unanswered calls, and escalates to the senior engineer when both on-call attempts fail. Email and Slack jobs go through RabbitMQ, so a slow notification provider cannot hold up the call path, and Twilio callbacks update PostgreSQL and signal the running workflow when somebody acknowledges.
Architecture
The Next.js app handles Google sign-in, organization setup, integrations, and incident views. The Express API handles short-lived work: Grafana webhooks, Twilio voice and status callbacks, health checks, and signals sent to Temporal. PostgreSQL is the application source of truth for mappings, incidents, attempts, and audit events; Temporal owns timers, retries, and long-running workflow state.
The worker is deliberately separate from the API. It polls Temporal, runs call activities, publishes RabbitMQ jobs, and consumes the email and Slack queues without needing a public URL. The web app runs on Vercel, the API on Render, and managed Supabase, Temporal Cloud, and CloudAMQP provide the shared state and messaging that let the pieces communicate.
Bugs I actually debugged
Bug 01
A healthy deployment that could not open one more database connection
Google sign-in started looping back to the login page while the API health check stayed green. The Vercel logs finally showed the real failure: Supabase's session pool had reached its fifteen-client limit. Every serverless instance could create a Prisma client, and reusing one client inside a warm process could not stop multiple processes from holding sessions.
The web deployment moved to Supabase's transaction pooler with a one-connection Prisma limit, while the Prisma client is cached globally inside each warm runtime. That matches the connection model to serverless execution: short requests borrow a pooled connection instead of every function instance trying to own a database session.
Bug 02
New worker source running an old workflow
A Temporal activity failed because attemptNumber was missing, even though the TypeScript source clearly passed it. The worker imported the workflows package through its compiled dist output, and that output was older than the source. Restarting the worker kept loading the stale bundle, so the failure looked like a data problem instead of a build problem.
The worker's development command now builds the workflows package before it starts. I also treated already-running workflows separately: Temporal preserves their event history, so changing source code cannot rewrite arguments already recorded for an old execution. Stale executions are terminated and new incidents start against the fresh bundle.
Bug 03
One free process trying to be both API and worker
Running the Express API and Temporal worker in one Render service looked convenient, but the service became inconsistent under a small memory and CPU budget. The worker bundles workflow code, holds long-lived Temporal and RabbitMQ connections, and runs consumers, while the API needs predictable resources for webhooks, callbacks, and health checks. A failure in either process also brought down the other.
I separated the API from the worker again. Render now serves only HTTP traffic, while the worker runs independently and connects outbound to Temporal Cloud, Supabase, CloudAMQP, and Twilio. There is no worker URL because nothing calls into it; it polls for work and can reconnect without changing any webhook configuration.
Bug 04
A live counter that froze at 09
The overview counters animated correctly on page load but failed during polling. When call attempts moved from 9 to 10, the UI displayed 09 and stayed there; other increases and decreases appeared unchanged until a manual reload. The polling data was fresh. The animation effect updated the same state listed in its dependency array, so its cleanup ran immediately and cancelled the frame that would have completed the transition.
I replaced the per-digit reset with a stable previous-value-to-next-value transition driven only by the server value. The old and new numbers are separate layers, increases and decreases move in opposite directions, and an internal state update can no longer cancel its own animation. That also removed the temporary leading zero at digit boundaries.
Results
- A mapped Grafana alert becomes one deduplicated incident with an assigned on-call engineer, durable retries, senior escalation, and a complete call and audit history.
- The webhook returns after validation and persistence; calls, timers, retries, and notifications continue independently through Temporal and RabbitMQ.
- Email, Slack, and voice are separate delivery paths, so one provider failing or slowing down does not block the others.
- The web app and API run behind permanent custom domains, while the worker connects outbound and can move to any always-on host without changing public callbacks.
Stack
- Frontend
- Next.js · TypeScript · Tailwind CSS · Auth.js
- Backend & Data
- Node.js · Express · PostgreSQL · Prisma
- Workflow & Messaging
- Temporal · RabbitMQ
- Integrations
- Grafana · Twilio · Slack · Resend
- Infrastructure
- Vercel · Render · Supabase · CloudAMQP