On May 22, 2026, InfoQ published Discord Rebuilds Database Operations Around Automation to Manage ScyllaDB at Massive Scale. Discord built an in-house orchestration platform called "Scylla Control Plane" (SCP), allowing a small infrastructure team to automate large-scale ScyllaDB cluster operations (adding nodes, rebalancing, backups, recovery, and version upgrades). They realized a world where operational tasks that previously required days of manual effort are completed with a single command or automated orchestration. This serves as a concrete example of transitioning from the classic model where "dedicated DBAs protect massive databases through manual work" to a new standard of "declarative management via a control plane."
From our perspective supporting mid-market enterprise DB infrastructure through custom development, this provides architectural guidance for tackling typical pain points—such as "DBA shortages," "siloed DB operations reliant on specific individuals," and "burnout from night and weekend support"—using control plane patterns like Discord's. In contrast to the DB auditing and investigative scope covered in our Bintrail MySQL Time Travel Binlog DB Forensics Client Services and the analytics data governance covered in our Monzo dbt Governance Data Mesh Client Services, this article outlines our architectural development services for "replacing production DB operations themselves with an automated platform."
Why "adopting a control plane is the turning point"
| Dimension | Conventional DBA operations (manual) | Discord-style control plane |
|---|---|---|
| Node addition | Several hours to several days (runbook-based) | A few minutes (declarative orchestration) |
| Cluster expansion | Dedicated DBA required | Safely executable by any engineer |
| Backup / recovery | Combination of scripts and manual steps | Pipeline automation |
| Version upgrades | Maintenance window + entire team on standby | Canary deployment + automated rollback |
| Incident response | DBA on-call standby | Runbook → automated remediation attempts |
| Individual dependency | Operations stall if the core DBA leaves | Transferable via specifications and code |
| Small team limitations | Breaks down as scale expands | Operate at multiples of scale without growing team size |
In short, adopting a control plane is a new model that replaces "operational ceilings determined linearly by DBA headcount and experience" with "exponential scaling powered by software and declarative design."
Three structural changes beneficial to custom development projects
Pillar 1: From "relying on DBAs" to "software-driven DB operations"
Many mid-market enterprises maintain their clusters with just 1 or 2 dedicated DBAs, leaving business continuity in immediate jeopardy if those individuals depart. Adopting a control plane "converts operational know-how into code and declarations," structurally increasing organizational resilience against personnel turnover, resignations, and medical leaves. This represents a step toward applying the specification-driven approach for analytics covered in our Monzo dbt Governance Data Mesh Client Services to production DB operations as well.
Pillar 2: From "night and weekend on-call response" to "automated remediation attempts + alerts"
Tasks where on-call DBAs are rousted in the middle of the night shorten the tenure of top-tier talent. A control plane designed around "attempt initial automated remediation → escalate to a human only upon failure" can reduce on-call incidents by 70% to 90%. Combined with the runtime monitoring covered in Custom eBPF Kernel-Level Security Monitoring, this enables the construction of a database equivalent to "observe → auto-diagnose → auto-remediate."
Pillar 3: From "blindly offloading to managed services" to "hybrid optimization"
The misconception that "using managed DBs (RDS / Cloud SQL / Atlas) eliminates operational overhead" inevitably hits a wall regarding cost, performance, and customization. Discord achieved operational efficiency surpassing managed services by combining self-hosted ScyllaDB with an in-house control plane. Through custom development, we can design and propose the optimal operational balance across "managed, self-hosted, and hybrid" infrastructure tailored to each client's workload characteristics.
The 5 phases of our "DB Ops automation platform" custom development offering
Phase 1: Current state assessment (3 weeks)
- Inventorying existing DB clusters (types, versions, scale, workloads)
- Inventorying operational tasks (expansion, recovery, backups, verification, migrations)
- Measuring baseline operational hours, on-call incidents, and system failures
- Assessing skill and structural gaps across DBA and SRE teams
- Prioritizing candidate tasks for control plane automation
Phase 2: Control plane design (2–3 weeks)
- Selecting orchestration foundations (in-house vs. Crossplane / Argo / commercial tools)
- Declarative configuration modeling (IaC + DB configuration)
- Designing safety mechanisms (dry runs, approvals, phased rollouts)
- Audit logging, monitoring, and alerting strategies
- Phased rollout planning (task by task)
Phase 3: PoC build (3–4 weeks)
- Automating an initial set of 3–5 representative tasks
- Behavioral validation in canary environments
- Testing automated remediation attempts and fallback procedures
- Audit log integration and dashboards
- Interviews and iterative feedback with operations teams
Phase 4: Production deployment (4–8 weeks)
- Phased task automation (backups → expansion → recovery)
- Gradual handover to 24/7 operational rotations
- Runbook and knowledge transfer
- Governance design (approvals, auditing, SLAs)
- Training for developers and SREs
Phase 5: Monthly operational reviews (ongoing)
- Monitoring automation coverage and remediation success rates
- Evaluating additional candidate tasks for automation
- Tracking version upgrades for ScyllaDB, PostgreSQL, and MySQL
- Incorporating post-mortem action items into automation
- Cost optimization (node count and instance types)
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| DB engine | ScyllaDB / PostgreSQL / MySQL / Cassandra | DynamoDB / Spanner |
| Orchestration | Crossplane / Argo Workflows / Kubernetes Operator | In-house Go controller |
| IaC | Terraform / OpenTofu / Pulumi | CloudFormation |
| Deployment / failover | ArgoCD / FluxCD | Spinnaker |
| Monitoring | Prometheus + Grafana + Alertmanager | Datadog / New Relic |
| Tracing / logging | OpenTelemetry + Loki + Tempo | Honeycomb / Splunk |
| Backups | Velero / custom snapshot pipelines | Veeam |
| Secrets | Vault / AWS Secrets Manager | 1Password Connect |
| Runbooks | Notion / Runbook as Code (Rundeck) | Confluence |
Which projects need this and which do not
| Projects requiring this | Projects not requiring this |
|---|---|
| DB clusters with 5+ nodes | Single-instance architecture |
| Operating multiple clusters with 1–3 DBAs | No dedicated DB staff; fully managed |
| Night and weekend on-call duties have become routine | Business hours only |
| Hitting cost and customization limits of managed DBs | Fully satisfied with managed services |
| Frequent cluster scaling and migrations | Fixed configuration unchanged for years |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Target clusters | Production / staging / verification | Business impact level |
| Automation scope | Backups / scaling / recovery / migrations | Phased rollout schedule |
| On-call SLA | Detection → automated remediation → human escalation | Business continuity requirements |
| Approval flows | Automated / manual / hybrid | Risk tolerance |
| Audit log retention | Retention period + encryption + storage | Audit compliance requirements |
| Handover Upon Project Completion | Control plane codebase + runbooks + knowledge base | Internal operational continuity |
Client ROI projection (assuming 15 clusters / 2 DBAs / frequent nighttime on-call incidents)
| Item | Existing (manual DBA operations) | After control plane implementation | Difference |
|---|---|---|---|
| Operational hours (annual) | 3,200h | 1,000h | -2,200h |
| Nighttime on-call incidents (annual) | 240 incidents | 40 incidents | -200 incidents |
| Mean time to recovery (MTTR) | 4 hours | 30 min | -3.5h |
| Impact of DBA departure risk | 1–2 weeks of service disruption | A few days | -10 business days |
| Cluster expansion lead time | 5 business days | 4 hours | -36 hours / event |
| Annual benefit | — | — | Equivalent to approx. ¥36 million + improved business continuity |
Calculated at an hourly rate of ¥8,000, we estimate an annual labor savings of over ¥18 million, alongside reductions in on-call allowances and turnover risk. When making investment decisions, the standard approach is to compare these savings against the build and maintenance costs of the control plane to project the payback period.
Five common pitfalls
Pitfall 1: Assuming managed services solve everything
Assuming that "using RDS or Cloud SQL eliminates the need for operations" leaves you vulnerable when cost and operational constraints fail to keep pace with business growth. Evaluate a hybrid architecture combining managed services + a control plane from early on.
Pitfall 2: Starting automation with destructive tasks
Beginning your automation with destructive tasks like node deletion or schema migrations risks triggering production incidents during the PoC phase. Progress incrementally, starting with read-only, auditing, and backup workflows.
Pitfall 3: Treating audit logging as an afterthought
Assuming that "audit logging for control plane operations can be added later" will lead to major roadblocks during compliance audits. Ensure all operations are logged for auditing right from the initial build.
Pitfall 4: Treating DBAs as adversaries of automation
Approaching the project under the premise that "automation makes DBAs obsolete" will cause the rollout to collapse due to internal pushback. Redefine the DBA's role from "manual operator" to "designer and owner of the control plane's evolution."
Pitfall 5: Full automation without fallback paths
Leaving the boundaries vague regarding when automated remediation should "escalate to human operators upon failure" causes tasks to stall in limbo between automation and manual intervention. Clearly demarcate the sequence: automation → warning → human approval → manual execution.
90-day action plan
| Week | Action |
|---|---|
| Week 1〜3 | DB cluster + operational task inventory |
| Week 4〜6 | Control plane design + orchestration platform selection |
| Week 7〜10 | PoC automation for 3–5 tasks |
| Week 11 | End-to-end automated remediation testing in canary environments |
| Week 12 | Phased rollout to the first production cluster |
| Week 13 | Monthly operational review + runbook refinement |
Summary — From "DBA headcount × experience" to "control planes × declarative design"
Discord's Scylla Control Plane demonstrates that the classical model of "dedicated DBAs manually protecting large-scale databases" has come to an end. From our perspective supporting mid-market enterprise DB infrastructure through custom development, "DB Ops Automation Platforms"—combining control plane architecture + phased automation + unified auditing + monthly operations—represent a key service offering.
Building a DB Ops automation platform varies significantly in design and required effort depending on cluster scale, DB engines, and the scope of automated tasks. For challenges such as "operations halting if a DBA leaves," "burnout from night on-call duties," or "hitting cost limits with managed DBs," we provide custom quotes after assessing your current infrastructure. Please feel free to reach out via our contact form.
Sources
- Discord Rebuilds Database Operations Around Automation to Manage ScyllaDB at Massive Scale（InfoQ）
- Bintrail MySQL Time Travel Binlog DB Forensics Client Services (GH Media)
- Monzo dbt Governance Data Mesh Development for Clients (GH Media)
- eBPF Kernel-Level Security Monitoring Client Services (GH Media)
