On May 15, 2026, InfoQ published Discord Reveals How a Hidden Circular Dependency Triggered Its March Voice Outage, reporting on a detailed postmortem by Discord regarding the "undetected circular dependency" that caused a complete shutdown of its voice services on March 25, 2026.
The core issue was simple: "Service A depended on B, B depended on C, and C was waiting for A to finish starting up." Dependencies that resolved smoothly in sequence under normal conditions created concurrency conflicts during simultaneous startup within a specific incident recovery sequence, driving the entire voice infrastructure into a cascading failure. Outages of this kind, which even Discord with its 600 million monthly active users could not prevent, routinely lurk in mid-sized enterprise production systems that have run for 3 to 10 years. This article outlines the design of an "architecture dependency audit" service delivered for clients.
Why hidden circular dependencies pose the greatest outage risk for mid-sized enterprises
| Structure | Why it goes unnoticed |
|---|---|
| Midway through microservice migration | Service boundaries shift frequently, leaving dependency maps outdated |
| Bidirectional usage of shared libraries | Paths like A → Shared → B and B → Shared → A are overlooked |
| Deferred resolution in DI containers | Startup appears sequential initially, but collapses upon restart |
| Vendor-provided packages | Internal dependencies are black boxes that cannot be scanned |
| Siloed architectural knowledge | Dependency maps are lost when original designers leave |
Uncovering architectures with "benign during normal operations, lethal during incident recovery" characteristics requires specialized architecture dependency audits. While we covered graph management of model dependencies in Netflix Model Lifecycle Graph for Custom MLOps Governance, this article extends that approach to enterprise systems overall.
Three lessons extracted from the Discord postmortem
Lesson 1: Dependencies that cannot be verified during normal operations are the biggest landmines
Discord's outage surfaced for the first time during an incident recovery sequence. Unless you intentionally reproduce recovery sequences using chaos testing, dependency issues remain invisible.
Lesson 2: Service boundary diagrams always drift from actual traffic
Discord had internal dependency diagrams. However, because "diagrams unupdated for over six months" diverged from reality, the circular dependency was never detected. A mechanism to "generate dependency graphs from live traffic" is indispensable.
Lesson 3: Impossible to detect with unit testing
Circular dependencies cannot be detected by unit tests, integration tests, or E2E tests. It is necessary to continuously scan runtime dependencies at the service mesh layer or via eBPF.
The four phases of building an architecture dependency audit for clients
Phase 1: Baseline assessment (3–4 weeks)
Collect the client's architecture diagrams, repository structures, and service mesh configurations to establish an "official dependency graph." In parallel, collect 2 weeks of actual traffic logs using tools like OpenTelemetry, Cilium Hubble, or Pixie.
Phase 2: Automated dependency graph generation and diff analysis (2–3 weeks)
Generate a dynamic dependency graph from live traffic logs and report the diff against the official diagram. In our experience, 30% to 60% of active dependencies are missing from official diagrams in the vast majority of cases.
Phase 3: Circular dependency detection + risk prioritization (2–3 weeks)
Extract circular paths using NetworkX, Graphviz, Backstage Tech Insights, or similar tools, prioritizing risk based on "blast radius during outages × recovery difficulty."
Phase 4: Chaos testing + remediation roadmap (4–6 weeks)
Reproduce incident recovery sequences using chaos testing for the top 5 to 10 priorities, formulating a roadmap of remediation proposals to break cycles. Propose locking startup ordering for the short term, and making dependency flows strictly unidirectional over the medium to long term.
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| Dynamic dependency collection | Cilium Hubble + eBPF | OpenTelemetry + Tempo |
| Static dependency analysis | Backstage Tech Insights | dependency-cruiser / Madge |
| Dependency visualization | Backstage + Graphviz | Structurizr / C4 model |
| Chaos testing | Chaos Mesh / Gremlin | Litmus Chaos |
| Observability | Grafana + Prometheus | Datadog |
| Continuous scanning | GitHub Actions + weekly Routine | Argo Workflows |
| Report generation | Notion + Looker Studio | Confluence |
Combining this particularly with Continuous Profiling for Clients With Grafana Pyroscope 2.0 establishes a robust foundation to monitor both "dependencies and performance" continuously.
Which organizations this fits and which it does not
| Suitable organizations | Unsuitable organizations |
|---|---|
| Operating microservices for 3+ years | Operating standalone monoliths |
| Experienced unexplained outages in the past year | Virtually zero history of outages |
| Service boundaries change frequently | Architecture remains static |
| SRE teams in early setup phase | Mature SRE culture firmly established |
| Organizations advancing in-house engineering | Operating exclusively on third-party vendor packages |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Audit scope | Target services and environments (production / staging) | Liability for out-of-scope systems |
| Data collection scope | Agreements on trace and log collection | Handling personal data |
| Chaos testing timeframes | Production vs. staging, blast radius | User impact tolerances |
| Remediation liability boundaries | Limited to proposals, or including implementation | Allocation of implementation responsibility |
| Report distribution scope | Executive leadership / engineering teams / audit firms | Scope of non-disclosure obligations |
| Contract renewal terms | Annual / semi-annual audit renewals | Fee adjustments upon renewal |
Client-side ROI estimation (model for mid-sized SaaS with 200 employees)
| Item | Before audit | After audit | Difference |
|---|---|---|---|
| Annual incident count | 12 incidents | 4 incidents | -8 incidents |
| Recovery time per incident | Avg. 4.5 hours | Avg. 1.2 hours | -3.3 hours |
| Annual downtime from incidents | 54 hours | 4.8 hours | -49 hours |
| Opportunity loss per hour | 1,000,000 yen | 1,000,000 yen | — |
| Annual opportunity loss | 54 million yen | 4.8 million yen | -49.2 million yen |
Standard: Even at this investment level, the ROI exceeds 6x. Even for large-scale engagements, it delivers returns recoverable within the first year.
Five common pitfalls
Pitfall 1: Auditing solely from official diagrams
Audits relying solely on official diagrams are proven to miss 30% to 60% of active dependencies. Dynamic analysis using live traffic logs is indispensable.
Pitfall 2: Running chaos testing directly in production
Running chaos tests in production without staging validation carries an exceptionally high probability of causing genuine outages. The sequence of two staging cycles followed by production is non-negotiable.
Pitfall 3: Settling for "fixed restart ordering" to resolve dependencies
While effective as a short-term workaround, having humans memorize restart sequences leaves knowledge siloed. A roadmap to make dependency flows strictly unidirectional over the medium to long term is essential.
Pitfall 4: Leaving third-party vendor packages as "black boxes"
Internal dependencies of commercial packages can be surfaced by adding trace instrumentation and requesting disclosures from vendors. Be sure to include contract negotiations within the audit scope.
Pitfall 5: Stopping at deliverable reports
Simply handing a report to executives will not move engineering teams. The golden rule is to mandate "integration into the next fiscal year's roadmap" within the contract.
90-day action plan
| Week | Action |
|---|---|
| Week 1〜4 | Baseline assessment + dynamic dependency collection |
| Week 5〜7 | Dynamic dependency graph generation + diff analysis |
| Week 8〜10 | Circular dependency detection + risk prioritization |
| Week 11〜13 | Chaos testing + presenting remediation roadmap |
Summary — Preempting through client services what 600-million-user Discord could not prevent
Discord's March voice outage demonstrated to the world that "dependencies undetectable during normal operations" represent the single greatest outage risk. Auditing this entirely in-house is impractical for mid-sized enterprises, making specialized audit services from external partners the most cost-effective countermeasure.
If you are dealing with challenges like "an increase in unexplained outages" or "tangled service boundaries that obscure the current state," feel free to contact us via our contact form.
Sources
- Discord Reveals How a Hidden Circular Dependency Triggered Its March Voice Outage（InfoQ）
- Netflix Model Lifecycle Graph for Custom MLOps Governance (GH Media)
- Continuous Profiling for Clients With Grafana Pyroscope 2.0 (GH Media)
- Kubernetes Autonomous AI Agents for Custom Projects (GH Media)
- Slack ChatOps × AI Infra Agent SRE Services for Clients (GH Media)
