Unexplained data corruption occurs intermittently over several months. Reproduction steps cannot be established. Code reviews reveal nothing. Logs show no anomalies. Under these circumstances, continuing to suspect one's own code is a natural response.
The cause that Tailscale reached after six months of investigation was not there. It was a race condition bug inside SQLite itself that had lingered for 16 years since 2010. The SQLite development team dubbed this the "WAL-Reset bug."
Consuming this as an anecdote about a rare bug yields zero practical value. Rather, what should be asked is where your business systems assume "this component won't break," and whether they are architected to notice when that assumption fails.
What was happening
In SQLite's WAL (Write-Ahead Logging) mode, writes are initially appended to a WAL file and then applied collectively to the main database file through a process called checkpointing.
At this time, if a write transaction and a checkpoint happened to coincide at a specific moment, SQLite could mistakenly consider certain WAL pages as "already copied to the main database." Misidentified pages were discarded without being written to the main file, and that data was permanently lost.
What makes this insidious is that other pages referencing the lost pages (such as indexes) are written normally. As a result, indexes pointing to non-existent locations remain, leaving the database file corrupted. Writes appear successful, and the corruption is discovered only much later.
The conditions to trigger it are restrictive: running in WAL mode, having two or more connections to the same file opened across separate threads or processes, and having both perform writes and checkpoints at the exact same moment. It occurs only when all conditions align.
To illustrate how rare it was, the SQLite development team had to add code to their test environment that intentionally forced the condition just to reproduce the defect. That is to say, normal testing could never hit it.
Scope of impact and what to check now
Targeted versions span from SQLite 3.7.0 (2010-07-21) to 3.51.2 (2026-01-09). Fixes are included in 3.51.3 (2026-03-13) and later.
Before concluding "we don't use SQLite," a deeper check is necessary. SQLite is rarely chosen and installed as a standalone system; rather, it is a component that enters bundled inside other software.
| How it enters | Setting Location |
|---|---|
| Bundled in language runtimes | Python's sqlite3.sqlite_version, or versions bundled with Go and Node.js drivers |
| Embedded in applications | Dependency libraries for desktop apps, mobile apps, and CLI tools |
| Used internally by middleware | Cases where it is used internally to store administrative metadata |
The verification method itself is straightforward: run select sqlite_version(); via SQL. Because OS package versions and app-bundled versions sometimes conflict, the key point is to inspect the version that the application actually links against.
Tailscale was prone to hitting this bug because they controlled checkpointing themselves and executed it extremely aggressively. Conversely, in standard usage where checkpointing is left to SQLite, the probability of encountering it drops even further. However, "low probability" does not mean "impossible."
Why taking six months matters more
In this incident, what impacts practical operations more than the bug's details is the fact that it took six months to reach the cause.
The investigation dragged on because, in order of suspicion, proprietary in-house code comes first. That is the correct order, and nine times out of ten, the issue is found there. The problem is that when it is not found there, there is no predetermined rule for "what to suspect next."
At many organizations, the next move is not prepared. Consequently, teams investigate the same code repeatedly. What should be prepared is an inventory of which dependent components are treated under the assumption that they "won't break": databases, file systems, runtimes, and cloud persistence layers. These are typically excluded from testing. Because they are assumed to be reliable, they are left untested, and they will not surface unless consciously listed out.
The premise that anomalies cannot be detected without first defining the normal state mirrors monitoring design. This concept is explored in Monitoring starts with defining "normal".
Three checks clients can verify
Even from the position of commissioning business systems externally, there are things you can verify without delving into technical details.
1. Is data integrity inspected regularly? In SQLite, this is PRAGMA integrity_check, and other databases have equivalent mechanisms. This is the bare minimum to avoid remaining unaware of corruption for months. Saying "we take backups" does not eliminate the possibility that corrupted data is being continuously backed up in its corrupted state.
2. Have you actually tested restoring from backups? Check restoration drill records rather than backup success logs. Having actually restored data even once a year prevents finding out that data had become unrecoverable without anyone noticing.
3. Under whose contract, and when, are dependency library versions upgraded? Even when upstream fixes are released as in this case, software continues running on outdated versions unless an update owner is defined. Cases where this remains ambiguous in maintenance contracts are not uncommon.
The third point becomes an issue in the exact same manner when vulnerabilities are disclosed. Risks of systems operating unmaintained are compiled in Vulnerabilities carried by unmaintained websites. If you are at a stage of reconsidering database selection itself, Choosing between SQLite and Postgres provides decision criteria.
What to do next
Try running select sqlite_version(); on your systems. If it is prior to 3.51.3, it is worth confirming with your development partner whether your setup matches the affected conditions. Even if it does not, the fact that you verified it is meaningful in itself. That is because it exposes whether there are other components whose running versions cannot be answered immediately.
Then, count how many of the three points above you can answer with "we have actually tested it." If the answer is zero, there are actions to take before worrying about rare bugs.
At GleamHub, our custom development, AI, and automation consultations cover inventorying existing system dependencies and verifying the effectiveness of recovery procedures. Because scopes to verify vary by architecture, please consult us individually. Reach out through Contact Us.
Sources
- How Tailscale helped find the SQLite WAL-Reset bug — Tailscale Blog
- Deeply buried 16-year-old SQLite bug caused last year’s Tailscale outages — The Register
- Breaking the WAL — Antithesis
- Write-Ahead Logging — SQLite
- PRAGMA integrity_check — SQLite
- Old SQLite bug caused months of outages at Tailscale — Techzine
