"A record entered into the core ERP system was not reflected in the inventory database"—in custom projects integrating multiple systems, reports of occasional dropped records frequently surface well after operations have begun. A common implementation involves reading an entire database via overnight batch jobs and copying it over to the other side. While this works in most cases, discrepancies quietly accumulate as updates occurring mid-copy slip through or records deleted since the previous sync are mishandled. When running a full copy fails to resolve discrepancies, the root cause lies in trying to infer changes after the fact by comparing the whole picture. The system fails to precisely capture what changed and when.
The standard method for capturing and delivering changes without omission is Change Data Capture (CDC). Write-Ahead Intent Log: A Foundation for Efficient CDC at Scale (InfoQ), introduced by InfoQ in June 2026, explores designs that make CDC reliable without dropped updates, even at scale. Scale aside, the mindset of "tracking changes themselves rather than running full copies" is particularly potent for the understated system integrations common in custom development. This article outlines how to conceptualize CDC in custom development and evaluate its adoption.
Why full-copy approaches drop data
Integrations that read and copy all records via overnight batch jobs are easy to build and run smoothly at first. Dropped records occur because the following inherent weaknesses of this approach surface as operations mature.
First, updates occurring during a copy cannot be captured. If another process updates a record while tens of thousands of rows are being read, whether the read data is old or new depends entirely on timing. The outside world does not freeze while batch queries run.
Second, deletions are not communicated. When source data is deleted, full-copy jobs struggle to inform destination databases that a record is gone. Stale remnants linger in target systems, causing both sides to drift out of sync.
Third, latency and system loads are substantial. With once-a-day batches, changes that occur in the interim remain unreflected for up to a full 24 hours. Conversely, increasing frequency places heavy loads on production databases, as reading entire datasets becomes progressively heavier as data volumes grow.
Fourth, the root causes of discrepancies cannot be traced. Because there is no record of "what changed and when," detecting a discrepancy does not reveal which specific update was missed. Preventing recurrence inevitably devolves into re-running full copies every time.
Losing sight of how data flows across multiple systems shares the same root cause as losing track of service connections. The approach to mapping what connects where was covered in our article on visualizing service dependencies; CDC is a mechanism that turns that "data flow" into a dependable, one-way pipeline.
CDC basics: moving from "comparing results" to "tracking changes"
The CDC concept does not rely on comparing results after the fact between source and destination; instead, it captures and delivers the actual changes occurring in the source in sequential order. The linchpin is the internal change log maintained by most modern databases.
Before writing data, databases record an intention log ("this is what is about to change") prior to executing the actual update—a mechanism known as a Write-Ahead Log (WAL). Although originally built for crash recovery, reading this log reveals precisely what row changed, how it changed, and when, in the exact sequence events occurred. CDC reads these change records sequentially and dispatches them to external systems.
全件コピー方式:
毎晩、テーブル全体を読む → コピー先と突き合わせる
（コピー中の更新・削除を取りこぼしやすい／重い）
CDC方式:
DBの変更記録を、起きた順に読む
「Aを追加」「Bを更新」「Cを削除」を一件ずつ届ける
（変更そのものを追うので、取りこぼしと削除漏れが起きにくい）Making this switch reverses the previous weaknesses.
| Dimension | Full copy | CDC |
|---|---|---|
| Updates during copy | Prone to drops | Hard to miss because changes are captured sequentially |
| Communicating deletions | Hard to convey | Deletions arrive as discrete change events |
| Propagation speed | Delayed by batch intervals | Near real-time per change |
| Production load | Heavy; reads all data each run | Light; handles only change deltas |
| Tracking discrepancies | Difficult to identify root causes | Can trace change history |
The property of "delivering changes reliably in order" pairs naturally with resilient execution models that resume from where they left off if interrupted mid-process. When combined with the principles covered in our article on durable execution workflows, destination processing can be architected to ensure integrations remain resilient without dropping data, even during failures.
Pitfalls when adopting it in client web development
In a custom wholesale system whose integration refactoring we took over (company name withheld), core order data was copied to inventory management via overnight batches, resulting in several orders failing to reflect in inventory each month. The operational team had normalized manual daily reconciliation, "visually cross-checking every morning because numbers didn't match." The causes were two-fold: missing orders registered or modified while batches ran, and cancellations (deletions) failing to propagate to inventory.
We phased out full-copy jobs and transitioned incrementally to a CDC architecture that reads core database change logs and streams them to inventory. Rather than converting all tables at once, our strategy was to apply CDC first only to the orders table, where drops caused tangible operational harm, expanding the scope only after confirming results. New orders, modifications, and cancellations began arriving in inventory in exact chronological order, rendering morning manual reconciliations obsolete. All we did was replace the retrospective comparison model with sequential change tracking.
The most valuable lesson from this refactor was designing the receiving side to remain resilient if the same change arrives twice. Because CDC can trigger replays and resumptions, duplicate events can occur. If the receiver naively processes an event by executing "add order," duplicate entries will be created. Giving each change a unique identifier and making consumption idempotent ("do nothing if already processed") is imperative. Neglecting this replaces dropped records with duplicate records—swapping one bug for another.
Another pitfall is preparing for changes in data structures (schemas). When table definitions change on the source side, the structure of CDC events changes too, preventing the receiving side from parsing incoming data. As integrations proliferate, schema mismatches become breeding grounds for incidents. The philosophy for governing and managing transmitted schemas was detailed in our article on curbing schema sprawl; when implementing CDC, pairing it with schema governance from day one is the safest approach.
Where to begin
If your cross-system data integrations suffer from occasional missing records, unpropagated deletions, or normalized manual morning reconciliations, switching from full copies to change tracking is worth evaluating. However, you do not need to migrate every table to CDC all at once.
As a first step, pick a single table where dropped updates cause the most direct damage (such as orders, inventory, or billing) and integrate only that table via CDC. Then, ensure the receiving side is idempotent—so receiving the same change twice breaks nothing—and agree on schema definitions. This alone stops data loss in your most troublesome integration, allowing you to validate results before expanding scope incrementally.
If data occasionally falls out of sync across systems, deletions or cancellations fail to reach downstream services, or manual reconciliation has become routine, please reach out via GleamHub's contact page. We will evaluate your current integration model and partner with you to design a phased migration toward drop-free, CDC-based pipelines without taking systems offline.
