Mike's Notes
Here is an InfoQ article by Apoorv Mittal & Rafal Gancarz, referenced in Data Engineering Weekly. It covers a valuable way to migrate data while keeping critical production going. It is something to use in the future.
Resources
References
Repository
- Home > Ajabbi Research > Library > Subscriptions >Data Engineering Weekly
Last Updated
14/04/2025
Shadow Table Strategy for Seamless Service Extractions and Data Migrations
By: Apoorv Mittal & Rafal Gancarz
InfoQ: 09/04/2025
Apoorv Mittal is a passionate Software Engineer at Block (CashApp) based in Seattle, WA. With over a decade of experience in distributed systems and fintech, he has led transformative projects at Block, Dropbox, and AWS. His expertise spans modernizing legacy systems into scalable microservices, architecting resilient financial infrastructures, and pioneering cloud security innovations, notably through his patented AWS Traffic Mirroring solution. You can find Apoorv on LinkedIn.
Key Takeaways
- The shadow table strategy creates a synchronized duplicate of the data that keeps the production system fully operational during changes, enabling zero-downtime migrations.
- Database triggers or change data capture frameworks actively replicate every change from the original system to the shadow table, ensuring data integrity.
- The shadow table strategy supports diverse scenarios - including database migrations, microservices extractions, and incremental schema refactoring - that update live systems safely and progressively.
- Shadow tables deliver stronger consistency and simplify recovery compared to dual-writes or blue-green deployments.
- Industry case studies from GitHub, Shopify, and Uber demonstrate that the shadow table approach drives robust large-scale data migrations by actively maintaining continuous data integrity and offering rollback-friendly safeguards.
Introduction
Modern software systems often need to evolve without disrupting users. When you split a monolith into microservices or modify a database schema, you must migrate data with minimal downtime and risk. Shadow tables have emerged as a powerful strategy to achieve this. In a nutshell, the shadow table approach creates a duplicate of the data (a shadow version) and keeps it in sync with the original, allowing a smooth switchover once the new setup is ready.
This article explores how shadow tables help in different migration scenarios — database migrations, service extractions, and schema changes — while referencing real case studies and comparing this approach to alternatives like dual-writes, blue-green deployments, and event replay mechanisms.
What is the Shadow Table Strategy?
The shadow table strategy maintains a parallel copy of data in a new location (the "shadow" table or database) that mirrors the original system’s current state. The core idea is to feed data changes to the shadow in real time, so that by the end of the migration, the shadow data store is a complete, up-to-date clone of the original. At that point, you can seamlessly switch to the shadow copy as the primary source. In practice, implementing a shadow table migration typically follows a pattern:
- Create a Shadow Table: Prepare a new table (or database) with the desired schema or location. Although initially empty, you structure it to accommodate the migrated data.
- Backfill Initial Data: Copy existing records from the original data store into the shadow table, processing them in chunks to avoid overloading the system.
- Sync Ongoing Changes: As the system runs, apply every new write or update from the original data to the shadow. Use database triggers, change data capture (CDC) events, or application-level logic to propagate each INSERT, UPDATE, or DELETE from the source to the shadow to remain in sync.
- Verification: Optionally, run checks, such as comparing row counts or sample records, to confirm that the shadow’s data matches the source, giving you confidence that no data was missed.
- Cutover: Point the application to the shadow table (or perform a table rename/swapping in the database) once you verify it is up to date. The switch occurs with negligible downtime because you have kept the shadow current.
- Cleanup: Retire the old data store after cutover or keep it in read-only mode as a backup until you no longer need it. By using this approach, you can complete migrations with zero downtime. The production system continues running during the backfill and sync phases because reads and writes still hit the original data store while you build the shadow. When you are ready, you can quickly switch to the new store, often through a simple metadata update like a table rename or configuration change.
Figure 1: Data migration using the shadow table strategy
This strategy is sometimes also called the ghost table method (notably by GitHub’s schema migration tool gh-ost) because the new table is like a "ghost" of the original (gh-ost: GitHub's online schema migration tool for MySQL - The GitHub Blog).
Use Cases Where Shadow Tables Shine
Shadow tables offer a robust and flexible mechanism for managing complex migrations, service extractions, and schema refactorings while keeping production systems running uninterrupted. There are three common scenarios where shadow tables can be especially beneficial: database migrations with zero downtime, service extractions in a microservices transition, and incremental schema changes with data model refactoring.
Database Migrations with Zero Downtime
Modern applications often rely on large, heavily used production databases that cannot afford extended downtime for schema modifications or engine migrations. Direct alterations — like adding a new column, changing data types, or indexing — can cause long locking periods and stall critical operations. Shadow tables provide an alternative approach that minimizes the risk of disruption.
Begin by creating a new table that mirrors the structure of the production table while incorporating the desired schema changes. Although this shadow table starts empty or partially populated, you fill it using a controlled backfill procedure. A robust backfill procedure copies historical data from the production table into the shadow table in controlled batches, allowing the system to run concurrently.
After the backfill, set up a continuous synchronization mechanism by leveraging database triggers or CDC frameworks that propagate every new insertion, update, or deletion from the production table to the shadow table. This dual-write mechanism ensures that the shadow table remains an up-to-date replica of the production system.
Simultaneously, automated verification processes continuously compare key metrics between the two tables. Checksums, row counts, and deep object comparisons confirm data integrity and ensure that the shadow table accurately mirrors the production data. Only once these validations confirm that the shadow is consistent with the source can the final cutover be executed, often through a fast, atomic table rename or pointer switch. This approach enables the migration to be completed with minimal downtime, reducing risk and preserving user experience.
Service Extractions in a Microservices Transition
Transitioning from a monolithic architecture to a microservices-based system requires more than just rewriting code; you often must carefully migrate data associated with specific services. Extracting a service from a monolith risks inaccuracy if you do not transfer its dependent data accurately and consistently. Here, shadow tables play a crucial role in decoupling and migrating a subset of data without disrupting the existing system.
In a typical service extraction, the legacy system continues to handle all live operations while developers build a new microservice to handle a specific functionality. During extraction, engineers mirror the data relevant to the new service into a dedicated shadow database. Whether implemented through triggers or event-based replication, the dual-write mechanism ensures that the system simultaneously records every change made in the legacy system in the shadow database.
Once the new microservice processes data from the shadow database, engineers perform parallel validation to ensure that its outputs match expectations. A comparison framework automatically checks that the outputs of the new service match the expected results derived from the legacy system. This side-by-side validation allows engineers to identify discrepancies in real time and make adjustments as necessary.
Teams carefully manage the gradual transition of traffic from the legacy system to the new microservice. By initially routing only a small portion of user requests to the new service, teams can monitor performance, validate data consistency, and ensure that the new system behaves as expected.
Once the shadow database and the new microservice have proven to maintain the same level of data integrity and functionality as the legacy system, engineers execute a controlled, incremental cutover. Over time, they shift all operations to the new service and gradually reduce the legacy system’s role until they fully decommission it for that functionality. This phased approach mitigates risk and provides a built-in rollback mechanism if they detect any issues during the transition.
Incremental Schema Changes and Data Model Refactoring
Even for smaller-scale changes, such as refactoring a table or updating a data model, shadow tables offer a powerful way to mitigate risk. In many systems, evolving the data model is an ongoing challenge, whether splitting a single table into multiple logical parts, merging fields, or adding non-null constraints to previously optional columns.
Instead of applying changes directly to the live table, engineers create a shadow version to reflect the new design. The system simultaneously writes data to both the original and shadow tables, ensuring that it captures any update in real time across both structures. This dual-writing approach allows continuous validation of the new schema against the existing one, enabling engineers to compare outcomes and ensure that the refactored data model handles all business logic correctly.
Automated comparison tools play an essential role during this phase. By continuously monitoring and comparing data between the old and new schemas, the tools can detect discrepancies early — whether they arise from differences in data type conversions, rounding issues, or unforeseen edge cases. Once engineers have thoroughly validated the shadow table and adjusted for anomalies, they can seamlessly switch the application to the new schema. They can then gradually phase out the original table, with the shadow table taking over as the primary data store.
This incremental approach to schema changes minimizes the need for extended maintenance windows and reduces the risk of data loss or service interruptions. It provides a controlled path to evolve the data model while maintaining full operational continuity.
Industry Examples and Best Practices
Successful migrations using shadow tables have been reported by many organizations, forming a set of best practices:
- Online Schema Change Tools: Companies like GitHub and Facebook built tools (gh-ost and OSC) to perform online schema changes using shadow/ghost tables. These tools have become open-source solutions that others use. MySQL migrations now use the standard procedure of creating a shadow table, syncing changes, and then renaming (Zero downtime MySQL schema migrations for 400M row table). Similarly, used the open-source LHM gem in their Rails applications to safely add columns, as it "uses the shadow-table mechanism to ensure minimal downtime" Shopify (Safely Adding NOT NULL Columns to Your Database Tables - Shopify). The best practice here is to automate the shadow table process with rigorous checks (row counts, replication lag monitoring, etc.) and fallback paths if something goes wrong (for example, aborting the migration leaves the original table untouched, which is safer than a half-completed direct ALTER).
- Strangler Pattern for Microservices: Combining the strangler fig pattern with shadow reads/writes has proven to be a successful approach for migrating from a monolith. Amazon, Netflix, and others have used the idea of routing a portion of traffic to a new system in shadow mode to build confidence. Over time, they shifted reads and finally writes to the new service, effectively strangling out the old component. Best practice here: migrate in phases (e.g., shadow/dual-run, verify, then cutover) and use monitoring/metrics to ensure the accuracy of the new service’s data. The shadow phase can catch any discrepancies early, avoiding faulty migrations.
- Data Pipeline and CDC Usage: When using event streams for migration, you must ensure ordering and idempotency. Teams often choose Kafka or similar durable logs to replay events to the shadow database. The order of events must match the source’s commit order to maintain consistency. Industry best practice recommends schema versioning and backward-compatible change events when using this method, so that the new system can process events even if the schema evolves during the migration. Decoupling the pipeline (so that the old and new systems communicate via the event log rather than direct dual writes) also reduces risk to the production load. However, teams should monitor the lag between source and shadow and have a way to reconcile differences if the pipeline falls behind.
- Fallback and Rollback Plans: A migration is not truly safe without a rollback plan. In many cases, shadow table strategies lend themselves to easy rollback. If you find a problem during verification, simply discard the shadow table before switching over; this will not impact users. Even after a cutover, if the new system/table misbehaves, switch back to the old one (provided you kept it intact for a while). Uber’s migration post-mortem stresses having the ability to reverse traffic back to the old system if needed (Uber’s Billion Trips Migration Setup with Zero Downtime). As a best practice, keep the old system running in read-only mode for a short period after cutover, just in case you need to fall back. This safety net, combined with thorough monitoring, makes the migration resilient.
Comparing Shadow Tables to Alternative Migration Approaches
While shadow table (or shadow database) migrations are powerful, you should choose the right strategy for your situation.
Shadow Tables vs. Dual-Write Approach
Shadow table strategy often uses triggers or external pipelines to sync data, whereas a pure dual-write approach relies on the application to perform multiple writes. Dual-writing can achieve a similar goal of keeping two systems in sync, but the complexity of distributed transactions comes with it.
Without careful design, dual writes can lead to race conditions or partial failures – for example, the app writes to the new database but crashes before writing to the old one, leaving data out of sync. To mitigate this, developers use patterns like the Outbox Pattern, where the application writes changes to the primary DB and also to a special outbox table in the same transaction; the application then asynchronously publishes these changes to the second system.
In contrast, a trigger-based shadow table inherently ties the two writes into the source database’s transaction (the trigger runs inside the commit), and a CDC-based approach will capture the exact committed changes from the log. Such an approach often makes shadow table strategies more reliable for consistency than ad-hoc dual-write logic.
Figure 2: Data migration using dual-write approach
However, when you control both systems, dual writes may be simpler to implement at the application level, and they avoid the need for database-level fiddling or extra tooling. In summary, dual writes give you more control in application code, but you must exercise extreme care to avoid inconsistency. In contrast, shadow table methods leverage the database or pipeline to guarantee consistency.
Shadow Tables vs. Blue-Green Deployments
Shadow table strategy complements blue-green setups: one can see the shadow table as part of the green environment being prepared. The key difference is that blue-green by itself doesn’t specify how to keep the data in sync – it assumes you have a way to copy and refresh data in the green environment. A full outage could do this (not ideal), or a shadow/copy process could. So, in many cases, shadow table migrations are an enabling technique to achieve a blue-green style cutover for databases.
Figure 3: Blue-green deployments working with shadow tables
The ability to test the entire new stack in parallel is the advantage of a blue-green deployment. For example, you might run a new version of your service against the shadow database (green) while the old version runs against the old database (blue). You can then switch over when ready, and even switch back if something fails, since the blue environment is still intact. The downside is cost and complexity: temporarily doubling your infrastructure.
Maintaining two full environments (including databases) and keeping them in sync is not trivial. Shadow tables ease this by focusing on the data layer sync. If your migration is purely at the database layer (e.g., moving to a new database server or engine), a shadow table approach is a blue-green deployment of the database. If your migration also involves application changes, you might do a blue-green deployment of the app in tandem with the shadow table migration of the data.
Both strategies share the goal of a zero-downtime switch, and they pair well, but blue-green is a broader concept encompassing more than data. In contrast, the shadow table strategy is laser-focused on data consistency during the transition.
Shadow Tables vs. Event Replay (Rebuilding from Event Logs)
Event replay leverages an event log or sequence of change events to build up the state in a new system. It’s related to the CDC but slightly different in intent. In a replay scenario, you might start a brand new service by consuming a backlog of historical events (for example, reprocessing a Kafka topic of all transactions for the past year) to reconstruct its database state. Alternatively, if your system is event-sourced (storing an append-only log of changes), you can initialize a new read model or database by replaying all events from the start. This approach ensures that the new database’s state is equivalent to that of the old system, which is derived from the same sequence of inputs.
Figure 4: Data migration using event replay
Unlike shadow tables, event replay can be more time-consuming and is usually done offline or in a staging environment first because processing a considerable history of events can take a while. Shadow table migrations tend to operate on live data in real time, whereas you might use replay to bootstrap and then switch to a live sync method (like CDC) for the tail end. Another difference is that event replay might capture business-level events rather than low-level row changes.
For example, instead of copying rows from a SQL table, you might replay a stream of "OrderPlaced" and "OrderShipped" events to rebuild the state. This approach can be useful if you’re also transforming the data model in the new system (since the new system can interpret events differently). However, if you miss any events or the event log isn’t a perfect record, you risk an incomplete migration.
In practice, engineers often use event replay in combination with shadow strategies: one might do an initial event replay to catch up a new system, then use incremental CDC or dual-writes to capture any new events that occur during the replay (so the shadow doesn’t fall behind). The combination yields the same outcome: a fully synced shadow ready to take over. The choice between using database-level shadow copy versus event-level replay often comes down to what data you have available.
Replay might be straightforward if you have a clean event log (like an append-only journal). Otherwise, tapping into the database (via triggers or log capture) might be more manageable. Both approaches aim for eventual consistency, but shadow table syncing (especially trigger-based) will typically have the new store up-to-date within seconds of the original, whereas an event replay might apply changes in batches and catch up after some delay.
Conclusion
The shadow table strategy has proven effective in performing complex data migrations safely and incrementally. Teams keep a live replica of data changes; this enables them to migrate databases, extract services, or refactor schemas without halting the application. Companies apply this pattern to add columns without downtime, migrate enormous tables, or gradually siphon traffic to new microservices, all while preserving data integrity.
Of course, no single approach fits all situations. Shadow tables shine when you need up-to-the-second synchronization and confidence through parallel run comparisons. Alternatives like dual-writes or event replay might be more appropriate in systems built around event messaging or in simpler scenarios where a full shadow copy is overkill. Many real-world migrations end up using a blend of these techniques. For example, one might do an initial bulk load (replay), then switch to a live shadow sync, or use dual-writes in the app plus a trigger-based audit to double-check consistency.
It’s essential that software engineering teams plan migrations as first-class projects and leverage industry best practices: they should run systems in shadow mode to validate behavior, keep toggles or backstops for quick rollback, and monitor everything. When executed with discipline, the shadow table strategy provides a moderate complexity path to achieve significant changes with little downtime. It enables the evolutionary changes that modern software demands, all while keeping users blissfully unaware that anything changed under the hood.
