Arch Forge Solutions

SOFTWARE ARCHITECTURE DESIGN

Migration & Replication

Design for initial migration of data at production start from legacy systems or manual entry. Design for migration at new version release upgrade of existing data structures. Replication of business data for reconciliation or data warehousing.

Data Migration understood in a software development context is a process of porting data from one component of the system to another component of a system. Migration can be done between two separate systems that need to move data, or between two versions of the same system during a software version upgrade. Typical use cases are:

Moving data from a legacy system out to a newly created system that will be used from now on (Go Live Migration)
Moving data from previous version of a database scheme to a newer version where data structures have to be adjusted
Moving a subset of data for a tenant in a multitenant system or SaaS. Typical when switching SaaS providers or seeding account with existing data

If the migration is a not one-time operation, then typically we would not name it migration but replication.

Data Replication is a type of backend integration that deals specifically with transferring a large number of data entries between landscape systems for operational purposes. Microservices also use data replication to some extent. In cases where we want to completely separate data models between components, replication allows to transfer data in a near real-time manner. Typically, we see two types of replication:

Data replication in master/slave mode between whole systems in a landscape or between business parties
(big complex migration with many objects)
Data replication between microservices to achieve local data independency (small models 1-2 entities)

Typical software challenges we encounter with customer projects:

Lack of proper data model definition for migration (ex: no object versioning)
Bad split of model objects for migration causing time dependencies and synchronisation issues
Excessively large data sets for migration that cause long system unavailability when migrating at Go Live or upgrade
Migration that is not idempotent = can’t be executed many times on same data set without breaking the data
Objects processed in wrong order cause data inconsistency in target system or rejection of data at all
No migration selectivity for fixing individual migration errors
No data validation checks for data quality before migration
No monitoring makes reporting on migration results and errors impossible
Lack of performance because of bad bulk and bundle processing including improper error handing of transactions
Improper queue management causing message resending to components
Bad usage of initial and delta mode in data stream
No monitoring for data flows to alert staff early
Lack of proper Migration documentation for debugging and fixed in the future (No interface contract!)

Do you want to risk rolling back an entire Go Live of a system because of a failed migration?

Do you want to suffer system errors and complete system block because of wrong data?

Imagine users complaining that records are not updated in their system, or customers complaining about services not working because of old data. Or worse: Money discrepancies between systems in the financial services industry that raise fraud questions and contract disputes?

If you feel that you would benefit in your project by fixing such points or mitigating presented risks, we can help.

What we can do to solve typical migration/replication problems:

Fix migration interfaces addressing all known migration issues based on our extensive experience
Analyse data domain and use proper migration processing order that will not damage your data
Build a redundant solution that allows faster error fixing and partial migrations
Introduce monitoring and reporting for stakeholders so that they know if data is usable in target system
Optimise for better performance so that the migration window time is minimal
Prepare documentation based on created design for later reference
Help at development and final execution of migration at Go Live
Provide support at and after Go-Live to achieve required transfer quota

Technology

Technologies & tools that we typically use in our migration and replication projects.