9. Data migration

Moving sufficient and accurate data from ‘source to target’ systems is known as Data Migration (DM). This will need to cover static and dynamic, including in-flight transactions. Also, feeding downstream systems and archiving should be considered.

There are four main stages of data migration:

• Extract, which involves retrieving the data from the source system(s).
• Transform, which changes the data so it functions properly in the new ERP system.
• Load, which involves injecting the transformed data into the new ERP.
• Validate, which checks that the data is correct.

In reality, the process becomes more complex but is based on these fundamentals.

Key points

• Data migration must be optimised to ensure it can be done within cutover time constraints.
• Data must be validated by Functional Leads at its destination to make sure it’s correct.

Information needed in data migration:

• ETL (Extract Transform Load) logic
• Data source list
• Data mapping document
• Data reconciliation reports
• Data cut-offs (last entries to source system)
• Data protection documentation
• ETL duration time constraints

The following more detailed flow shows an ERP data migration approach without delta migration. It requires the least effort for extraction and transformation but may lead to a longer data freeze or additional work through interim processes (compared to other options, discussed later).

This flow illustrates an ERP data migration with delta migration. While it requires the most effort for extraction and transformation, it minimises data freeze, reduces user impact, and typically eliminates the need for interim processing.

DM optimisation vs DM in cutover

The data migration process is repeated multiple times before cutover to optimise and refine each step. This ensures that the process is complete, accurate, repeatable, and achievable within the available timeframe during cutover. This preparation is vital, as during the data migration in cutover, many business processes will need to pause or have interim processes set up, which can involve significant effort. Therefore, an ‘acceptable data freeze window’ should be established early on, defining the target maximum duration for the migration and other cutover tasks.

The following steps detail the typical activities required to prepare the data migration process for cutover readiness.

Define the scope and data requirements: Agree on the scope of the new solution and identify the specific data needed. This determines the data objects to migrate.

Agree on the ‘acceptable data freeze window’ Check when the data freeze can occur to minimise disruption.

Identify data sources: Locate where each data object resides. This defines the list of source systems (the source list), which may include multiple platforms.

Build and test the ETLV process: Develop and test the Extract, Transform, Load, and Validate (ETLV) process.

Prepare for delta or interim migrations: If delta migrations are planned, create these processes. If not, prepare interim business processes for the coming data freeze and communication plans.

Document the migration process: Capture all steps in the data migration flow to build a comprehensive data migration runbook.

Run multiple ETLV cycles: Execute repeated test cycles, using reconciliation and validation reports to confirm data accuracy, durations and process stability.

Understand business process dependencies: Identify the business processes that must pause during migration and establish clear data cut-off points.

Define extraction timing: Determine the precise timing for data extraction from each source system.

Plan for payroll testing: Extract and retain payroll data for comparison testing, ensuring timing aligns with a complete pay period.

Confirm load sequencing and approvals: Define the sequence for data loading and specify who is responsible for validation and sign-off at each stage.

Monitor readiness for cutover: Track and report ETLV duration and accuracy in the lead-up to cutover to confirm data migration readiness.

Agreeing on the data migration approach early in the strategy phase is essential, as making changes later in the programme becomes difficult once other workstreams have already begun. If the ‘acceptable data freeze window’ and the organisation’s tolerance are minimal, adopting a delta migration approach may be the most effective solution.

Pitfall: Taking a low-cost proposal without delta migrations.

‘No-delta’ approaches often rely on the manual capture and load of data off-system, or even dual keying. This can be a huge undertaking.

Solution: In the strategise stage, understand if the data migration strategy would work for cutover. Consider delta or ‘top-up’ migrations to minimise any data freeze periods.