Testing – Data Migration https://www.softwaretestingportal.com Let's learn something new.....together Sun, 05 Apr 2026 08:55:31 +0000 en-US hourly 1 https://wordpress.org/?v=7.0.4 https://www.softwaretestingportal.com/wp-content/uploads/2018/09/cropped-Logo-again-middle-1-1-32x32.jpg Testing – Data Migration https://www.softwaretestingportal.com 32 32 Data Migration Testing | Test Strategy | Testing Phases | Challenges https://www.softwaretestingportal.com/data-migration-testing/ https://www.softwaretestingportal.com/data-migration-testing/#comments Sun, 30 Aug 2020 03:58:40 +0000 http://www.softwaretestingportal.com/?p=1474

In the last article on Data Migration, we learnt What is the Data Migration, Types of it and various strategies. Now it’s the time to learn about Data Migration Testing.

What is Migration Testing?

  • Migration Testing is a verification process of migration of the legacy system to the new system with minimal disruption/downtime, with data integrity and no loss of data, while ensuring that all the specified functional and non-functional aspects of the application are met post-migration.
  • Migration Testing verifies the process of migrating the data from the legacy, or current system, to the new target system. It does so with limited downtime and no loss of data or issues with data integrity. In addition, it does so while making sure functional and non-functional requirements / expectations of the application, are met post-migration.

Why Migration Test?

 
As we know, the application migration to a new system could be for various reasons, system consolidation, obsolete technology, optimisation, or any other reasons.
Hence while the System in Use needs to be migrated to a new system, it is essential to ensure the below points:
  1. Any kind of disruption/inconvenience caused to the user due to migration needs to be avoided or minimised like downtime, loss of data.
  2. Need to ensure if the user can continue to use all the features of the application.
  3. To ensure compatibility of the new/upgraded application with the hardware and software that the legacy application supports.
  4. To ensure the existing functionalities works as in the legacy application seamlessly. 
  5. Critical Defects related to the data and data type need to be identified & fixed during testing.
  6. To ensure no performance degradation post migration.
  7. To ensure if the connection between servers, hardware, software and therefore the Data flow between different components remains intact.
Hence in order to ensure a smooth migration by eliminating the above defects, it is essential to carry out in-depth Migration Testing.

Importance of Data Migration Testing

There are some challenges that could arise during data migration testing, which is precisely why the testing itself is so crucial. Some of these challenges include:
  • Data corruption
  • Testing the mapping or path of data.
  • Unintended data modifications in the legacy system and in the data structures mapping.
By uncovering and resolving these issues during the testing phase can be priceless and a success.

Test Phases in Migration Testing

The different phases of Migration test need to be considered:
1. Pre-Migration Testing
2. Data Cleansing
3. Post Migration Testing

Phase 1 : Pre-Migration testing

 
Before any migration has taken place, the following testing actions should take place:
  • Ensure that the scope of the data is clearly understood what data has to be included, what data has to be excluded, which data needs.
  • Ensure that the destination as well as the load process is understood. 
  • Perform data mapping between legacy and the new application.
  • Ensure that the data scheme is known including mandatory fields, field names, field types, data types etc. for both the original data source and the destination system.
  • Understand the data cleansing requirements.
  • Understand any interfacing with other systems or secondary suppliers of data into the system.
  • Tables in the legacy system are to be noted down and if any tables are dropped and added post migration needs to be verified.
  • Records in each table, views should be noted in the legacy application.
  • Prepare test cases, test scenarios, and use cases for new conditions in the new applications.
  • Execute a set of test cases, scenarios with a set of users and keep the results, logs stored. The same needs to be verified after Migration to ensure that legacy data and functionality are intact.
  • Count of the data and records should be noted down clearly, it needs to be verified after Migration for no loss of data.
  • Ensure the mapping to user interface is correct.
  • Ensure the mapping to business process is correct.
  • Test tool configuration and script generation using field details.
  • Identify data subset (sampling) details for testing.
  • Ensure all business cases, user stories, or use cases are understood.

Phase 2 : Data Cleansing

It is vitally important that data cleansing is performed on a system that houses and maintains data.  The following data cleansing tasks should be planned:
  • Understand the error types: these may include blank fields, too-long data lengths, or bad characters.
  • Identify method of checking the data: either interrogation through database using SQL commands or the use of data migration tools.
  • Understand the data dependencies i.e. if changing the data in one table impacts the data in another linked table.
  • Fix the data issues.
  • Verify the count of the data.

Phase 3 : Post-Migration Testing

Following the migration of the application is migrated successfully, Post-Migration testing comes into the picture.
In addition to testing the known business flows, the testers should carry out the following testing, including negative testing approaches, which primarily ensure that data cleansing is being carried out at run time within the system:
  1. Check whether all the data in the legacy is migrated to the new application within the downtime that was planned. Example compare the number of records between legacy and the new application.
  2. Check whether all the schema changes as per the new system are updated.
  3. Data migrated from the legacy to new application should retain its value and format.
  4. Test the migrated data type including business data types and technical data types.
  5. Check the data flow within the application
  6. Check and verify the earlier supported functionality
  7. Check for legacy data’s redundancy. No legacy data should be duplicated itself during migration
  8. Input bad data: attempt to violate the validation rules including the use of boundary value analysis, equivalence partitioning and error guessing.
  9. Bypass mandatory data – attempt to proceed further without filin the mandatory data fields.
  10. Checking data locks:  Where data is being written, it should not be possible for multiple users to access the same new record within the database.
  11. Check for database security and data integrity.
  12. Create new users on the system and carry out tests to ensure that functionality is accessible to the newly created user.
  13. Data segregation – Deleting the data in the new application, should not delete data in legacy as well. Similarly, any data addition in the new application should not reflect back on the legacy system.
  14. Performance Testing: To ensure that migration has not degraded the performance of the system and Data is accessible in accordance to the required performance.
  15. Security Testing: Access to the data should be restricted appropriately.
  16. Usability: Verify the ease of Use of the functionalities migrated for the end user
  17. Backward Compatibility Testing: Migration of the system also calls for the testers to verify the ‘Backward Compatibility’, wherein the new system introduced is compatible with the old system.
  18. Rollback Testing – Migration failure test scenarios need to be designed as part of negative testing and rollback mechanism needs to be tested in case of any issues while carrying out the migration or if there is a migration failure at any point of time during migration.

Data Migration Testing Strategy

Designing the test strategy for migration include a set of activities to be performed and few aspects to be considered. This is to minimise the errors and risks that occur as a result of migration and to perform the migration testing effectively.
Activities in this Testing:

Specialized team formation: Form the testing team with the members having the required knowledge & experience and provide training related to the system that is being migrated.
Business risk analysis: Current business should not be hampered after migration and hence carry out ‘Business Risk Analysis’ meetings involving the right stakeholders and identify the risks and the implementable mitigation. The testing should include scenarios to uncover those risks and verify if proper mitigation have been implemented.

Possible errors analysis: Conduct ‘Possible Error Analysis’ using appropriate ‘Error Guessing Approaches’ and then design tests around these errors to unearth them during testing.
Migration scope analysis and identification: Analyse the clear scope of the migration test as when and what needs to be tested.

Identify the appropriate Tool for Migration: While defining the strategy of this testing, automated or manual, identify the tools that are going to be used. e.g. Automated tool to compare source and destination data.
Identify the appropriate Test Environment for Migration:
Identify separate environments for Pre and Post Migration environments to carry out any verification that is required as part of testing. Understand and document the technical aspects of the Legacy and New system of Migration, to ensure that the test environment is set up as per that.

Migration Test Specification Document and review: Prepare Migration Test Specification document which clearly describes the test approach, areas of testing, testing methods i.e. automated, manual, testing methodology Number of cycles of testing, schedule of testing, approach of creating data and using live data , test environment specification, testers qualification etc. and run a review session with the stakeholders.

Production launch of the migrated system: Analyse and document the to-do list for production migration and publish it well in advance.

Challenges in Data Migration Testing

 
Challenges faced in this testing are mainly with data. Below are few in the list:
  1. Data Quality: We may find that the data used in the legacy application is of poor quality in the upgraded application. In such cases, data quality has to be improved to meet business standards. Some of the ways to resolve this may be execute some mock migrations and pilot migrations before the final migration. It may give some errors which we never expected during any of the previous phases.
  2. Truncation and precision of data: In this case, the data to be migrated is partially moved to the target system. It can be challenging because of the additional padded spaces or a smaller number of bytes in the source data.
  3. Data type and Data Mismatch: Data and Data type migrated from the legacy to the upgraded application may mismatch in the new one. This may be due to the change in data type, format of data storage, the purpose for which the data is being used may be redefined. SO The data types must be mapped correctly. 
  1. Data Loss: Data might be lost while migrating from the legacy to the upgraded application. This may be with mandatory fields or non-mandatory fields. If the data lost is for non-mandatory fields, that is less risky as it can be updated again but if the mandatory field’s data is lost, then the record itself becomes void and it cannot be retracted. This might result in data loss and should have to be retrieved.
  1. Null data translation: Null data should be translated to the Target as null itself and not as spaces or default values. The application can have some checks for null validation. 
  2. Data Volume: Huge Data that requires a lot of time to migrate within the downtime window of the migration activity. 
  1. Extra records:Duplicate records can come from different data sources. So, before loading the data to the target system, a filtering and transformation of the data needs to be done
  2. Simulation of a real-time environment:Simulation of a real-time environment in the testing lab is another real challenge, as testers get into different kind of issues when they test the application / system with the the real data and the real system. So, data sampling, replication of real environment, identification of volume of data involved in migration is quite amportant while carrying out data Migration Testing.
  1. Unclear / Missing requirements: There are instances where some requirements are missed due to various reasons including lack of communication with end users or subject matter experts. If it’s not addressed properly, it affects the data migration project hugely.
  2. Extraneous / Duplicate records:Duplicate records can come from different data sources. So, before loading the data to the target system, a filtering and transformation of the data needs to be done in order to remove the duplicate records.

Final Thoughts

According to the Gartner Group “83% of data migrations fail outright or exceed their allotted budgets and implementation schedules”.
Data is a critical component to the success of any business. That data needs to be easily accessible and found in its rightful place, on the right system, and using the correct optimization.
To avoid all the aforementioned risks as much as possible, the testing performed on data migration ensures that data is kept intact and readily available.
Overall, in order to make data migration project a success, a combination of a stringent approach, proactive risk mitigation techniques and comprehensive test activities are inevitable.
]]>
https://www.softwaretestingportal.com/data-migration-testing/feed/ 1
Data Migration – Plan, Strategy, Process and more https://www.softwaretestingportal.com/data-migration/ Mon, 10 Aug 2020 10:31:18 +0000 http://www.softwaretestingportal.com/?p=1418
 
Businesses are spending billions of dollars in migrating data in migrating systems, yet many of the new systems fail to meet expectations.
In fact More than 50% of data migration projects will exceed budget and/or failing the business due to flawed strategy and execution, according to Gartner. Nevertheless, data migrations are a necessity and fact of IT life.
So let’s understand this complex though inevitable process at a glance.

What's Data Migration!

Data Migration is a process that includes moving data between formats, computer systems, and storage types.
We can also say that it is the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another.
Additionally, the validation of migrated data for completeness and the decommissioning of legacy data storage are considered part of the entire data migration process.

Why Data Migration!

 
  • Upgrading Computer / Legacy systems
  • Changing Computer / Legacy systems
  • Merging systems i.e. Data Integration
There are many reasons your enterprise might need to undertake a data migration project. For example, you might be replacing servers or storage devices or consolidating or decommissioning data center.
Data migration is also an essential step in the overall process of migrating on-premises IT infrastructure to a cloud computing environment.
Whether you’re moving to a public cloud, private cloud, hybrid cloud, or multi cloud environment, you’ll need to find a secure, cost-effective, and efficient method of migrating your data to its new storage location.
Improved technology often means business is thriving and is improving alongside its competitors. That said, the act of migrating data can leave previous information vulnerable. If critical data is lost, the business could be left at a standstill.

And What’s not the Data Migration!

 
Note that data migration is not the same thing as data conversion or data integration:

Data Migration

Data Conversion

Data Integration

 
Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.
 
Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.
 
Combines stored data residing in different systems to create a unified view and global analytics.

6 Types of Data Migration

 
 
Data is stored on various media in files or databases, and is generated and consumed by software applications, which in turn support business processes. The need to transfer and convert data can be driven by multiple business requirements, and the approach taken to the migration depends on those requirements. The major migration categories are as follows:

Storage migration

Data center migration

Application migration

A business may choose to rationalise the physical media to take advantage of more efficient storage technologies.This will result in having to move physical blocks of data from one tape or disk to another, often using virtualisation techniques.
Migrating an entire data centre environment to the cloud or another location is a large-scale, comprehensive process. Completing such a migration project successfully—with minimal downtime or disruption to operations—requires careful planning and coordination.
Changing application vendor—for instance a new CRM or ERP platform—will inevitably involve substantial transformation as almost every application or suite operates on its own specific data model.

Database migration

Business process migration

Data migration to the cloud

  • It may be necessary to move from one database vendor to another, or to upgrade the version of database software being used.
  • Some modern applications are written to be almost entirely agnostic to the database technology, so a change from Sybase or SQL Server to Oracle should only require a testing cycle.
  • Database migration is an example of specialised workload migration.
  • Many public and private cloud providers offer tools that can facilitate or automate parts of the database migration process to ensure that your database remains secure throughout the transfer and that no data loss or corruption occurs. 
 
  • Business processes operate through a combination of human and application systems actions.
  • When these actions change they require the movement of database or application data.
  • Examples of such migration drivers are mergers and acquisitions, business optimisation, and reorganisation to enter new markets or respond to competitive threat.
  • The last two categories directly affect the operational users of processes and applications.
  • A highly adaptive approach,  a business-oriented audit and clear visibility for stakeholders are the key requirements in such migrations.
 
 
  •  
  • Today, businesses generate ever-growing amounts of data and face increasingly urgent pressure to maximise the value they extract from it. 
  • Therefore, many organisations are choosing to move workloads to the cloud for a cost-effective and best-performing IT environment.
  • There are several options for transferring data from a local data centre to the cloud, broadly, they fall into following two categories.

    1. Online migration, in which data moves across the Internet or a private or dedicated WAN connection.

    2. Offline migration, in which data is transferred via a storage appliance that’s physically shipped between data centre and the target cloud storage location.

9 Steps for Successful Data Migration

1. Design a Strategy

  •  
  • Selecting a data migration strategy depends on the need for data migration. Is it because the organization is consolidating systems following an acquisition or a merger or is it because of data overload?
  • The design phase is where organisations define the type of migration — big bang or trickle. This also involves drawing out the technical architecture of the solution and detailing the migration processes.
  • Then you needs to decide whether to go for data migration services or one of the ETL tools to facilitate data migration.
  • These tools will identify the source and destination systems, generate workflows, and migrate the content. Alternatively, a code-free data migration solution can give you the freedom to carry out the migration at your convenience, speeds up the project, and reduces cost and risks.

2. Assess the Data source and Analyse

Before migrating data, you must know what you’re migrating, as well as how it fits within the target system. 
So the next step is to analyse enterprise data. Answering the following questions can help with this assessment:
  • How much data is pulling over?
  • What that data looks like?
  • How much of the enterprise data requires migration?
  • Is the data structured or unstructured?
  • Is data redundancy going to be an issue?
  • What needs to migrate over, what can be left behind
  • How much of the data is ROT (redundant, obsolete, or trivial)?
  • Is the information old or recent?

3. Build a Migration Solution

Since, data migration is a one-time and large activity, it’s crucial to get it right.
A common tactic is to break the data into subsets and build out one category at a time, followed by a test. For particularly large migration projects, it might make sense to build and test in parallel.

4. Collect and Cleanse Data

This step involves removing data which is not required, compressing the remaining content, and converting data into the desired format.
Manual execution of these tasks can be time and resource intensive for large datasets. Therefore, automating these tasks can make the migration process more efficient.

5. Sort Data

Once you have profiled the data into a high quality and usable format, the next step is to categorise it according to the migration requirements.
It makes easier to route data into the right buckets. For example in Banking, data can be organised based upon the customer types, product type, portfolio types, account types and any other criteria.

6. Validate Data

This step involves reviewing the execution process. Assess the data rules and check if they are working the way they are supposed to, and watch for any exceptions in your dataflow.
After sorting the data, apply the data quality rules to determine the errors based on defined rules and to identify and eliminate the records in which the field ‘Quantity’ has a zero value.

7. Migrate

This is the final step in which everything falls into place. The steps mentioned above will give you well-organised and clean datasets.
Now all we have to do is migrate them from one system to another. Once the data is migrated to the target system, which can be a database or a data repository.

8. Conduct a Live Test

The testing process isn’t over after testing the code during the build phase.
It’s important to test the data migration design with real data to ensure the accuracy of the implementation and completeness of the application.

9. Audit

Once the implementation has gone live, set up a system to audit the data in order to ensure the accuracy of the migration.

Data Migration Strategies

“Big Bang” Migration

“Trickle” Migration

 
 
In a big bang data migration, as the name suggests, full transfer is completed within a limited window of time.
Live systems experience downtime while data goes through ETL processing and transitions to the new database.
It all happens in one time-boxed event, requiring relatively little time to complete. The pressure can be intense, as the business operates with one of its resources offline. This risks a compromised implementation.
 
 
Trickle migrations, in contrast, complete the migration process in phases.
During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions.
Compared to the big bang approach, trickle implementation can be fairly complex in design. However, usually reduces risks.

Advantages of big bang migration

Advantages of trickle migration

  • Less costly
  • Usually less complex
  • All changes happen one time only, in a relatively short space of time.
  • Less prone to expensive surprises.
  • Zero downtime required – as the migration is incremental
  • Easy rollback – If a single phase fails, it’s only that phase that needs to be rolled back and repeated

Disadvantages of a big bang

Disadvantages of a trickle

 
  •  
  • A high risk of expensive failure as problems may be discovered after complete migration.
  • If the migration fails, a complete roll back is required.
  • Requires downtime.
 
  •  
  • More expensive.
  • It requires to maintain multiple live environments

Risks and challenges

1. Data Loss

2. Security

3. Compatibility

During the data migration process, data loss can occur.
On a small scale, this may not be a problem as there is less data or easy to restore the missing data or backup. However this is a major challenge in large scale migration programs.
Organisations need to make sure that all data is securely encrypted before migration.
For offline migrations that involve shipping data storage devices, it’s important to verify security of the shipper’s freight and logistics services.
Compatibility issues are also possible in data transfer, such as changed operating systems and unexpected file formats.
Although the data is not actually lost, however, business cannot access it in the target system.

4. Long transfer times

5. Unexpected Higher Costs

 
It can be challenging to predict  exact online transfer times.
Network impediments may restrict connection speeds or hardware restrictions might throttle the amount of data.
Unexpected costs often result from improper planning. For example, delays in online transfers additional charges.
Also, Keeping Vendor-provided storage appliances for longer than the agreed times can also raise the expanses.

Best Practices

1. Understand the data usage

2. Assess the environments

 
 
  • Who uses the data,
  • Who will use it in the future,
  • How will it be used? 
  • Formatting requirements ?
    Be sure to gather the requirements from relevant stakeholders and business units throughout the migration process.
 
  • Will the same operating system be running in source and target environments?
  • Will database schema or other formatting need to change?
  • Are there any issues that need to be addressed pre-migration? 

3. Verify business requirements

4. Back up the data

 
  • What kind of migration timeline is necessary?
  • If a data centre is being decommissioned, when will its lease expire?
  • What types of data security must bee maintained throughout the migration process?
  • Is any data loss or corruption tolerable, and if so, how much?
  • How would unexpected delays affect the business?
  • Make sure there are backup resources and that they’ve been tested before proceeding.
  • One of the good practice is to create backup images that can be immediately restored to the original system.

5. Stick to the strategy

6. Test, test, test

 
  • The migration process can be complicated at times, so be ready for it and stick to the plan.
  •  
  • Make sure everything’s where it should be, create automatic retention policy, clean up the data and double check permissions.
  • During the planning and design phases, and throughout implementation and maintenance, test the data migration to make sure to achieve the desired outcome.

Data Migration Tools

On-premise data migration tools

Open source data migration tools

Cloud-based data migration tools

  • On-premise solutions are designed to migrate data between two or more servers or databases within a large or medium enterprise/network, without moving data to the cloud.
  • These solutions are optimal if you are performing tasks like changing data warehouses or moving the location of your primary data store.
  • Some companies prefer on-premise solutions due to security restrictions.
  • Open source is software you can use, modify, and share because its design is publicly accessible.
  • Typically, open source solutions are free or lower in cost than commercial alternatives.
  • Open source data migration tools can be a practical option for migrating your data if the project is not too large or complex.
 
  •  
  • Cloud-based data migration solutions are the latest generation and are designed to move data to the cloud.
  • Cloud-based solutions are optimal if you are already storing your data in the cloud or if you intend to move your data to the cloud.
  • Many organisations are now going for cloud based solutions considering cost efficiencies and enhanced security in moving data from on-premise to the cloud.
  • Moreover, cloud-based data migration tools tend to be flexible about the types of data they can handle.
 
 
Today, there are plenty of tools to facilitate enterprise data migrations. Generally, Data migration strategy  determines which tools work best for the specific project.
Some popular Data Migration tools include the following:

Rsync for host-based file-level migration
Veeam to speed and simplify the process of migrating VM-based workloads.
EMC Rainfinity for network-based file-level migration.
Zerto offers a converged disaster recovery, backup, and workload mobility platform.
HDS Universal Replicator for array-based block-level migration
Rclone, an open source command line utility, used to migrate data to and from cloud object storage.
Brocade DMM for network-based block-level migration
Cyberduck is an open source FTP and SFTP client, used to transfer individual files or file volumes.
Volume managers for host-based block-level migration
Windows 10 Migration Accelerator
Zinstall
• Acronis True Image
• Minitool Partition Wizard Free
• Laplink PCmover
• Samsung Data Migration Tool
• AOMEI Backupper Standard
• EaseUS Todo Backup/PC Trans Free
• Paragon Drive Copy Professional
• Samsung Smart Switch

Conclusion

Whatever the reason for the data migration, its ultimate aim should be to improve corporate performance and deliver competitive advantage.
To succeed, data migrations must be given the attention they deserve, rather than simply being considered part of a larger underlying project.
Lacking this and without proper planning, there is a high risk that the project will go over budget, exceed the allotted time, or even fail completely.
Following a structured methodology will reduce the pain of managing a complex data migration, whereas the correct choice of technology will lead to a successful outcome.
]]>