Businesses are spending billions of dollars in migrating data in migrating systems, yet many of the new systems fail to meet expectations.
In fact More than 50% of data migration projects will exceed budget and/or failing the business due to flawed strategy and execution, according to Gartner. Nevertheless, data migrations are a necessity and fact of IT life.
So let’s understand this complex though inevitable process at a glance.
What's Data Migration!
Data Migration is a process that includes moving data between formats, computer systems, and storage types.
We can also say that it is the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another.
Additionally, the validation of migrated data for completeness and the decommissioning of legacy data storage are considered part of the entire data migration process.
Why Data Migration!
Upgrading Computer / Legacy systems
Changing Computer / Legacy systems
Merging systems i.e. Data Integration
There are many reasons your enterprise might need to undertake a data migration project. For example, you might be replacing servers or storage devices or consolidating or decommissioning data center.
Data migration is also an essential step in the overall process of migrating on-premises IT infrastructure to a cloud computing environment.
Whether you’re moving to a public cloud, private cloud, hybrid cloud, or multi cloud environment, you’ll need to find a secure, cost-effective, and efficient method of migrating your data to its new storage location.
Improved technology often means business is thriving and is improving alongside its competitors. That said, the act of migrating data can leave previous information vulnerable. If critical data is lost, the business could be left at a standstill.
And What’s not the Data Migration!
Note that data migration is not the same thing as data conversion or data integration:
Data Migration
Data Conversion
Data Integration
Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.
Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.
Combines stored data residing in different systems to create a unified view and global analytics.
6 Types of Data Migration
Data is stored on various media in files or databases, and is generated and consumed by software applications, which in turn support business processes. The need to transfer and convert data can be driven by multiple business requirements, and the approach taken to the migration depends on those requirements. The major migration categories are as follows:
Storage migration
Data center migration
Application migration
A business may choose to rationalise the physical media to take advantage of more efficient storage technologies.This will result in having to move physical blocks of data from one tape or disk to another, often using virtualisation techniques.
Migrating an entire data centre environment to the cloud or another location is a large-scale, comprehensive process. Completing such a migration project successfully—with minimal downtime or disruption to operations—requires careful planning and coordination.
Changing application vendor—for instance a new CRM or ERP platform—will inevitably involve substantial transformation as almost every application or suite operates on its own specific data model.
Database migration
Business process migration
Data migration to the cloud
It may be necessary to move from one database vendor to another, or to upgrade the version of database software being used.
Some modern applications are written to be almost entirely agnostic to the database technology, so a change from Sybase or SQL Server to Oracle should only require a testing cycle.
Database migration is an example of specialised workload migration.
Many public and private cloud providers offer tools that can facilitate or automate parts of the database migration process to ensure that your database remains secure throughout the transfer and that no data loss or corruption occurs.
Business processes operate through a combination of human and application systems actions.
When these actions change they require the movement of database or application data.
Examples of such migration drivers are mergers and acquisitions, business optimisation, and reorganisation to enter new markets or respond to competitive threat.
The last two categories directly affect the operational users of processes and applications.
A highly adaptive approach, a business-oriented audit and clear visibility for stakeholders are the key requirements in such migrations.
Today, businesses generate ever-growing amounts of data and face increasingly urgent pressure to maximise the value they extract from it.
Therefore, many organisations are choosing to move workloads to the cloud for a cost-effective and best-performing IT environment.
There are several options for transferring data from a local data centre to the cloud, broadly, they fall into following two categories.
1. Online migration, in which data moves across the Internet or a private or dedicated WAN connection.
2. Offline migration, in which data is transferred via a storage appliance that’s physically shipped between data centre and the target cloud storage location.
9 Steps for Successful Data Migration
1. Design a Strategy
Selecting a data migration strategy depends on the need for data migration. Is it because the organization is consolidating systems following an acquisition or a merger or is it because of data overload?
The design phase is where organisations define the type of migration — big bang or trickle. This also involves drawing out the technical architecture of the solution and detailing the migration processes.
Then you needs to decide whether to go for data migration services or one of the ETL tools to facilitate data migration.
These tools will identify the source and destination systems, generate workflows, and migrate the content. Alternatively, a code-free data migration solution can give you the freedom to carry out the migration at your convenience, speeds up the project, and reduces cost and risks.
2. Assess the Data source and Analyse
Before migrating data, you must know what you’re migrating, as well as how it fits within the target system.
So the next step is to analyse enterprise data. Answering the following questions can help with this assessment:
- How much data is pulling over?
- What that data looks like?
- How much of the enterprise data requires migration?
- Is the data structured or unstructured?
- Is data redundancy going to be an issue?
- What needs to migrate over, what can be left behind
- How much of the data is ROT (redundant, obsolete, or trivial)?
- Is the information old or recent?
3. Build a Migration Solution
Since, data migration is a one-time and large activity, it’s crucial to get it right.
A common tactic is to break the data into subsets and build out one category at a time, followed by a test. For particularly large migration projects, it might make sense to build and test in parallel.
4. Collect and Cleanse Data
This step involves removing data which is not required, compressing the remaining content, and converting data into the desired format.
Manual execution of these tasks can be time and resource intensive for large datasets. Therefore, automating these tasks can make the migration process more efficient.
5. Sort Data
Once you have profiled the data into a high quality and usable format, the next step is to categorise it according to the migration requirements.
It makes easier to route data into the right buckets. For example in Banking, data can be organised based upon the customer types, product type, portfolio types, account types and any other criteria.
6. Validate Data
This step involves reviewing the execution process. Assess the data rules and check if they are working the way they are supposed to, and watch for any exceptions in your dataflow.
After sorting the data, apply the data quality rules to determine the errors based on defined rules and to identify and eliminate the records in which the field ‘Quantity’ has a zero value.
7. Migrate
This is the final step in which everything falls into place. The steps mentioned above will give you well-organised and clean datasets.
Now all we have to do is migrate them from one system to another. Once the data is migrated to the target system, which can be a database or a data repository.
8. Conduct a Live Test
The testing process isn’t over after testing the code during the build phase.
It’s important to test the data migration design with real data to ensure the accuracy of the implementation and completeness of the application.
9. Audit
Once the implementation has gone live, set up a system to audit the data in order to ensure the accuracy of the migration.
Data Migration Strategies
“Big Bang” Migration
“Trickle” Migration
In a big bang data migration, as the name suggests, full transfer is completed within a limited window of time.
Live systems experience downtime while data goes through ETL processing and transitions to the new database.
It all happens in one time-boxed event, requiring relatively little time to complete. The pressure can be intense, as the business operates with one of its resources offline. This risks a compromised implementation.
Trickle migrations, in contrast, complete the migration process in phases.
During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions.
Compared to the big bang approach, trickle implementation can be fairly complex in design. However, usually reduces risks.
Advantages of big bang migration
Advantages of trickle migration
- Less costly
- Usually less complex
- All changes happen one time only, in a relatively short space of time.
- Less prone to expensive surprises.
- Zero downtime required – as the migration is incremental
- Easy rollback – If a single phase fails, it’s only that phase that needs to be rolled back and repeated
Disadvantages of a big bang
Disadvantages of a trickle
- A high risk of expensive failure as problems may be discovered after complete migration.
- If the migration fails, a complete roll back is required.
- Requires downtime.
- More expensive.
- It requires to maintain multiple live environments
Risks and challenges
1. Data Loss
2. Security
3. Compatibility
During the data migration process, data loss can occur.
On a small scale, this may not be a problem as there is less data or easy to restore the missing data or backup. However this is a major challenge in large scale migration programs.
Organisations need to make sure that all data is securely encrypted before migration.
For offline migrations that involve shipping data storage devices, it’s important to verify security of the shipper’s freight and logistics services.
Compatibility issues are also possible in data transfer, such as changed operating systems and unexpected file formats.
Although the data is not actually lost, however, business cannot access it in the target system.
4. Long transfer times
5. Unexpected Higher Costs
It can be challenging to predict exact online transfer times.
Network impediments may restrict connection speeds or hardware restrictions might throttle the amount of data.
Unexpected costs often result from improper planning. For example, delays in online transfers additional charges.
Also, Keeping Vendor-provided storage appliances for longer than the agreed times can also raise the expanses.
Best Practices
1. Understand the data usage
2. Assess the environments
- Who uses the data,
- Who will use it in the future,
- How will it be used?
- Formatting requirements ?
Be sure to gather the requirements from relevant stakeholders and business units throughout the migration process.
- Will the same operating system be running in source and target environments?
- Will database schema or other formatting need to change?
- Are there any issues that need to be addressed pre-migration?
3. Verify business requirements
4. Back up the data
- What kind of migration timeline is necessary?
- If a data centre is being decommissioned, when will its lease expire?
- What types of data security must bee maintained throughout the migration process?
- Is any data loss or corruption tolerable, and if so, how much?
- How would unexpected delays affect the business?
- Make sure there are backup resources and that they’ve been tested before proceeding.
- One of the good practice is to create backup images that can be immediately restored to the original system.
5. Stick to the strategy
6. Test, test, test
- The migration process can be complicated at times, so be ready for it and stick to the plan.
- Make sure everything’s where it should be, create automatic retention policy, clean up the data and double check permissions.
- During the planning and design phases, and throughout implementation and maintenance, test the data migration to make sure to achieve the desired outcome.
Data Migration Tools
On-premise data migration tools
Open source data migration tools
Cloud-based data migration tools
On-premise solutions are designed to migrate data between two or more servers or databases within a large or medium enterprise/network, without moving data to the cloud.
These solutions are optimal if you are performing tasks like changing data warehouses or moving the location of your primary data store.
Some companies prefer on-premise solutions due to security restrictions.
Open source is software you can use, modify, and share because its design is publicly accessible.
Typically, open source solutions are free or lower in cost than commercial alternatives.
Open source data migration tools can be a practical option for migrating your data if the project is not too large or complex.
Cloud-based data migration solutions are the latest generation and are designed to move data to the cloud.
Cloud-based solutions are optimal if you are already storing your data in the cloud or if you intend to move your data to the cloud.
Many organisations are now going for cloud based solutions considering cost efficiencies and enhanced security in moving data from on-premise to the cloud.
Moreover, cloud-based data migration tools tend to be flexible about the types of data they can handle.
Today, there are plenty of tools to facilitate enterprise data migrations. Generally, Data migration strategy determines which tools work best for the specific project.
Some popular Data Migration tools include the following:
• Rsync for host-based file-level migration
• Veeam to speed and simplify the process of migrating VM-based workloads.
• EMC Rainfinity for network-based file-level migration.
• Zerto offers a converged disaster recovery, backup, and workload mobility platform.
• HDS Universal Replicator for array-based block-level migration
• Rclone, an open source command line utility, used to migrate data to and from cloud object storage.
• Brocade DMM for network-based block-level migration
• Cyberduck is an open source FTP and SFTP client, used to transfer individual files or file volumes.
• Volume managers for host-based block-level migration
• Windows 10 Migration Accelerator
• Zinstall
• Acronis True Image
• Minitool Partition Wizard Free
• Laplink PCmover
• Samsung Data Migration Tool
• AOMEI Backupper Standard
• EaseUS Todo Backup/PC Trans Free
• Paragon Drive Copy Professional
• Samsung Smart Switch
Conclusion
