Overcome Data Migration Hurdles
We overcome data migration hurdles when we plan around them. Recently, we migrated a client’s 20 terabyte SQL server on an on-prem database running an old operating system. We loved the challenge of the task and each problem we solved. We learned a few lessons along the way, and we’d like to share them with you. This article examines possible hurdles you might face when migrating an overextended SQL server with legacy schema to Azure, and which data migration method might serve your organization’s needs.
Azure Site Recovery
Azure provides a number of technologies around site recovery. Those technologies can also be used to trick your system into a behind-the-scenes migration to the cloud. Typically, this Azure Site Recovery is used to create a backup in case of a failover. However, you can set up an existing on-prem datacenter to backup to the Azure datacenter. So, not only is the data backed-up, but it’s now on the cloud (and ready to make use of other Azure services). Beware though: some data centers are running on older operating systems that won’t support Azure Site Recovery.
Physical Migration with Data Box
Physical migration is like using a giant USB stick called a data box. In this scenario, the data box is physically hooked up to the data center, and all the SQL data files are transferred. The device is then taken to a Microsoft server facility where it is hooked up, and all the data is downloaded to several storage accounts. Keep in mind: the data center and Microsoft server facility would need to be relatively close in proximity to one another and you will need to migrate any new data accumulated from your backup point.
Replication
Replication is another method that can be used for data migration. Replication copies and distributes data and database objects from one database to another. It then synchronizes between databases to maintain consistency. However, this method can take a long period of time because it is restricted by the bandwidth available. Relying on a relatively slow bandwidth to migrate 20 terabytes of data could take a month for the system to sync up.
Replication can be use to migrate any data that may have accumulated while transporting the data box. Since the data box would have all the data up to a specific backup point, replication could be used to synchronize, and therefore, migrate the remaining data. However, when replication is used, the data schema in your product needs to be conducive to replication. That means, tables need to use a primary key, which uniquely identifies each row/record in a database table, and Azure backups need to be stored in standard SQL backup format. Even with these things in place, using replication still might not be an option if there is legacy technical debt in your schema.
Azure ExpressRoute
Azure ExpressRoute lets you connect your on-prem networks to Azure over a private connection. Since the connections don’t go over the public Internet, this offers more reliability, faster speeds, consistent latencies, and higher security. Another data migration hurdle is having a lot of data to migrate and a small window to do it in (e.g. 48 hours over the weekend). Having a faster network speed is crucial in this scenario. Watch out for bottlenecks! Read on to plan ahead.
To avoid a bottleneck, you will need to find a balance between your network, VM, and disk speeds. Here are a few things you’ll want to consider:
- Is your VM storage optimized? Storage optimized VM sizes offer high disk throughput and input-output speeds. This is ideal for Big Data, SQL, NoSQL databases, data warehousing, and large transactional databases.
- Is your VM memory optimized? Memory optimized VM sizes offer a high memory-to-CPU ratio.
- Do you have the right disk size? The wrong disk size can limit your speed because it won’t have the throughput needed.
- Are you copying from on-prem disks to storage accounts on Azure or to managed disks? If so, you’ll need to use a copy tool like AZCopy. However, depending on what you’re copying from and to, there might not be a commercially available tool.
Overcoming data migration hurdles can quickly get quite complex. Leveraging the help of Azure experts can save you time and keep you on budget. Contact us to schedule a complimentary discovery session with one of our solution architects.
