Moving big data to the cloud – what you need to know

Lawrence Jones

Thursday 25 June 2015

Banks, government organisations and large multi-nationals are just some of those with massive data stores making the move to the cloud. They are all attracted by the easy access to data, secure hosting and reduced costs.

Small data moves are relatively simple – in most cases it’s no different from dragging and dropping. But when organisations decide on moving everything, things can get complicated.

Here are some ways you can avoid some of classic cloud data transfer problems.

Knowing the risks

Your data needs to be moved securely. The only way to do so is to understand and categorise the risks you might face. The Cloud Security Alliance’s Consensus Assessment is an industry standard benchmark for those embarking on a data migration. Working through the free questionnaire is a great way to identify any potential security concerns you might have.

Done correctly, it can help you mitigate against most potential issues. It might be the most valuable few hours you spend in your whole transfer process.

Planning for success

Once you know where the problems are, you can create a plan to tackle them. A comprehensive, large scale data move doesn’t happen overnight (although, ironically, that may be the best time to actually move the data). A migration plan is therefore essential if you’re to move big data without disrupting business.

Within your plan, you need flexibility. Any data move could take days, or even weeks, so your plan needs to build in contingencies in case of business emergency or the dreaded project creep.

Moving terabytes or even petabytes of data to the cloud is going to put a strain on your bandwidth, which could have unforeseen consequences. A 100Mbit per second connection should theoretically enable you to move 1Tb per day. It’s simple mathematics, but it doesn’t take into account other users, emergencies or network downtime.

Break down the transfer task into individual chunks, keeping the existing infrastructure live for as long as possible. It’s not about creating and maintaining two systems, it’s about keeping your business operational and productive throughout the transfer.

Security is a concern for any business, with the cloud still – to some minds, at least – representing a risk. Your data is likely to be encrypted in the cloud, but what about the transfer process? Industry best practise is to encrypt the data before sending, storing it in the object format stored in the cloud. This removes at least some of the concerns you and your organisation might have.

Managing the move

Moving the data is one challenge; managing it and using it is another. Data migrations demand new skills of users in accessing the data in new locations and through new software. Any migration is also likely to challenge the skills of even the most experienced of IT teams.

Businesses are increasingly tempted to outsource cloud solutions and, while tempting, there will always need to be in-house advice and guidance – so training before and after a migration is a sound investment whatever transfer model you choose.

There’s no simple way to move big data to the cloud, but the benefits of doing so are evident. Take it slowly, plan accordingly and expect the unexpected.

Building the next-gen data centre

Where traditional and web-scale apps co-exist