Storage Migration and Data Transfer Planning for the Cloud

Data is the heaviest thing you will move

Servers can be rebuilt, but data has gravity. A file server of many terabytes does not move over a weekend on an ordinary office connection, and a bucket of small files can crawl even on a fast link because of per-object overhead. Teams that leave storage planning to the end find that their cutover date is set by physics, not by the project plan.

Storage and data transfer planning is part of our cloud migration service. It feeds the roadmap with realistic durations and sits alongside server and database migration.

What good looks like

• Every dataset has an owner, a size, a change rate and a target.
• The transfer method was chosen from calculated durations, not habit.
• Copies run in the background with throttling, and production is unaffected.
• Integrity is verified and the evidence is kept.
• Permissions and access paths work for real users before cutover.
• Data that does not need to move has been identified and handled.

Do the sum first

The simplest planning tool is a calculation. A link of 1 Gbps does not move 1 Gbps of data in practice. Protocol overhead, shared use and throttling mean usable throughput is often a fraction of the nominal rate. Measure it with a test transfer or a tool such as iperf3 over the actual path, at the time of day you intend to run the copy.

# Rough duration in hours: size in TB, usable throughput in Mbit/s
python3 - <<'EOF'
size_tb = 40
usable_mbps = 400          # measured, not the line rate
seconds = (size_tb * 8 * 1_000_000) / usable_mbps
print(f"{seconds/3600:.1f} hours (~{seconds/86400:.1f} days)")
EOF

If the answer is longer than the project can tolerate, you have three levers: a faster or dedicated link (see network architecture), an offline appliance, or moving less data. The right choice is the one that fits the timeline and budget, and it is often a combination.

Online or offline

Offline appliances are not magic. You still need a plan for changes that occur while the device is in transit, usually an incremental online sync after the appliance data lands. For many mid-sized estates a combination works best: appliance for the bulk, online for the delta.

Choosing the right target

Moving storage is a chance to choose a better home. Files that applications read through paths may suit a managed file service. Unstructured data, backups and archives usually belong in object storage with lifecycle rules. Block volumes attached to rehosted servers move with those servers. We match each dataset to a service, and set storage classes and lifecycle policies up front so you do not pay hot-tier prices for cold data. This links to cost optimisation.

A simple lifecycle rule in Terraform for a bucket holding migrated archives:

resource "aws_s3_bucket_lifecycle_configuration" "archive" {
  bucket = aws_s3_bucket.migrated_archive.id

  rule {
    id     = "tier-down-old-objects"
    status = "Enabled"
    filter {}

    transition {
      days          = 90
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 365
      storage_class = "GLACIER"
    }
  }
}

Check retrieval times and minimum storage durations for each class before applying rules to data people actually read.

Verification and permissions

Two things are checked as carefully as the transfer itself. The first is integrity: checksums, object counts and size totals, ideally generated independently of the copy tool. The second is access. A file share copied perfectly but with broken permissions is a failed migration from the user’s point of view. We test with representative users and groups, with identity integration in place as described in IAM and directory integration.

Decisions and trade-offs

Move everything or filter first. Years of accumulated data often include duplicates and expired records. Cleaning first saves transfer time and storage cost, but it needs business sign-off and a retention policy, which can take longer than the copy. We offer both paths.

Hybrid access or full cut. Keeping a gateway lets on-premises users continue working through the transition. A hard cut is simpler to operate. Choose by user impact.

When not to move. Data subject to residency rules that the target region cannot satisfy, or data needing local low-latency access to equipment, should stay.

A worked example: a file server in three passes

For a departmental file server, the first pass copies everything in the background over a few evenings, throttled so users notice nothing. The second pass, a week later, copies only what changed and takes far less time. The third pass runs at cutover: the share is made read-only, the last changes are copied, and a comparison report is generated before users are pointed at the new location. The old server stays read-only for the agreed period in case anyone finds a missing folder.

Common failure modes

• Millions of tiny files, making throughput far lower than expected.
• Copy jobs saturating a link shared with production traffic.
• Timestamps or metadata changing, which breaks backup tools or sync logic.
• Encryption and key ownership decided late.
• No delta strategy, so cutover requires an extended outage.

Delivery and handover

You receive the transfer plan, the scripts or task definitions in code, verification reports and a decommissioning checklist for old storage. Retention and recovery for the new location follow our backup and disaster recovery approach.

How to tell it is working

Delta syncs get smaller each pass, verification reports show zero unexplained differences, users reach their files with the right permissions, and the final cutover window is minutes or hours rather than days. For help sizing your transfer, book a consultation.

References

• What is AWS DataSync?, AWS documentation
• Copy or move data to Azure Storage by using AzCopy v10, Microsoft Learn
• AWS Snowball Edge Developer Guide, AWS documentation
• Microsoft Azure Data Box overview, Microsoft Learn