How to Send Large Datasets to Collaborators Without Losing a Weekend

२०२६ अगस्ट १९

3 min read

Archive first, check your upload speed, use an expiring link, and collect data back with vouchers. A practical workflow for moving research-scale datasets between institutions.

To send a large dataset to a collaborator: archive it into a single file, upload it to a large-file transfer service, set an expiry date and a password, and send the link. Do not split it across emails, and do not hand out cloud storage credentials. The whole process is four steps and the upload runs unattended.

1. Archive before you upload

A dataset of 40,000 small files and a single 20 GB archive contain the same bytes, but they do not transfer at the same speed. Every file pays fixed overhead on both ends — a request, a response, a filesystem entry. Bundling into one .zip or .tar removes that overhead entirely and usually cuts transfer time dramatically.

It also preserves your directory structure, which matters more than people expect when the recipient is trying to reproduce your results.

2. Check your upload speed, not your download speed

Most consumer and many institutional connections are asymmetric. A line advertised as 100 Mbps may only offer 10 Mbps upstream, which is about 4.5 GB per hour. Before you promise a colleague the data by lunchtime, run a speed test and look at the upload number.

Rough arithmetic:

10 Mbps up — ~4.5 GB/hour
50 Mbps up — ~22 GB/hour
100 Mbps up — ~45 GB/hour
1 Gbps up — ~450 GB/hour

No transfer service can exceed your own uplink. What a good one does is make sure you reach it, and that an interruption does not send you back to zero.

3. Set an expiry and a password

Unpublished research data should not sit on a permanent public link. Set the expiry to the shortest window that is realistic for the other party, and add a password if the data is sensitive or subject to an ethics agreement. Send the password through a different channel than the link — a message, not the same email.

4. Collect data back with a voucher

Collaboration is bidirectional, and the usual friction is that the other institution will not create an account on a service they did not choose. A voucher is a link you send them that lets them upload directly into your account without registering. You set the limit and the expiry; you are notified when the upload lands.

A note on interrupted uploads

Multi-hour uploads get interrupted. Laptops sleep, VPNs reconnect, campus Wi-Fi drops. A transfer service that uploads in chunks and tracks which ones completed will resume from the last finished chunk. One that does not will start the file over, and on a 200 GB dataset that is the difference between finishing and giving up.