Sending Machine Learning Datasets and Model Checkpoints

To move a training set or a model checkpoint, archive it, upload it to a large-file transfer service, and share a password-protected link that expires when the job is done. It avoids provisioning a bucket, minting credentials, and remembering to revoke them afterwards.

Why object storage is not always the answer

A cloud bucket is the right tool when the data is being read repeatedly by infrastructure. It is a poor tool for handing a dataset to a person once. You have to create the bucket, write a policy, issue credentials scoped to it, explain to the other party how to install and configure a CLI, and then remember to tear all of it down. For a one-off handoff, a link is simply a better interface.

Checkpoints are worse than datasets

Large model checkpoints are single files in the tens or hundreds of gigabytes, already in a compressed binary format. They do not benefit from zipping, they cannot be split without extra tooling on the other end, and they are exactly the shape that breaks naive upload implementations. What matters here is chunked upload with resume — a single interrupted attempt on a 60 GB checkpoint should not cost you the whole upload.

Confidentiality

Training data frequently carries licensing terms, and checkpoints frequently embed commercially sensitive work. Two things are worth checking before you upload anywhere:

• Does the service train on your files? Several consumer transfer services have amended their terms to permit it. TeraSender does not use uploaded content to train models.
• When is the data deleted? A transfer with an expiry date is deleted on schedule. A file dropped in a permanent share stays until someone remembers it.

A workable pattern

1. Archive the dataset — one .tar, not 200,000 JPEGs.
2. Record a checksum before uploading, so the recipient can verify the file end-to-end.
3. Upload with an expiry that matches the length of the engagement.
4. Password protect it and send the password separately.
5. Enable download notifications so you know when it arrived.
6. Use a voucher to collect the trained artefacts back.

Related: big data transfer and security.