12 November 2026 · VRAMfarm team

Move datasets to a GPU server and back: Hugging Face, rclone, rsync, scp

Uploading an 80 GB dataset from home over a 40 Mbit/s connection takes about four and a half hours, and if a GPU server is already running, you pay for every one of them. The same 80 GB pulled from Hugging Face or an S3 bucket onto a machine with a 1 Gbit/s link takes 15–20 minutes. So the rule is simple: data should travel from the cloud to the server, not from your laptop, and the slow upload, if you need one, should happen once, without a GPU waiting.

Pick the route by where the data is now

The machine’s download and upload speeds are shown for each machine before launch, in Mbit/s, and the catalogue can sort and filter by them. Divide to estimate: 100 GB is 800 gigabits, so at 1,000 Mbit/s it’s about 13 minutes at full speed, and 15–25 minutes in practice. At a home upload of 40 Mbit/s, the same 100 GB takes five and a half hours.

If your data lives only on your computer and you’ll use it more than once, upload it once to a private Hugging Face dataset or your own bucket, overnight, with no GPU running. After that, every server pulls it at data-centre speed.

Hugging Face: datasets and models

The hf command comes with the huggingface_hub package:

pip install -U huggingface_hub
hf auth login                      # only for private or gated repos; paste a read token
hf download <user>/<dataset> --repo-type dataset --local-dir /workspace/data

Add --include "train/*.parquet" to fetch only part of a repo. Recent versions of huggingface_hub download files stored on Hugging Face’s Xet storage in many parallel chunks on their own, so a single hf download usually fills a fast link. Downloading with a token also avoids the stricter limits on anonymous requests.

Where the files land matters for your next session. Anything in /workspace can be saved with Save and stop; anything in ~/.cache/huggingface is gone after the stop. If you want the Hugging Face cache in the saved folder, set export HF_HOME=/workspace/hf before downloading. If you don’t (public data you can fetch again in minutes), leave it outside and pay no storage for it.

To send results back, the same tool uploads:

hf upload <user>/<results-repo> /workspace/outputs --repo-type dataset --private

Use a write token for this, and revoke it when the project is done.

rclone from S3, R2 or Google Drive

rclone speaks to almost every storage service. Install it on the server (the install script needs unzip, which slim images often lack):

apt-get update && apt-get install -y unzip curl
curl https://rclone.org/install.sh | bash

For S3-compatible storage you can skip the config file and pass the settings as environment variables, which keeps your keys out of /workspace. The remote name in the variables (S3 here) becomes s3: in the command:

export RCLONE_CONFIG_S3_TYPE=s3
export RCLONE_CONFIG_S3_PROVIDER=AWS
export RCLONE_CONFIG_S3_REGION=eu-central-1
export RCLONE_CONFIG_S3_ACCESS_KEY_ID=...
export RCLONE_CONFIG_S3_SECRET_ACCESS_KEY=...

rclone copy s3:my-bucket/dataset /workspace/data --transfers 16 --progress

For Cloudflare R2 or Backblaze B2, set PROVIDER to Cloudflare or Other and add RCLONE_CONFIG_S3_ENDPOINT with the endpoint from your account. --transfers 16 copies 16 files at once, which helps a lot with many medium-sized files. Results go back the same way:

rclone copy /workspace/outputs s3:my-bucket/runs/2026-11-12 --progress

Google Drive needs a browser login, and the server has no browser. Run rclone config, add a remote of type drive, choose the read-only scope if you only need to download, and answer n when asked about auto config. rclone prints an rclone authorize "drive" ... command. Run it on your own computer (with rclone installed there), log in, and paste the token it prints back into the server. Drive is slower than a bucket and has daily download quotas, so for big or repeated transfers, a bucket is the better home.

Give the server the narrowest key that works: read-only, one bucket, deleted after the project. The reasons are in is my data safe on a rented GPU.

From your computer: scp, rsync and the Jupyter upload

scp copies files over the same SSH key you use to log in. Take the port from My servers:

scp -P 31001 data.tar root@vramfarm.com:/workspace/
scp -P 31001 root@vramfarm.com:/workspace/outputs.tar .

rsync is better for anything large: it shows progress, skips files that are already there, and when the connection drops, running the same command again continues where it stopped. The server images don’t always include it, so install it there first:

# on the server
apt-get update && apt-get install -y rsync

# on your computer (Mac, Linux or WSL)
rsync -avP -e "ssh -p 31001" ./data/ root@vramfarm.com:/workspace/data/
rsync -avP -e "ssh -p 31001" root@vramfarm.com:/workspace/outputs/ ./outputs/

On Windows, scp works in PowerShell and rsync runs inside WSL; the Windows SSH guide covers both, plus WinSCP if you prefer a window.

The Jupyter upload (drag files onto the file browser) is the easiest and fine for scripts, notebooks, a LoRA or a few hundred megabytes of images. For several gigabytes it gets fragile: the browser tab has to stay open, an interrupted upload starts over, and dropping 10,000 files at once is slow. Download works with right-click → Download, one file at a time, so pack a folder into an archive first.

Pack it, and compress only what shrinks

Many small files are the slowest thing you can move. A million 100 KB images copy far slower than one 100 GB archive, because each file costs a round trip. Pack them first, or store the dataset in shards (tar, Parquet, WebDataset) to begin with:

tar cf images.tar images/           # no compression: JPEGs don't shrink
tar czf logs.tgz logs/              # text, CSV and JSON shrink 3–10 times

Images, video, audio, model weights and Parquet are already compressed, so gzip only costs CPU time. Text compresses well. If zstd is installed (apt-get install -y zstd), tar --zstd -cf data.tar.zst data/ packs and unpacks several times faster than gzip at a similar size.

Unpack while downloading when the archive sits behind a URL, so it never takes disk space twice:

curl -L "https://example.com/dataset.tar.gz" | tar xz -C /workspace/data

Check the disk before you start. You choose the disk size at launch, and an archive plus its unpacked copy needs twice the space for a while. If you hit “No space left on device”:

df -h /workspace                        # how full the disk is
du -sh /workspace/* | sort -h           # what takes the space
rm -rf ~/.local/share/Trash/*           # files deleted in Jupyter go here first

That last line catches many people: files you delete in the Jupyter file browser are moved to a trash folder and keep using disk until you empty it. Depending on the setup, the trash can also be a hidden .Trash-0 folder at the top of /workspace; ls -a /workspace shows it, and it would be included in a saved folder, so empty it too.

What it costs, and where the data should live

Traffic is billed per GB in both directions at the machine’s own rate, shown in the catalogue before you launch. Most machines charge around a cent per GB or less, some charge nothing, and some B200 and H200 machines charge more. A 100 GB dataset in and 5 GB of results out, at a cent per GB, comes to about €1.05.

The bigger cost is usually the GPU waiting. Twenty minutes of download on an H100 costs €1.30; the same twenty minutes on an RTX 4090 costs €0.20. So start the download first thing, and use the waiting time to upload your scripts or check the config.

Should the dataset live in your saved folder? Storage costs €0.03 per GB per month, so 100 GB is 100 × €0.03 a month. Restoring the folder on a new server also counts as traffic, and it isn’t always faster than Hugging Face or a bucket in a good region. In practice:

The full keep-or-download calculation, with numbers, is in what happens to your files when you stop a GPU.

Before you stop, run one last rsync or rclone copy of your outputs, or click Save and stop. The server disk is deleted with the server, and a dataset you can download again in 15 minutes is easy to replace. A week of results is not.