level: technical
hugging face and skypilot now let you mount hugging face storage directly into compute jobs on any cloud. you specify a bucket or repo with an hf:// url and your hf_token, and skypilot runs the job on available gpus across over 20 clouds, kubernetes, or on-prem clusters. the mount uses a fuse filesystem that lazily reads only the data your code touches, keeping a local cache so repeated reads stay fast. this means your models and datasets live on the hub while compute moves to wherever gpus are free.
the key benefit is zero egress fees for reading data. hugging face storage charges no egress or cdn fees, so pulling model weights or streaming datasets onto gpus on any cloud costs nothing. this removes the cross-cloud transfer tax that often forces teams to pin jobs to the cloud where their data sits, leaving other reserved capacity idle. writing checkpoints back still incurs your compute cloud's usual egress, but for most ai workloads reads dominate, making this a practical way to use multi-cloud gpu capacity without duplicating data.
storage is backed by xet, which uses content-defined chunking to deduplicate data. only changed chunks upload for incremental checkpoints or model variants, saving time and bandwidth. re-uploading an existing file skips already-stored chunks. the integration is open source and works with unprivileged containers, so it fits into existing kubernetes setups. to start, install skypilot with huggingface support, authenticate, and add an hf:// mount to your task yaml.
why it matters: it lets ai teams run training and inference on any available gpus without paying egress to read their data, making multi-cloud strategies cheaper and simpler.