TopolearnBig files (datasets, backups, model weights) need parallelism, resumability and progress reporting to be practical.
upload_file, upload_fileobj, download_file and download_fileobj use the S3 transfer manager (s3transfer). Above a size threshold it switches to multipart: the file is split into parts uploaded in parallel threads, each retried independently, then S3 assembles them. Downloads use parallel ranged GETs.
boto3.s3.transfer.TransferConfig tunes it:
multipart_threshold(default 8 MB): size at which multipart kicks in.multipart_chunksize(default 8 MB): part size. S3 allows at most 10,000 parts, so for very large objects raise it (a 1 TB file needs parts of at least ~105 MB).max_concurrency(default 10): parallel threads per transfer;use_threads=Falsedisables threading.
Pass Config=TransferConfig(...) and Callback=fn; the callback receives the number of bytes transferred since the last call, from worker threads, so make it thread-safe.
Operational notes:
- Failed multipart uploads leave incomplete parts that are billed until aborted. Add a lifecycle rule
AbortIncompleteMultipartUploadto every bucket. - Multipart objects get an ETag that isn't the file's MD5; use checksums (
ChecksumAlgorithm="SHA256"in ExtraArgs) for integrity checks. - Installing
boto3[crt]enables the AWS Common Runtime transfer client, which can be much faster on high-bandwidth machines.
Example
import os, threading
import boto3
from boto3.s3.transfer import TransferConfig
s3 = boto3.client("s3", region_name="us-east-1")
s3.create_bucket(Bucket="demo-big-2026")
with open("big.bin", "wb") as f:
f.write(os.urandom(30 * 1024 * 1024)) # 30 MB test file
class Progress:
def __init__(self, size):
self.size, self.seen, self.lock = size, 0, threading.Lock()
def __call__(self, nbytes): # called from worker threads
with self.lock:
self.seen += nbytes
cfg = TransferConfig(multipart_threshold=8 * 1024**2, multipart_chunksize=8 * 1024**2,
max_concurrency=4)
p = Progress(os.path.getsize("big.bin"))
s3.upload_file("big.bin", "demo-big-2026", "data/big.bin", Config=cfg, Callback=p,
ExtraArgs={"ChecksumAlgorithm": "SHA256"})
print(p.seen == p.size) # True
etag = s3.head_object(Bucket="demo-big-2026", Key="data/big.bin")["ETag"]
print(etag.endswith('-4"')) # True: 4 parts, not an MD5Remember
- upload_file/download_file use multipart and parallel threads automatically
- TransferConfig sets threshold, chunk size and concurrency
- Progress callbacks get byte deltas from worker threads
- Abort incomplete multipart uploads with a lifecycle rule
Watch out for
- Using the default 8 MB chunk size for multi-terabyte objects and hitting the 10,000-part limit.