TopolearnLarge files break the default load-everything behaviour, and uploads have their own body formats.
By default Requests downloads the entire body into memory before returning. For large files pass stream=True: only the headers are read, and you consume the body in pieces with r.iter_content(chunk_size=...) (bytes) or r.iter_lines() (for line-based streams such as server-sent events or NDJSON).
With streaming, the connection is held until the body is fully read or the response is closed. Use the response as a context manager, with s.get(url, stream=True) as r:, so the connection returns to the pool even if you stop early. iter_content transparently decompresses gzip responses; r.raw gives the undecoded urllib3 stream if you really need raw bytes.
For uploads, there are two shapes:
- Multipart form uploads, as browsers do:
files={"file": open("report.pdf", "rb")}or a tuple("name.pdf", fileobj, "application/pdf")to set the filename and content type. You can combinefiles=withdata=for extra form fields. Open files in binary mode. - Streaming the body: pass a file object or a generator as
data=. A file is streamed from disk without reading it into memory. A generator sends the body with chunked transfer encoding.
Note that multipart encoding with files= builds the whole body in memory. For very large multipart uploads, the requests-toolbelt package provides a streaming MultipartEncoder.
Example
import requests
with requests.Session() as s:
with s.get("https://httpbin.org/bytes/100000", stream=True, timeout=10) as r:
r.raise_for_status()
size = 0
with open("download.bin", "wb") as f:
for chunk in r.iter_content(chunk_size=16_384):
f.write(chunk)
size += len(chunk)
print(size) # 100000
with open("download.bin", "rb") as f:
r = s.post("https://httpbin.org/post",
files={"upload": ("data.bin", f, "application/octet-stream")},
data={"note": "test"}, timeout=10)
echo = r.json()
print(echo["form"], list(echo["files"])) # {'note': 'test'} ['upload']
print(echo["headers"]["Content-Type"][:30]) # multipart/form-data; boundary=Remember
- stream=True plus iter_content downloads in chunks
- Use the streamed response as a context manager to release the connection
- files= sends multipart; a file object or generator as data= streams the body
Watch out for
- Opening upload files in text mode instead of 'rb'.