TopolearnBulk downloads and quick load checks become one efficient command instead of a slow shell loop.
curl expands globs in URLs itself (quote them so your shell does not):
- Ranges:
[1-100], with leading zeros[001-100], steps[0-100:10], letters[a-z]. - Lists:
{alpha,beta,gamma}. - Several globs multiply:
{eu,us}/day[1-7]is 14 URLs.
When saving, #1, #2... in the -o name refer to the current value of the first, second glob: -o 'page_#1.json'. With -O, the remote names are used.
By default curl transfers the URLs one after another, but it reuses connections, so even serial runs are much faster than a shell loop starting a new curl (and a new TLS handshake) per URL.
-Z / --parallel runs transfers concurrently, up to 50 at a time by default (--parallel-max changes that). With HTTP/2 several of them can even share one connection. Output order is not guaranteed in parallel mode, so write each URL to its own file.
Use -g / --globoff when a URL legitimately contains [ ] or { }, such as IPv6 addresses or some query syntaxes.
Several plain URLs also work: curl -O url1 -O url2; each URL needs its own -o/-O, or use --remote-name-all. Combine with --fail and --retry for robust bulk jobs, and respect the target's rate limits.
Example
curl -sS -o 'page_#1.json' 'https://api.example.com/items?page=[1-5]'
ls page_* # page_1.json ... page_5.json
curl -sS -o '#1_#2.csv' 'https://data.example.com/{eu,us}/day[1-7].csv' # 14 files
curl -Z --parallel-max 10 -fsS --remote-name-all \
https://dl.example.com/part[001-120].bin
curl -g 'http://[::1]:8080/health' # globbing off for IPv6 literal
curl -sS -w '%{url} %{http_code} %{time_total}\n' \
-o /dev/null https://example.com/ -o /dev/null https://example.com/aboutRemember
- [1-10], [a-z], {a,b} globs; #1 in -o names the output
- One curl reuses connections across URLs; -Z adds concurrency
- -g disables globbing for URLs with brackets
Watch out for
- Forgetting to quote a globbed URL, so the shell expands or rejects the brackets and braces.