Extensions, S3 and other databases

DuckDB · Advanced · 14 min

Extensions let DuckDB query data where it lives, which removes whole export and import steps from pipelines.

Much of DuckDB's functionality ships as extensions: INSTALL name; LOAD name;. Core extensions are signed, downloaded from the official repository and cached locally; many (like httpfs, json, parquet) autoload the first time you use a feature that needs them. SELECT * FROM duckdb_extensions() lists what is installed and loaded.

The most used ones:

  • httpfs: read and write https://, s3://, gcs:// and Azure paths. Configure credentials with the secrets manager: CREATE SECRET (TYPE s3, PROVIDER credential_chain) picks up your AWS environment, or give KEY_ID/SECRET/REGION explicitly. Add PERSISTENT to store it across sessions.
  • postgres, mysql, sqlite: ATTACH 'dbname=shop host=...' AS pg (TYPE postgres) exposes the remote tables; you can join them with local Parquet and even write back.
  • iceberg and delta: read lakehouse table formats.
  • spatial (geometry types and GIS functions), fts (full-text search), excel, vss (vector similarity index).

Community extensions are installed with INSTALL name FROM community. Because extensions are native code, only load ones you trust; settings like allow_community_extensions and autoinstall_known_extensions let you lock this down in production.

Example

-- read Parquet straight from S3 using your AWS credential chain
INSTALL httpfs; LOAD httpfs;
CREATE SECRET aws (TYPE s3, PROVIDER credential_chain);

SELECT count(*) FROM 's3://my-bucket/events/2025/*.parquet';

-- join a live PostgreSQL table with a local file
INSTALL postgres; LOAD postgres;
ATTACH 'dbname=shop host=localhost user=app' AS pg (TYPE postgres, READ_ONLY);

SELECT c.country, sum(e.amount) AS total
FROM 'events.parquet' e
JOIN pg.public.customers c ON c.id = e.customer_id
GROUP BY ALL;

Remember

  • INSTALL downloads once; LOAD activates; many core extensions autoload
  • CREATE SECRET manages S3/GCS/Azure credentials
  • ATTACH ... (TYPE postgres|mysql|sqlite) queries other databases in place

Watch out for

  • Pulling an entire remote PostgreSQL table into DuckDB when a filter could have been pushed to the source; check EXPLAIN.

Check yourself

  1. What is the role of CREATE SECRET when reading s3:// paths?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: duckdb.org/docs/current/extensions/overview

Builds on

Unlocks

Nothing builds on this yet.