Cost control and warehouse scaling

Snowflake · Expert · 16 min

Snowflake bills by consumption, so cost engineering is a core skill for anyone running it at scale.

Cost in Snowflake is compute credits, storage and serverless features (Snowpipe, clustering, tasks, search optimization). Compute usually dominates, so focus there.

Multi-cluster warehouses (Enterprise edition) add clusters when queries queue: MIN_CLUSTER_COUNT = 1, MAX_CLUSTER_COUNT = 4. SCALING_POLICY = STANDARD starts clusters quickly to avoid queuing; ECONOMY waits until there's enough load to keep a new cluster busy, saving credits at the price of some queuing. Scale up (size) for slow individual queries, out (clusters) for concurrency.

Resource monitors cap credits per warehouse or account per period. They can notify at a threshold and suspend warehouses at the limit, either letting running queries finish or killing them immediately.

Query Acceleration Service offloads parts of large scans to serverless compute for outlier queries, without resizing the warehouse. Statement timeouts (STATEMENT_TIMEOUT_IN_SECONDS) kill runaway queries.

Measure, then act:

  • SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY gives credits per warehouse per hour.
  • QUERY_HISTORY (and QUERY_ATTRIBUTION_HISTORY for per-query credits) finds the expensive patterns.
  • ACCOUNT_USAGE views lag real time by up to a few hours.

Typical wins: right-size warehouses and auto-suspend, consolidate idle warehouses, fix queries that scan entire tables, use transient tables for reloadable data, and shorten Time Travel on churny staging tables.

Example

ALTER WAREHOUSE bi_wh SET
  MIN_CLUSTER_COUNT = 1
  MAX_CLUSTER_COUNT = 4
  SCALING_POLICY = 'ECONOMY'
  STATEMENT_TIMEOUT_IN_SECONDS = 1800;

USE ROLE ACCOUNTADMIN;
CREATE RESOURCE MONITOR bi_monthly WITH
  CREDIT_QUOTA = 500
  FREQUENCY = MONTHLY
  START_TIMESTAMP = IMMEDIATELY
  TRIGGERS ON 80 PERCENT DO NOTIFY
           ON 100 PERCENT DO SUSPEND;
ALTER WAREHOUSE bi_wh SET RESOURCE_MONITOR = bi_monthly;

-- Credits per warehouse over the last 30 days
SELECT warehouse_name, ROUND(SUM(credits_used), 1) AS credits
FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY
WHERE start_time >= DATEADD(day, -30, CURRENT_TIMESTAMP())
GROUP BY warehouse_name
ORDER BY credits DESC;

Remember

  • Scale up for slow queries, out (multi-cluster) for concurrency
  • Resource monitors notify and suspend at credit thresholds
  • Measure with ACCOUNT_USAGE metering and query history before tuning

Watch out for

  • Raising warehouse size to fix queuing, which doubles cost without adding concurrency.

Check yourself

  1. What does SCALING_POLICY = 'ECONOMY' trade off?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.snowflake.com/en/user-guide/warehouses-multicluster