TopolearnYou can't operate what you can't see; CloudWatch is the default place AWS services report their health.
CloudWatch metrics are time series identified by a namespace (AWS/EC2, AWS/ApplicationELB, or your own), a name and dimensions (such as InstanceId). AWS services publish metrics automatically: EC2 every 5 minutes (1 minute with detailed monitoring), most others every minute. EC2 doesn't report memory or disk usage from inside the OS; install the CloudWatch agent for that. Publish your own business metrics with PutMetricData or, more cheaply, the Embedded Metric Format in logs.
Alarms watch a metric statistic over periods and move between OK, ALARM and INSUFFICIENT_DATA. Use "M out of N" datapoints to avoid flapping, choose how missing data is treated, and trigger actions: an SNS topic for paging, Auto Scaling policies, or EC2 recovery. Composite alarms combine others to reduce noise. Alarm on symptoms users feel (5xx rate, latency, queue age) rather than every cause.
CloudWatch Logs organises logs into log groups (per application) and streams (per source). Lambda, ECS with awslogs, API Gateway and many services write here directly. Set a retention policy on every group; the default is to keep logs forever and you pay for that.
Logs Insights queries log groups with a pipe-based language (fields, filter, stats, sort, parse), ideal during incidents. Metric filters turn log patterns into metrics you can alarm on, and subscription filters stream logs to Lambda, Firehose or OpenSearch. Dashboards put the key graphs for a service in one place.
Example
# Page when the ALB's targets return more than 10 5xx per minute, 3 of 5 minutes
aws cloudwatch put-metric-alarm --alarm-name orders-api-5xx \
--namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=app/web-alb/0123456789abcdef \
--statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
--threshold 10 --comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall
aws logs put-retention-policy --log-group-name /ecs/orders-api --retention-in-days 30
# Errors per 5 minutes over the last hour
QID=$(aws logs start-query --log-group-name /ecs/orders-api \
--start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
--query-string 'fields @timestamp, @message
| filter @message like /ERROR/ | stats count() as errors by bin(5m)' \
--query queryId --output text)
sleep 5; aws logs get-query-results --query-id "$QID"
aws cloudwatch put-metric-data --namespace Shop --metric-name OrdersPlaced \
--dimensions Env=prod --value 1 --unit CountRemember
- Metrics = namespace + name + dimensions; EC2 memory needs the agent
- Alarm on user-facing symptoms with M-of-N datapoints
- Set log retention on every log group
- Logs Insights for ad-hoc queries; metric filters to alarm on log patterns
Watch out for
- Alarming on every CPU spike, training the team to ignore pages.