Manage Capacity Policies
A Capacity Policy defines the EC2 capacity envelope for a class of workloads: which instance families Karpenter can provision, whether to prefer Spot or On-Demand, how much total capacity to allow, and how to prioritize competing queries. Each Capacity Policy maps to a Karpenter NodePool in your cluster — Icebreaker creates and manages these NodePools for you.
You configure Capacity Policies in the Icebreaker admin UI under Icebreaker Agents → Capacity Policies.
Recommended starting configuration
Section titled “Recommended starting configuration”For most deployments — including production — one Capacity Policy with these settings covers all workloads:
| Setting | Recommended value |
|---|---|
| Instance family | c7a (AMD compute-optimized) |
| Capacity type | Spot |
| Catalog | Lakekeeper |
The sections below explain why. The catalog is configured separately from the Capacity Policy — see the link above for setup.
Spot vs On-Demand capacity
Section titled “Spot vs On-Demand capacity”Use Spot. Icebreaker’s architecture makes Spot safe for production workloads in a way that shared-cluster query engines cannot match, and the cost difference is substantial.
Why Spot interruptions are non-events
Section titled “Why Spot interruptions are non-events”In a traditional shared query cluster — a Spark cluster, a shared warehouse — a Spot interruption takes down the cluster. All in-flight queries fail. Users see errors until the cluster recovers, because the cluster holds query state, in-progress results, and cached data in shared memory. The unit of failure is the cluster.
Icebreaker has no shared state. Every query runs in its own ephemeral pod, started fresh and destroyed when the query finishes. When a Spot node is reclaimed:
- Only the queries running on that node are affected — typically one or two.
- The Job Manager detects the interrupted queries and resubmits them automatically. The SQL client sees a brief delay, not an error.
- Every other query continues running. The fleet is unaffected.
The unit of interruption is a single query, not the cluster. This is a direct consequence of query isolation — there is no shared state to lose.
Cost impact
Section titled “Cost impact”Spot pricing for c7a instances typically runs 70–90% below On-Demand rates. Icebreaker compounds this with serverless elasticity: you are not paying for a cluster that sits idle between queries. Combined, most customers see 85%+ cost reduction compared to a static On-Demand cluster sized for the same workload.
On-Demand fallback
Section titled “On-Demand fallback”You can configure a Capacity Policy to fall back to On-Demand when Spot capacity is unavailable in your chosen availability zones. Enable this when:
- You have SLA commitments that cannot tolerate even brief queue delays during an AZ-level Spot interruption event.
- You are running on an instance family with historically elevated interruption rates.
For latency-sensitive interactive workloads, you can also create a dedicated Capacity Policy backed by a small pool of pre-warmed On-Demand capacity — while routing batch and exploratory queries to a separate Spot policy. See When to add a second Capacity Policy.
When On-Demand only makes sense
Section titled “When On-Demand only makes sense”- Regulated workloads where query retry must be explicitly logged or approved before resubmission.
- Dev/test environments where cost is secondary to absolute predictability.
Choosing instance families
Section titled “Choosing instance families”c7a (AMD compute-optimized) is the recommended default for SQL workloads:
- High memory bandwidth for scan-heavy queries on large Parquet files.
- Competitive Spot availability and low interruption rates.
- Broad AZ coverage makes Spot interruptions rare.
When to use other families:
| Use case | Instance family |
|---|---|
| Large aggregations or joins that spill to disk | r7a or r7i (memory-optimized) |
| GPU-accelerated ML inference in-query | g5 or p4d |
Spot fallback if c7a availability is low | c6a, c6i |
When using Spot, specifying multiple compatible instance families significantly reduces interruption risk — Karpenter sources capacity from whichever family has availability in a given AZ. Adding c6a and c6i as alternatives to c7a costs little and meaningfully improves resilience.
Quotas and concurrency limits
Section titled “Quotas and concurrency limits”Each Capacity Policy can cap the total compute it consumes at once. This is the primary mechanism for cost governance and tenant isolation.
| Control | Purpose |
|---|---|
| Max vCPU | Hard cap on total vCPU across all running pods in this policy |
| Max memory | Hard cap on total memory |
| Concurrency limit | Maximum number of simultaneously running queries |
| Priority | Queue order when multiple Capacity Policies compete for capacity |
When a query arrives and the policy is at its quota, it enters the queue. The Job Manager drains the queue as running queries complete. Queries are never dropped due to quota — they wait.
When to add a second Capacity Policy
Section titled “When to add a second Capacity Policy”Start with one. Add a second when a class of workloads has meaningfully different requirements from your default:
- Specialized hardware — ML or memory-intensive workloads that need instance types you do not want available to general queries.
- Tenant isolation — different quota envelopes for different customers, teams, or workload tiers. A premium customer gets more headroom than a free-tier user.
- Latency tiers — a small pre-warmed On-Demand policy for sub-second interactive queries alongside a Spot policy for batch.
- Regulatory separation — a policy pinned to specific availability zones for data residency or compliance requirements.
Each Capacity Policy maps to its own Karpenter NodePool and Kubernetes namespace. Pods across policies cannot share nodes or communicate across namespace boundaries.
Pre-warming
Section titled “Pre-warming”By default, Icebreaker provisions nodes on demand as queries arrive. Query executor pod cold start is sub-second, but initial node provisioning typically adds 30–90 seconds to the first query after a period of inactivity.
If you have interactive SLA requirements for queries that arrive after an idle period, configure a Capacity Policy to keep a minimum number of nodes warm. Pre-warmed nodes run On-Demand and incur cost whether or not they are serving queries.