Architecture

This page is a detailed look at the components introduced in How Icebreaker is delivered. Read that page first if you have not.

Hosted by Icebreaker as multi-tenant SaaS:

• Account and identity. Customer accounts, users, and roles. You sign in here to manage your environment.
• Agent and instance registry. The authoritative record of which Icebreaker Agents are connected, which Data Servers should be running, and which Capacity Policies and catalogs each Icebreaker Agent should manage. Every configuration change in the admin UI is a write to this registry.
• Job monitoring. Aggregated query status across all of your agents.
• Execution metrics. Per-query timing for each stage: planning, scheduling, resource provisioning, and execution.
• Application management. The surface where you configure catalogs, ingress, Capacity Policies, and other operational settings.
• Billing and metering. Usage data that feeds your invoice.

Installed and reconciled by the Icebreaker Agent inside your EKS cluster:

The query entry point — an Arrow Flight SQL endpoint that receives SQL from clients and dispatches ephemeral query executor pods. Each Data Server is associated with exactly one Data Catalog and one Capacity Policy. You create Data Servers in the Icebreaker admin UI; one Icebreaker Agent can have multiple.

A Data Server is a Kubernetes Deployment containing:

• Logical Planner — Parses SQL and reads metadata from your Iceberg catalog. Produces a logical plan.
• Job Manager — Creates a query executor pod for each query, attaches the plan to the pod definition, monitors execution, and reports metrics back to the control plane.
• Results Reader — Streams Parquet results from S3 back to the SQL client.

Data Servers speak Arrow Flight SQL. Any JDBC tool that supports Arrow Flight SQL can connect.

One ephemeral pod per query. Each pod runs the DataFusion query engine and is sized to the query’s expected CPU and memory needs. It reads source partitions directly from S3 and writes Parquet results to a results bucket. When the query finishes, the pod terminates.

Query executor pods are placed in capacity-policy-specific namespaces. They never share state with other queries.

Coordinates pod placement and node provisioning. Built on Karpenter. When the Job Manager creates a query executor pod that does not fit on existing capacity, the Resource Manager provisions a new EC2 node based on the Capacity Policy’s preferences — instance family, on-demand vs. Reserved vs. Spot.

A Capacity Policy is your declared slice of capacity: which EC2 instance families to use, which capacity types (on-demand, Reserved, Spot) to prefer, what quotas to enforce. Each Capacity Policy maps to a Kubernetes namespace and a Karpenter NodePool in your cluster.

You define Capacity Policies in the admin UI; the Icebreaker Agent creates the corresponding namespace and NodePool in your cluster.

The catalog stays in your control, alongside the data it describes. Icebreaker reads table metadata from it at query planning time; you decide where the catalog runs, who can write to it, and how it authenticates. See Catalog integration for supported catalogs and configuration.

Icebreaker reads from a three-layer storage stack: Apache Iceberg as the table format, Parquet as the file format, S3 as the object store. The Iceberg catalog tells Icebreaker which Parquet files in S3 belong to each table snapshot; query executor pods read those Parquet files directly.

Query results are written back to S3 as Parquet, optionally registered in your catalog as a result table.

1. A SQL client connects to a Data Server over Arrow Flight SQL.
2. The Logical Planner reads metadata from your Iceberg catalog and produces a logical plan.
3. The Job Manager creates a query executor pod in the appropriate capacity-policy namespace, attaching the plan.
4. The Resource Manager schedules the pod onto an existing node, or provisions a new one matching the Capacity Policy’s preferences.
5. The query executor pod reads source partitions from S3 and writes Parquet results to your results bucket.
6. The Results Reader streams results to the SQL client.
7. Per-stage timing is reported to the Icebreaker control plane.