Data Server internals

A Data Server is a Kubernetes Deployment that the Icebreaker Agent runs in your cluster. From the outside it looks like a single Arrow Flight SQL endpoint; internally it is composed of several cooperating components.

Accepts incoming gRPC connections from SQL clients (JDBC, gRPC direct). Authenticates the bearer token, routes each query to the Planner, and streams results back to the client once they are available.

Parses the incoming SQL, reads table and schema metadata from your Data Catalog, and produces a logical query plan. The Planner hands the plan off to the Job Manager for execution.

Dispatches a per-query executor pod into your cluster for each logical plan the Planner produces. The Job Manager embeds a result-channel address in the request it sends to the executor, then waits for the result notification.

Query executor pods are ephemeral — one per query, torn down when the query completes. Karpenter provisions EC2 nodes for them on demand, per the Capacity Policy associated with the Data Server.

Subscribes to the result channel the Job Manager embedded in each executor request. When the executor publishes a completed result, the collector picks it up and forwards it to the Arrow Flight SQL endpoint for streaming back to the client.

The result channel is backed by a Redis instance (colibri-redis) that the Icebreaker Agent co-deploys alongside the Data Server in your cluster.

SQL client
  → Arrow Flight SQL endpoint   (authenticate, accept query)
  → Planner                     (parse SQL, read catalog metadata, produce plan)
  → Job Manager                 (dispatch query executor pod; Karpenter provisions EC2 per Capacity Policy)
  → Query executor pod          (read S3, compute results, publish to result channel)
  → Result collector            (receive from result channel)
  → Arrow Flight SQL endpoint   (stream results to SQL client)

Timing for each stage is reported to the Icebreaker control plane and will be surfaced as performance metrics and cost attribution in the Admin console.