Troubleshooting

It’s a good idea to review Prerequisites as a first troubleshooting step.

Issues here are organized by when they appear in the setup and query flow.

kubectl get nodes
# Unable to connect to the server: dial tcp: lookup <hex-id>.gr7.<region>.eks.amazonaws.com on 127.0.0.11:53: no such host

Cause: Your kubeconfig still points at a previous cluster’s API server endpoint. This is common right after uninstalling and recreating an EKS cluster rather than reusing the same one — the old endpoint hostname stops resolving once that cluster is deleted, but ~/.kube/config keeps the stale entry until you refresh it.

Remediation:

1. Confirm the cluster you intend to use actually exists and is ACTIVE:

   aws eks describe-cluster --name <your-cluster-name> --region <your-region> --query 'cluster.status'

   ResourceNotFoundException means the name or region doesn’t match anything in your account — check for typos or that you’re targeting the right region.
2. Refresh your kubeconfig against the current cluster:

   aws eks update-kubeconfig --region <your-region> --name <your-cluster-name>

3. Verify:

   kubectl get nodes

If it still fails and the cluster is confirmed ACTIVE, the problem is DNS resolution in your local environment rather than the cluster itself — see the CloudShell tip in Set up an EKS Auto Mode cluster to check from a known-good environment.

kubectl get pods -n eyrie-compute
# icebreaker-agent-eyrie-agent-* → ImagePullBackOff

Cause: Your cluster cannot pull the Icebreaker Agent image from Icebreaker’s private container registry.

Remediation:

1. Confirm with your Icebreaker contact that pull access has been granted to your AWS account ID.
2. If you received a pull secret instead of account-level access, verify imagePullSecrets is set in your Helm values — see Icebreaker Agent configuration.
3. If your cluster is in a private subnet with no internet egress, you need either a NAT gateway or to mirror the image into your own ECR — see IAM.

kubectl describe pod -n eyrie-compute <pod-name>
# Events: ... secret "icebreaker-agent-token" not found

Cause: The Kubernetes Secret containing the Agent token does not exist in the workload namespace, or the secret name in the Helm values does not match.

Remediation:

1. Verify the secret exists: kubectl get secret -n eyrie-compute
2. If it’s missing, re-create it: kubectl create secret generic icebreaker-agent-token --from-literal=token=<token> -n eyrie-compute
3. Confirm the Helm value matches the secret name: agent.existingSecret should equal the secret name you created.

After the Agent pod reaches Running status, the Icebreaker Agent in the Admin console should change its Last Seen value to Just now within ~30 seconds.

kubectl logs -n eyrie-compute -l app.kubernetes.io/instance=icebreaker-agent

Common causes:

After a Data Server is created in the Admin console, the Icebreaker Agent should deploy it within a few seconds. If the Data Server status stays at pending or moves to error:

kubectl get deployments -n eyrie-compute
kubectl describe deployment <data-server-deployment-name> -n eyrie-compute

The Icebreaker Agent has not yet reconciled the Data Server. Either the Agent is not connected (see above) or there is a delay. Check the Agent logs:

kubectl logs -n eyrie-compute -l app.kubernetes.io/instance=icebreaker-agent

The Deployment exists but has failed. Check the Deployment events and pod logs:

kubectl describe deployment <deployment-name> -n eyrie-compute
kubectl logs -n eyrie-compute -l <data-server-label>

Common causes: missing IRSA annotation (Data Server pod can’t assume the IAM role), missing colibri ServiceAccount (set rbac.createWorkloadRbac: true in Helm values), or the Data Server image is unavailable.

After a Data Server reaches running, the Connection URL in the Admin console should populate with an ELB hostname within ~30 seconds. If it stays “Not yet assigned” indefinitely, the Ingress that the Icebreaker Agent created has not received an ELB hostname from the AWS Load Balancer Controller. This is almost always one of three things, and they layer — fix them in order.

Start by looking at the Ingress:

kubectl get ingress -n eyrie-compute
kubectl describe ingress -n eyrie-compute <ingress-name>

What you find in ADDRESS and the Events section tells you which cause you’re hitting.

Cause: no IngressClass. The AWS Load Balancer Controller in EKS Auto Mode listens for IngressClasses pointing at eks.amazonaws.com/alb, and Auto Mode does not create one by default. Without an IngressClass, Ingresses are silently ignored.

Verify:

kubectl get ingressclass

If empty, follow Prepare ALB ingress — the AWS guide it links to walks through creating the IngressClass. The existing Ingress is reconciled retroactively — no need to recreate the Data Server.

Cause: VPC subnets are not tagged. The controller needs at least two subnets in different AZs tagged kubernetes.io/role/elb (internet-facing) or kubernetes.io/role/internal-elb (internal). Default and console-created VPCs typically lack these tags.

Follow Prepare ALB ingress — the AWS guide it links to walks through tagging subnets. The controller retries automatically — ADDRESS should populate within ~60s after tagging.

Cause: the ACM certificate is rejected by ALB. ALB enforces stricter requirements than ACM’s import path — most self-signed certificates that import cleanly are then rejected when attached to a listener. The supported path is a DNS-validated certificate for a domain you control. See Provision an ACM certificate for the proper-cert path.

After provisioning a new certificate, update the Helm release:

helm upgrade icebreaker-agent ./eyrie-agent-<version>.tgz --namespace eyrie-compute --reuse-values --set agent.albCertificateArn=<new-arn>

When a query arrives, the Data Server’s Job Manager dispatches a query executor pod. If queries hang or time out:

kubectl get pods -n eyrie-compute
# Look for query executor pods in Pending status
kubectl describe pod <executor-pod-name> -n eyrie-compute

Cause: Karpenter is not installed or not configured.

Check whether Karpenter is running:

kubectl get pods -n karpenter

Check whether a NodePool exists for the Capacity Policy:

kubectl get nodepools

If no NodePool exists, the Icebreaker Agent may not have been able to create it. Check the Agent logs for Karpenter-related errors. If Karpenter is not installed, set rbac.manageNodePools: false in your Helm values — but note that without Karpenter, no new EC2 capacity will be provisioned and query executor pods will only run on existing nodes that have spare capacity.

Cause: No EC2 capacity available matching the Capacity Policy.

Check Karpenter logs for provisioning errors:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

Common sub-causes: the Capacity Policy specifies instance families or capacity types not available in your region or AZ, your AWS account has hit an EC2 service quota, or Spot capacity is unavailable for the requested instance families.

Cause: Insufficient IAM permissions on the colibri ServiceAccount.

If the query executor pod starts but immediately fails or the Data Server logs show IAM errors, confirm the IRSA annotation is set and the role has the required S3 permissions — see IAM.

After the Data Server is running, queries fail immediately with a planning error like catalog error: connection refused or failed to fetch table metadata.

Cause: The catalog endpoint is not reachable from query executor pods.

Test reachability from inside the cluster:

kubectl run -it --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  -n eyrie-compute \
  -- curl -sv https://<your-catalog-host>/catalog/v1/config?warehouse=<warehouse>

Expected: a JSON response with "defaults" and "overrides" keys. If you see connection refused, no route to host, or a DNS resolution failure, the catalog endpoint is not reachable from the pod network.

Common causes:

Cause: Catalog URI is incorrect.

The catalog URI must point to the REST Catalog base path, not the catalog’s UI or a specific table. For Lakekeeper, the correct form is https://<host>/catalog — not https://<host>/ui or https://<host>/catalog/v1/namespaces/.... See Catalog integration for the expected format for your catalog type.

1. Confirm the Data Server status is running in the Admin console.
2. Confirm the ingress exists in your cluster:

   kubectl get ingress -n eyrie-compute    # nginx mode
   kubectl get ingress -n eyrie-compute    # ALB mode — check ADDRESS column is populated

3. In ALB mode: confirm the ALB is active in the AWS console and its target group shows healthy targets.
4. In nginx mode: confirm the wildcard DNS record resolves to the nginx controller’s LoadBalancer IP:

   nslookup <data_server_id>.<your-ingress-domain>

The raw ELB hostname (k8s-eyrieco-xxxxxxx.us-east-1.elb.amazonaws.com) does not match most certificates. Options:

• Configure your SQL client to skip TLS hostname verification (acceptable for evaluation).
• Point a CNAME from a name covered by your certificate to the ELB hostname and use the CNAME as the connection host. Update the JDBC URL manually.

See Networking and ingress for details.

A bare UNAVAILABLE with no further detail almost always means the ALB had no healthy target to route to, or it closed the connection while the query was still running. Check target health first:

EC2 console → Target Groups → select the Data Server's target group → Targets tab

Cause: No healthy targets.

If targets show unhealthy or draining with none healthy, the colibri pod behind the ALB isn’t ready yet. Check the pod directly:

kubectl get pods -n eyrie-compute -l colibri-instance-id=<instance_id>
kubectl logs -n eyrie-compute -l colibri-instance-id=<instance_id>

Common causes: image pull failure, or the pod can’t reach its configured Data Catalog endpoint — see Data Catalog not reachable above.

Cause: ALB idle timeout hit during a cold start.

If the query fails specifically after your typical cold-start duration (Karpenter provisioning a new node, common on Spot-backed capacity policies with no warm capacity), the ALB’s idle timeout — 60 seconds by default — is closing the connection before the query executor is ready. See Networking and ingress for how to raise it.

The nginx controller is receiving the connection but not routing it to the Data Server. Confirm:

1. The ingressTlsSecret is set and the Secret exists in the workload namespace:

   kubectl get secret <your-wildcard-tls-secret> -n eyrie-compute

2. The certificate in the Secret covers *.{your-ingress-domain}.
3. Arrow Flight SQL requires HTTP/2. Confirm your nginx controller has nginx.ingress.kubernetes.io/backend-protocol: GRPC or equivalent configured. The Icebreaker Agent sets this annotation on Ingresses it creates; verify it is present:

   kubectl get ingress <ingress-name> -n eyrie-compute -o yaml

On a fresh EKS Auto Mode cluster, kubectl get storageclass may show only gp2 with provisioner kubernetes.io/aws-ebs — a legacy in-tree driver that Auto Mode’s control plane does not run. PVCs that rely on the default StorageClass (such as Lakekeeper’s embedded PostgreSQL) will stay Pending indefinitely.

Create a CSI-backed StorageClass using EKS Auto Mode’s built-in EBS driver:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-csi
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.eks.amazonaws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

kubectl apply -f ebs-storageclass.yaml
kubectl get storageclass

ebs-csi should appear with (default). If gp2 also shows (default), remove it:

kubectl patch storageclass gp2 -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

If kubectl get storageclass shows two or more classes with (default) next to them, Kubernetes’ DefaultStorageClass admission controller picks arbitrarily between them — new PVCs may bind to either one, with non-deterministic behavior.

Remove the default annotation from the unwanted class:

kubectl patch storageclass <class-name> -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Verify only the intended class shows (default):

kubectl get storageclass