Troubleshooting

Start with kobe status (caller) or kubectl get clusterpool,clusterinstance,clusterlease -n kobe-system (operator). Then pick the symptom.

Pool stuck Creating

Instances stay in Creating until the control plane is reachable, addons have been applied, and every readinessGates entry passes. They recycle if they stay there too long (creatingTimeout, default 10m).

Check:

kubectl get clusterinstance -n kobe-system -o wide
kubectl describe clusterinstance <name> -n kobe-system
kubectl logs -n kobe-system deploy/kobe -c kobe-operator --tail=200

Common causes: image pull on the inner k3s/k0s version, not enough CPU/memory on the host for resources.limits, bootstrap Job failing (kobe_bootstrap_failures_total), datastore PostgreSQL unreachable.

kobe_instance_oldest_creating_age_seconds and kobe_instance_create_duration_seconds tell you whether this is slow or stuck. See Observability.

Lease stays Pending

The API accepted the request; no Ready instance was free. kobe_queue_depth and kobe_lease_oldest_pending_age_seconds show whether the queue is moving.

• Fixed size pool: raise size, wait for a recycle, or wait for spec.queueTimeout (default 5m) to expire the waiter.
• Scaling pool: maxClusters may already be hit, or spec.scaling.queueTimeout (default 5m) will expire the waiter.
• Pool phase Exhausted: every cluster is leased. The lease binds when another one ends.
• Lease message says the pool "is blocked by quarantined members": nothing binds until those members are cleared. See the next section.
• AccessPolicy maxConcurrentLeases returns 429, not a long Pending.

Quarantined

Verified cleanup could not prove that every exact resource is gone. The instance stays unavailable and still counts against pool capacity. The pool phase reads Quarantined instead of Healthy or Idle.

Inspect the ClusterInstance and ClusterLease receipt before retrying cleanup. ClusterLease.spec.cleanupMode is write-once. VerifiedDestroy is eligible only for k3s with embedded SQLite; unsupported backends are rejected before bind.

Do not delete ledger or receipt objects to "free" the slot. That is how you hand capacity back on no evidence.

kobe_quarantined{profile,kind} counts quarantined leases and instances per pool. kobe_quarantined_oldest_age_seconds shows how long the oldest instance has been held. Quarantine has no timeout: a value that keeps growing will not fix itself.

Release a quarantined lease or instance

Use this only when the evidence can never arrive, for example after the backend objects were removed by hand or the evidence store was lost. The release deletes the object without a verified teardown receipt. You are vouching that the tenant's data is gone.

The annotation kobe.kunobi.ninja/release-quarantine releases one object. Its value must be exactly that object's UID, with no surrounding whitespace, so it cannot match a same-named replacement or a copied manifest.

1. Confirm the backend resources are gone, or that you accept deleting them now:

   kubectl describe clusterinstance <instance> -n kobe-system
   kubectl get clusterlease <lease> -n kobe-system -o yaml

2. Release the instance. The operator deletes it and runs the ordinary backend delete once. It then waits for the backend to report the recorded footprint absent before dropping the finalizer, because dropping it frees the instance name and its CIDR for a replacement. The wait ends after 10 minutes with a warning, or at once, with a warning, if the backend cannot check absence for this instance. The pool then replaces it:

   kubectl annotate clusterinstance <instance> -n kobe-system \
     kobe.kunobi.ninja/release-quarantine="$(kubectl get clusterinstance <instance> -n kobe-system -o jsonpath='{.metadata.uid}')"

3. Release the lease. The operator revokes its connect token, drops the receipt finalizer, and deletes it:

   kubectl annotate clusterlease <lease> -n kobe-system \
     kobe.kunobi.ninja/release-quarantine="$(kubectl get clusterlease <lease> -n kobe-system -o jsonpath='{.metadata.uid}')"

Each completed release logs a QUARANTINE OVERRIDE warning, posts a QuarantineReleased event on the object, and increments kobe_quarantine_releases_total. These happen once the finalizer is gone, not when the operator starts the release. Alert on that counter.

The instance and the lease are released separately. Releasing only the lease leaves the instance quarantined and counting against maxClusters.

With teardownAuthority.separate=true, an admission policy normally stops anyone from dropping a lease's receipt finalizer before the receipt is acknowledged. It makes one exception: the operator's own service account may drop it from a Quarantined lease whose kobe.kunobi.ninja/release-quarantine annotation already equals the lease's UID. The annotation has to be in place before the operator's write, so set it first, as above.

In that mode, apply the annotation as yourself, not as the kobe service account. The policy refuses any attempt by the kobe control plane or the teardown authority to add or change it on a ClusterLease or ClusterInstance. The control plane can mark a lease Quarantined on its own, so the grant is what keeps the release a human decision. Removing the annotation is allowed from any identity.

A lease created for a Sandbox composition is not released this way. The operator posts QuarantineReleaseRefused once and keeps it, because its owning SandboxLease retires it.

Rolling back past the Exhausted phase

Operators released before the Exhausted pool phase fail to decode a pool whose status.phase is Exhausted, and one such pool fails their whole pool list. Before rolling back, clear the phase on each affected pool:

kubectl patch clusterpool <pool> -n kobe-system --subresource=status \
  --type=json -p '[{"op":"remove","path":"/status/phase"}]'

The older operator writes its own phase on the next reconcile. The same can happen for a few seconds during a rolling upgrade, while an old replica is still running. Operators that know Exhausted read any phase they do not recognise as unset.

If the operator is upgraded before the CRDs, the API server rejects Exhausted. The operator then reports ScalingUp and logs a warning. Apply the chart's CRDs, as in Deploy.

Sandbox API returns 503

The operator could not verify Sandbox admission against Kubernetes. Install the Sandbox CRDs (SandboxPool, SandboxLease; executions also need SandboxExecution) and set agentSandbox.mode to managed or external. The three CRDs are checked per endpoint, so a partial install fails only the routes that need the missing type.

kobe status reports the operator version. An old operator listing a new SandboxPool with portRange can fail the whole pool list; apply CRDs, upgrade the operator, then write ranged ports. See Ports.

kobe cannot reach the endpoint

The CLI classifies transport failures:

kobe config view shows where the endpoint came from (target vs --endpoint).

JSON mode never opens a browser or asks TOFU questions. Run kobe login in text mode first.

SSH to a sandbox fails

kobe doctor

Each line is a check: binary, target, auth, SSH-capable pools, default pool, ssh_config Include, ssh -G. Failures name the command that fixes them; most of the time that is kobe init. See SSH.

CLI older than the operator

kobe status and kobe lease print a warning. Upgrade the CLI from Installation. New CRD fields (for example portRange) require the matching operator version before you apply them.