Skip to main content

Cloud-native k8s with mecak8s

mecak8s runs Mecatl with Kubernetes-native defaults. The chart uses two replicas by default for availability; you can use one replica to reduce resource usage and simplify session routing. Agent pods hold no durable state: session snapshots and the event log live in Redis, while Kubernetes Leases enforce a single writer for each session.

If a pod stops, a surviving pod can acquire its leases and resume interrupted sessions from Redis. The pod is disposable; the session is not.

Try mecak8s locally​

The repository includes a disposable local Kind environment with Redis and two mecak8s replicas. Follow Try Mecatl on Kubernetes to create the cluster and connect with mecatui.

For the optional Keycloak qualification flow and implementation details, see the local Kind README.

Before you deploy​

You need:

  • A Kubernetes cluster, Helm, and kubectl. The cluster must be able to pull public images from ghcr.io.
  • A Redis endpoint reachable from the agent pods. Production deployments use verified TLS and can read ACL credentials from a Kubernetes Secret.
  • An OIDC issuer and audience for caller authentication.
  • A TLS certificate and key for the mecak8s Service, unless an operator-controlled gateway terminates TLS.
  • Credentials for an LLM provider.

The chart creates the ServiceAccount and namespace-scoped permissions for Kubernetes Leases. It does not create production Redis, TLS Secrets, provider Secrets, gateways, or general NetworkPolicies. A deployment that selects models.router.backend: jev must project TYPESAFE_API_KEY from a Kubernetes Secret and allow HTTPS egress to api.typesafe.ai; eligible delegated task text leaves the cluster for classification.

Quick start​

The following example stores credentials in Kubernetes Secrets and installs the published chart with two replicas. Choose a version from GitHub releases, then substitute its number without the leading v for <VERSION>. Replace the example endpoints and filenames with values for your environment.

  1. Create the namespace and Secrets. Set OPENAI_API_KEY in your current shell first. Redis and TLS values are read from files instead of being placed in Helm values:

    : "${OPENAI_API_KEY:?set OPENAI_API_KEY first}"
    kubectl create namespace mecatl
    kubectl create secret generic mecak8s-openai --namespace mecatl \
      --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY"
    kubectl create secret generic mecak8s-redis --namespace mecatl \
      --from-file=password=./redis-password \
      --from-file=ca.pem=./redis-ca.pem
    kubectl create secret tls mecak8s-tls --namespace mecatl \
      --cert=./server.crt --key=./server.key

  2. Save the deployment values as mecak8s-values.yaml:

    redis:
      endpoint: redis.example.internal:6379
      credentialsSecret: mecak8s-redis
      passwordKey: password
      caKey: ca.pem

    tls:
      enabled: true
      secretName: mecak8s-tls

    oidc:
      enabled: true
      issuer: https://idp.example.com
      audience: mecatl

    defaultProvider: openai
    model: gpt-5
    extraEnv:
      - name: OPENAI_API_KEY
        valueFrom:
          secretKeyRef:
            name: mecak8s-openai
            key: OPENAI_API_KEY

  3. Install the chart and wait for both replicas:

    helm upgrade --install mecak8s \
      oci://ghcr.io/stacklok/mecatl/charts/mecak8s \
      --version <VERSION> \
      --namespace mecatl \
      --values mecak8s-values.yaml
    kubectl rollout status deployment/mecak8s-mecak8s --namespace mecatl

The default Deployment name is <RELEASE>-mecak8s. Set fullnameOverride in the values file if you need a fixed name.

Deployment defaults​

mecak8s starts with unattended, Kubernetes-oriented defaults:

SettingDefault
ReplicasTwo
PostureHeadless with auto permissions
Session stateRedis
Session ownershipKubernetes Leases
FilesystemNo filesystem access
Network bindsPod network on 0.0.0.0
Metrics and OpenTelemetryOpt-in

Project-provided instructions, skills, agents, and read-only child Shell access still require explicit project trust. mecak8s does not include ACP, local JSONL storage, or the mecated operator subcommands.

Choose filesystem access​

mecak8s uses the same server-owned placement contract as mecated. Clients do not send workspace paths or private environment references. By default, new sessions have no filesystem access. Schedules retain that placement, and delegation cannot add filesystem access that the parent lacks.

See Execution environments for the shared placement, no-FS, child-environment, and reattachment model.

Kubernetes execution provider​

The optional execution provider runs in an operator-controlled cluster. It supports foreground commands and does not provide hostile multi-tenant isolation. Configure cluster isolation and network policy for the workloads you allow to run.

The optional execution provider runs as a separate service and controller with its own mecatl-execution chart. mecak8s remains a client and receives no Pod, PVC, custom-resource, or controller management permissions.

For private registry pulls, create pull Secrets in the execution namespace before installing the provider chart. Import a locally prepared Docker config without putting credential contents in Helm values or command arguments:

kubectl --namespace <NAMESPACE> create secret generic provider-registry \
  --type=kubernetes.io/dockerconfigjson \
  --from-file=.dockerconfigjson=<LOCAL_DOCKER_CONFIG_JSON>

Set provider.imagePullSecrets: [provider-registry] for the provider image and templates.<id>.revisions.<revision>.execution.imagePullSecrets: [workload-registry] for each workload image that needs one. Each optional list accepts up to eight unique Kubernetes Secret names. The chart creates a release-derived executor ServiceAccount with no RBAC and token automount disabled, retained while executor Pods may survive provider removal; executor Pods also disable token automount and use only their template revision's pull Secret names. Keep registry credentials out of workload environment variables and the security material Secret. Changing pull Secret names changes the canonical revision; publish a new revision and retain the old definition for existing allocations. Provider-only pull Secret names are not workload identity and may change in a compatible quiesced upgrade.

The provider supplies run-wide ownership, transactional references, controlled replacement and retirement, durable claim revocation, and reloadable TLS from platform Secrets. Production isolation requires a CNI that enforces NetworkPolicy and a RuntimeClass that supplies the isolation promised by your platform. See Native Kubernetes lifecycle and retention for restart recovery, fencing, and placement-capacity behavior.

Before installation, provide these values through your trusted image and Secret delivery system:

  • Use a workload base from the same supported Mecatl release as the provider: the provider's private gRPC protocol marker is execution-grpc/1, but that marker alone does not guarantee compatibility with an arbitrary executor build. The release builds the workload for Linux amd64 and arm64. Keep the release's /mecatl-executor at that absolute path; the provider starts it in the workload Pod and exchanges bounded JSON requests and responses on its standard input and output. The provider overrides the container command with /bin/sh -c 'trap : TERM INT; sleep infinity & wait', so the image must provide /bin/sh and sleep; keep the supported image's idle entrypoint for direct launches as well. The executor invokes /bin/sh -c for foreground commands with a fixed environment: HOME=/workspace, PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin, TMPDIR=/tmp, and GOTMPDIR=/tmp. Install extra tools on that PATH; don't rely on image or Pod environment variables being inherited by commands.
  • A derived image must extend that compatible executor image without replacing /mecatl-executor or its entrypoint. Run as UID/GID 65532 and preserve the writable /workspace PVC and /tmp mounts. Keep its request/response protocol intact; add tools without exposing credentials. Publish the resulting image under a distinct template ID or revision, pinned by its registry or loaded manifest digest (not its local image config ID). Retain old image digests and their pull Secrets for as long as retained allocations can restart, even after deprecation or revocation; removing a revision from the catalog does not remove its retained recipe or storage.
  • Security files supplied by operator-owned Kubernetes Secrets. The provider mounts their projected files without subPath; Secret bytes remain externally managed. Use securitySecretName for a single Secret or securitySources for explicit mappings from multiple Secrets.
  • A mecak8s client mTLS Secret containing ca.crt, tls.crt, and tls.key.
  • Explicit provider-client pod and namespace selectors, API-server CIDRs, and DNS resolver CIDRs.

The provider validates every configured RuntimeClass and StorageClass with cluster-scoped get requests before it becomes ready. RuntimeClasses must have no scheduling selectors or tolerations: only the operator profile may define executor scheduling. The chart grants those requests only for the names present in templates; it grants no cluster-wide list or watch access.

The operator owns each immutable recipe under a versioned template ID. The revision key is v1- plus SHA-256 of the bytes mecatl/execution-template/v1\0 followed by Go encoding/json of the validated ProfileSpec (including time.Duration as nanoseconds). Do not hash YAML or the image tag. The Kind fixture computes the revision with the same Go spec encoding and validates it through the production loader; it creates the output file only if absent:

go run -tags kind_execution_e2e ./deploy/mecatl-execution-kind/fixture/templates \
  deploy/mecatl-execution-kind/template-recipes.yaml \
  '<DIGEST_PINNED_WORKLOAD_IMAGE>' '<DIGEST_PINNED_DERIVATIVE_IMAGE>' \
  .scratch/templates-values.yaml

The fixture builds the workload from build/execution-workload/Dockerfile with a digest-pinned Go base; pin the base to the architecture used by your nodes and resolve the actual loaded manifest digest before rendering template values.

The following template shows every required chart key. Save it as execution-values.yaml and replace each placeholder:

fullnameOverride: mecatl-execution
provider:
  image: <PROVIDER_IMAGE>@sha256:<PROVIDER_DIGEST>
  imagePullPolicy: IfNotPresent
  replicas: 2
  securitySecretName: <PLATFORM_TLS_SECRET_WITH_CLIENT_CA>
  securityManifest: |
    {
      "version": 1,
      "generation": 42,
      "tls": {
        "certificateFile": "tls.crt",
        "privateKeyFile": "tls.key",
        "clientCAFile": "clients.pem"
      },
      "clients": [{
        "uri": "spiffe://cluster.example.com/ns/mecatl/sa/mecak8s",
        "mayAttestOwner": true,
        "executionTemplates": ["coding"],
        "administrator": false
      }, {
        "uri": "spiffe://cluster.example.com/ns/mecatl/sa/execution-admin",
        "mayAttestOwner": false,
        "administrator": true,
        "administratorFor": ["spiffe://cluster.example.com/ns/mecatl/sa/mecak8s"]
      }]
    }
  clientIngressSelectors:
    - namespaceLabels: {kubernetes.io/metadata.name: <CLIENT_NAMESPACE>}
      podLabels: {app.kubernetes.io/name: mecak8s}
  apiServerCIDRs: [<API_SERVER_IP>/32]
  dnsCIDRs: [<CLUSTER_DNS_IP>/32]

service:
  port: 8443

templates:
  coding:
    default: v1-<CANONICAL_64_HEX_SHA256>
    revisions:
      v1-<CANONICAL_64_HEX_SHA256>:
        display: {name: Coding}
        execution:
          image: <WORKLOAD_IMAGE>@sha256:<WORKLOAD_DIGEST>
          storageClass: <STORAGE_CLASS>
          storageSize: 2Gi
          cpuRequest: 100m
          memoryRequest: 128Mi
          cpuLimit: "1"
          memoryLimit: 1Gi
          ephemeralStorageRequest: 64Mi
          ephemeralStorageLimit: 1Gi
          tmpSizeLimit: 256Mi
          runtimeClassName: <RUNTIME_CLASS>
          # Optional operator-only scheduling constraints.
          nodeSelector:
            node.kubernetes.io/instance-type: <WORKER_TYPE>
          tolerations:
            - key: dedicated
              operator: Equal
              value: build
              effect: NoSchedule
          maxFileBytes: 5242880
          maxCommandBytes: 1048576
          maxCommandDuration: 5m
          maxEnvironments: 100

networkPolicy:
  enabled: true
  dnsPorts: [53]
  apiServerPorts: [443]
  workloadProfiles: {} # default deny; add explicit coding egress only if required

resourceGovernance:
  enabled: true
  pods: "200"
  persistentVolumeClaims: "100"
  executionEnvironments: "100"
  requestsCPU: "20"
  requestsMemory: 40Gi
  requestsStorage: 500Gi
  requestsEphemeralStorage: 40Gi
  limitsCPU: "40"
  limitsMemory: 80Gi
  limitsEphemeralStorage: 80Gi

nodeSelector and tolerations are optional, immutable, operator-only template execution settings; clients cannot select or change them. Omit either setting when unused: the chart rejects explicit empty maps and lists. A revision accepts at most 32 qualified label selectors and 16 unique tolerations. Tolerations use Exists (with an empty value) or Equal (with a key); tolerationSeconds must be between 0 and 86,400 seconds and only applies to NoExecute. New executor Pods explicitly include the 300-second NoExecute tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable unless the template already specifies that key and effect; previously created Pods may retain cluster-configured default durations of up to one day. The controller rejects other scheduling or affinity mutations. Scheduling changes the canonical execution revision. Publish a new revision, retaining all historical definitions used by allocations. Only display/deprecation policy may change without creating a new revision.

Platform PKI owns the provider server certificate (Service DNS SANs, serverAuth EKU), client certificates (exact URI SANs, clientAuth EKU), keys and trust bundles. The workload chart does not install cert-manager or generate a CA. Increase the manifest generation only when the authorized clients policy changes. Routine TLS renewal keeps the same file paths and generation.

To split TLS files across platform-managed Secrets, replace securitySecretName in provider values with explicit mappings, for example:

provider:
  securitySecretName: ""
  securitySources:
    - name: provider-tls
      items:
        - {key: tls.crt, path: tls.crt}
        - {key: tls.key, path: tls.key}
    - name: provider-client-roots
      items:
        - {key: ca.crt, path: clients.pem}

Create the named TLS Secrets in the provider namespace before installation. Each path is a unique basename, and every TLS file in the manifest needs a mapping. The chart projects only mapped Secret keys and the policy ConfigMap. Secret values belong in Kubernetes Secrets, not Helm values. A cert-manager Certificate can own each leaf Secret using a pre-existing namespaced Issuer; other external PKI that supplies equivalent projected Secret keys works the same way. The CA private key remains issuer-owned, never in a workload Secret or chart render. cert-manager is optional and the platform operator installs it separately. Avoid subPath: Kubernetes atomic projection and provider reload require full directory mounts. The provider rereads TLS independently of the policy manifest, so valid certificate renewal needs no Helm upgrade or Pod restart.

The Kind fixture is for local testing only. Its generated root expires after 14 days and its cert-manager-issued leaves last one hour. Do not use its issuer or root in a deployment. For a long-running deployment, configure a platform-managed issuer and make root renewal, overlapping trust publication on the provider and client, and withdrawal of old trust explicit platform PKI responsibilities.

Install the provider chart separately from mecak8s, before allocating any execution environments. Helm 3.16 is the minimum supported version. Helm 4 uses server-side apply by default, so each chart lifecycle command selects client-side mode when needed.

Use a namespace dedicated to this provider release:

HELM_APPLY_MODE=
case "$(helm version --template '{{.Version}}')" in
  v4.*) HELM_APPLY_MODE=--server-side=false ;;
esac
helm install mecatl-execution ./deploy/helm/mecatl-execution \
  --namespace <NAMESPACE> \
  --values execution-values.yaml \
  $HELM_APPLY_MODE
kubectl rollout status deployment/mecatl-execution --namespace <NAMESPACE>

Add the following block to the existing mecak8s values. The endpoint is a host:port gRPC target (no URL scheme) and must use the provider certificate's DNS identity. The private mecatl.execution.v1.ExecutionProviderService uses protocol execution-grpc/1 with mandatory mTLS. Keep OIDC caller authentication enabled; the remote binding is scoped to the verified issuer and subject. The client reloads its projected CA bundle and certificate/key together every two seconds; a changed complete set replaces the gRPC connection without restarting mecak8s. A missing, expired, or mismatched projection closes the previous connection and blocks new operations until valid material returns. Publish overlapping trust roots before switching issuers, and keep the old root until both ends have moved. Provider and client TLS projections reload independently from authorization policy; a routine PKI renewal needs no manifest generation change or Pod restart.

execution:
  enabled: true
  endpoint: mecatl-execution.<NAMESPACE>.svc:8443
  templateID: coding
  templateRevision: v1-<CANONICAL_64_HEX_SHA256>
  allowedSubjects: [<APPROVED_OIDC_SUBJECT>]
  tlsSecret: <MECAK8S_EXECUTION_MTLS_SECRET>
  caKey: ca.crt
  certKey: tls.crt
  keyKey: tls.key

Remote sessions receive the filesystem tools and foreground Shell, but do not ingest project instructions, rules, skills, or source from the remote PVC. Schedules, SkillDraft, background Shell, and delegated filesystem execution are unavailable with this client.

If allocation reports Ready=false with PVCUnavailable or ExecutorUnavailable, inspect namespace quota, admission failures, and the executor Pod's events. For a private-image pull failure, use kubectl --namespace <NAMESPACE> get pods and kubectl --namespace <NAMESPACE> describe pod <POD_NAME> for the provider or executor Pod; check the named pull Secret and image digest. Correct the failed prerequisite and wait for Ready=True. The controller keeps retrying through its rate-limited queue while the allocation exists; no reference edit, restart, or manual reconciliation is needed. Missing authoritative PVCs or Pods and ownership mismatches remain fail-closed and are never repaired by creating a replacement.

Run claims renew before their durable lease expiry. Renewal receipts retain the newest 32 identities. Retrying a retained operation ID preserves the claim and original expiry; conflicting or expired replays cannot extend a claim.

The provider retains one PVC per logical environment. Session deletion and chart uninstall do not delete committed workspace data. Retirement requires no live references and a terminal executor, and retains the PVC. A missing executor or lost terminal receipt moves the environment to FenceUnknown and requires external operator fencing. The built-in recovery RPC accepts only a still-observable Pod in Succeeded or Failed phase with every container terminated and the exact PVC still present. If that proof is missing, use your platform's external fencing runbook; there is no acknowledgement flag.

The built-in recovery RPC requires the same exact identity and an independently stable operation ID. It does not read a Secret and has no force or acknowledgement field:

grpcurl -cacert "$CA_FILE" -cert "$CERT_FILE" -key "$KEY_FILE" \
  -import-path contracts/proto -proto "$PROTO" \
  -d '{"environment":{"id":"<ENVIRONMENT_ID>","revision":"<REVISION>"},"owner":{"issuer":"<OWNER_ISSUER>","subject":"<OWNER_SUBJECT>"},"expectedExecutionEpoch":"<EPOCH>","expectedPodUid":"<POD_UID>","expectedPvcUid":"<PVC_UID>","operationId":"recover-<STABLE_UUID>"}' \
  "$EXECUTION_ENDPOINT" mecatl.execution.v1.ExecutionProviderService/RecoverEnvironment

Rotate execution-provider TLS and authorization policy​

PKI renews tls.crt, tls.key, and trusted CA files in the projected Secrets; provider replicas reload valid material without changing generation. mecak8s reconnects with its newly projected client material without a restart. Plan renewal well before expiry, and observe both provider replicas and client RPCs through each renewal. For CA changes, first stage an overlap of old and new roots on both sides, switch the leaf identities, then remove the old roots only after all callers use the new chain. A missing or invalid projection fails closed rather than skipping certificate verification.

For a client authorization policy change, publish a manifest with a higher generation. The retained authority ConfigMap stores {generation,digest} of normalized policy only; it must never be reset. Verify both replicas converge and a current run can read known workspace data. Requests on an existing HTTP/2 connection are subject to the new policy on every RPC. A same-generation policy edit or rollback is rejected; correct it with another incremented generation, never by deleting the ledger.

Upgrade, uninstall, and reinstall the execution provider​

The supported lifecycle keeps the same Helm release name, namespace, resource names and historical template execution definitions and network policy configuration. The chart does not own TLS Secrets; the platform issuer owns renewal and their lifecycle independently of Helm. Keep execution-values.yaml and the current nonsecret client-policy manifest in your operator configuration store.

Finish and verify any external authority rotation before starting a chart upgrade. Do not rotate the authority ConfigMap while Helm is writing chart resources. Each Helm lifecycle command below initializes HELM_APPLY_MODE for the current shell. Helm 4 selects client-side field ownership, while Helm 3.16 leaves the value empty. Continue to use reviewed current values and the quiescence procedure below. Do not use --force-conflicts or --take-ownership as a blanket takeover.

Before upgrading, verify that the retained templates ConfigMap contains a nonempty data["lifetime.json"], the authority and capacity ledgers are intact, and the release still owns its retained executor ServiceAccount. That account must have token automount disabled and no inherited pull Secrets. Stop if any of these checks fails; do not synthesize missing history or reset authority.

For a compatible provider upgrade:

  1. Stop new client traffic and finish or explicitly fence active work. Stop all provider replicas before Helm reads the ledgers:

    kubectl --namespace <NAMESPACE> scale deployment/mecatl-execution --replicas=0
    kubectl --namespace <NAMESPACE> wait --for=delete pod \
      --selector app.kubernetes.io/name=mecatl-execution --timeout=2m

  2. Apply the reviewed CRD schema before starting the upgraded provider. Helm does not upgrade existing CRDs:

    kubectl apply -f deploy/helm/mecatl-execution/crds/executionenvironment.yaml
    kubectl wait --for=condition=Established \
      crd/executionenvironments.execution.mecatl.dev --timeout=60s

  3. Upgrade with the preserved lifetime configuration and current authority manifest. Helm reads the existing ConfigMaps and includes their actual ledger data, rather than empty bootstrap data, in the new release:

    HELM_APPLY_MODE=
    case "$(helm version --template '{{.Version}}')" in
      v4.*) HELM_APPLY_MODE=--server-side=false ;;
    esac
    helm upgrade mecatl-execution ./deploy/helm/mecatl-execution \
      --namespace <NAMESPACE> --values execution-values.yaml \
      --wait --timeout=4m $HELM_APPLY_MODE

  4. Verify readiness, exact environment/PVC UIDs, data, and network confinement before resuming traffic.

Default uninstall removes the provider but keeps runtime CRs, PVCs, surviving executors, workload default-deny and template NetworkPolicies, both authority and capacity ledgers, and the templates/security-manifest ConfigMaps. Uninstall is not executor termination proof or storage disposal:

helm uninstall mecatl-execution --namespace <NAMESPACE>

Reinstall with the same identity and preserved values:

HELM_APPLY_MODE=
case "$(helm version --template '{{.Version}}')" in
  v4.*) HELM_APPLY_MODE=--server-side=false ;;
esac
helm install mecatl-execution ./deploy/helm/mecatl-execution \
  --namespace <NAMESPACE> --values execution-values.yaml \
  --wait --timeout=4m $HELM_APPLY_MODE

Helm adopts retained resources only when their managed-by label and release-name and release-namespace annotations match. The chart rejects missing or empty ledgers while allocations or capacity reservations survive, and rejects changes to its retained lifetime configuration. Dropping or changing historical execution definitions and editing egress for an existing template ID are outside this upgrade path. Publish additive revisions and retain old ones; policies must remain identical across revisions of the same ID. New IDs may introduce egress only when no retained workload Pod matches their ID label. The chart rejects inventories exceeding 64 total revisions before deployment. Display and deprecation policy may change. Retained allow policies are additive, so leaving an obsolete policy could widen access. The provider rejects an older client-policy manifest against the retained authority generation; never reset the policy digest ledger to make readiness pass.

Use live Helm install/upgrade for this lifecycle. Offline helm template cannot perform ownership or history lookups and is not an adoption mechanism. Keep provider writers stopped for upgrades; lookup plus apply is not a cross-resource transaction. Rendering rejects a nonzero existing provider Deployment or any remaining provider Pod, including a terminating Pod. Changed release/namespace adoption, chart rollback, --force-conflicts, --take-ownership, CRD or namespace deletion with retained resources, and force-finalizer cleanup are unsupported.

Final infrastructure decommission is not automated. Supported RetireEnvironment and DeleteRetiredEnvironment operations can terminate and dispose of eligible individual environments; they do not remove the retained NetworkPolicies, configuration, ledgers, or CRD. There is no supported final infrastructure-cleanup procedure here. Manual destructive cleanup is outside the supported lifecycle. Keep retention defaults and authority history, and never reset ledgers or remove finalizers to bypass a failed safety check.

Run an administrative lifecycle operation​

Administrative RPCs require administrator: true. For a distinct operations identity, set its administratorFor list to the canonical URI of each client that created the environments it may administer, as in the values example above. An absent or empty list permits only self-administration. Keep mayAttestOwner: false for an operations-only identity; administrative scope confers no filesystem, Shell, attach, run, or reference access. See the provider security material.

Before first publishing administratorFor, quiesce client traffic and upgrade all provider replicas to a version that understands the field. Older strict manifest decoders reject it; mixed-version rolling operation is unsupported. Preserve the authority high-water ConfigMap throughout the upgrade. Publish the reviewed manifest with a higher generation, verify provider readiness, then resume client traffic.

Every administrative request still requires the original exact owner and revision plus the operation's epoch, UID, schema, or generation preconditions. The scope names creators, not owners; it can include a creator whose login has been removed. After maintenance, remove its scope entry and increase generation again. Subsequent requests, including receipt retries on established connections, are denied; already admitted lifecycle operations may finish safe reconciliation.

An administrative not_found response deliberately does not distinguish a missing environment from a wrong owner, revision, or creator scope. Check the exact environment ID/revision and original owner against your authorized operations record, then check that administratorFor names the original creator's exact canonical client URI. If the scope is wrong, publish a reviewed manifest at a higher generation and verify readiness before retrying. The RPC will not disclose another creator's data to diagnose a scope mismatch.

First capture the private identity while the environment still has a reference:

kubectl --namespace <NAMESPACE> get executionenvironment <ENVIRONMENT_ID> \
  -o jsonpath='{.spec.revision}{"\n"}{.spec.ownerIssuer}{"\n"}{.spec.ownerSubject}{"\n"}{.status.epoch}{"\n"}{.status.pod.uid}{"\n"}{.status.pvc.uid}{"\n"}{.status.grantGeneration}{"\n"}'

Record those seven lines as <REVISION>, <OWNER_ISSUER>, <OWNER_SUBJECT>, <EPOCH>, <POD_UID>, <PVC_UID>, and <GRANT_GENERATION>. The owner hash is not reversible, so retain the bounded spec.ownerIssuer and spec.ownerSubject attestation in your authorized operations record before removing the final reference.

Set file references to trusted, mounted mTLS material. Keep private-key bytes out of shell arguments and manifests:

EXECUTION_ENDPOINT=mecatl-execution.<NAMESPACE>.svc:8443
CA_FILE=/var/run/secrets/mecatl-admin/ca.crt
CERT_FILE=/var/run/secrets/mecatl-admin/tls.crt
KEY_FILE=/var/run/secrets/mecatl-admin/tls.key
PROTO=contracts/proto/mecatl/execution/v1/execution.proto

Replace one executor by reusing the same operation ID for every retry:

grpcurl -cacert "$CA_FILE" -cert "$CERT_FILE" -key "$KEY_FILE" \
  -import-path contracts/proto -proto "$PROTO" \
  -d '{"environment":{"id":"<ENVIRONMENT_ID>","revision":"<REVISION>"},"owner":{"issuer":"<OWNER_ISSUER>","subject":"<OWNER_SUBJECT>"},"expectedExecutionEpoch":"<EPOCH>","expectedPodUid":"<POD_UID>","expectedPvcUid":"<PVC_UID>","operationId":"replace-<STABLE_UUID>"}' \
  "$EXECUTION_ENDPOINT" mecatl.execution.v1.ExecutionProviderService/ReplaceExecutor

To retire and then delete retained storage, call RetireEnvironment with the same identity fields and a new stable operation ID. Poll until .status.conditions[?(@.type=="Retired")].status is True, then call DeleteRetiredEnvironment with the retained PVC UID and another stable operation ID. The provider refuses either request while references, claims, identity proof, or UID checks are incomplete.

grpcurl -cacert "$CA_FILE" -cert "$CERT_FILE" -key "$KEY_FILE" \
  -import-path contracts/proto -proto "$PROTO" \
  -d '{"environment":{"id":"<ENVIRONMENT_ID>","revision":"<REVISION>"},"owner":{"issuer":"<OWNER_ISSUER>","subject":"<OWNER_SUBJECT>"},"expectedExecutionEpoch":"<EPOCH>","expectedPodUid":"<POD_UID>","expectedPvcUid":"<PVC_UID>","operationId":"retire-<STABLE_UUID>"}' \
  "$EXECUTION_ENDPOINT" mecatl.execution.v1.ExecutionProviderService/RetireEnvironment
kubectl --namespace <NAMESPACE> wait executionenvironment/<ENVIRONMENT_ID> \
  --for='jsonpath={.status.conditions[?(@.type=="Retired")].status}=True' \
  --timeout=10m
grpcurl -cacert "$CA_FILE" -cert "$CERT_FILE" -key "$KEY_FILE" \
  -import-path contracts/proto -proto "$PROTO" \
  -d '{"environment":{"id":"<ENVIRONMENT_ID>","revision":"<REVISION>"},"owner":{"issuer":"<OWNER_ISSUER>","subject":"<OWNER_SUBJECT>"},"expectedPvcUid":"<PVC_UID>","operationId":"delete-<STABLE_UUID>"}' \
  "$EXECUTION_ENDPOINT" mecatl.execution.v1.ExecutionProviderService/DeleteRetiredEnvironment

Revoke durable run authorization with the current generation and a stable operation ID. The response returns the new generation; retrying the identical request returns the same receipt. Claims from the old generation are denied after this CAS succeeds.

grpcurl -cacert "$CA_FILE" -cert "$CERT_FILE" -key "$KEY_FILE" \
  -import-path contracts/proto -proto "$PROTO" \
  -d '{"environment":{"id":"<ENVIRONMENT_ID>","revision":"<REVISION>"},"owner":{"issuer":"<OWNER_ISSUER>","subject":"<OWNER_SUBJECT>"},"expectedGrantGeneration":"<GRANT_GENERATION>","operationId":"revoke-<STABLE_UUID>"}' \
  "$EXECUTION_ENDPOINT" mecatl.execution.v1.ExecutionProviderService/RevokeEnvironment

Foreground command cancellation is cooperative and bounded. The helper attempts to terminate the command process group and reports a terminal receipt. If it cannot prove termination, the environment is fenced instead of admitting more work. Detached command status and cancellation are not available.

When execution.enabled is false, the mecak8s chart mounts no execution mTLS Secret and passes no execution-provider flags. The separate provider chart and any environments it owns continue independently.

The no-FS default applies when execution is disabled. With execution enabled, omitting execution binds the configured exact default template; only explicit execution:{none:{}} selects the filesystem-free surface. Clients never send a workspace path.

Mounted workspace (shared filesystem root)​

To give sessions a real filesystem, mount a volume into the pod and point --workspace at it (for example a PVC mounted at /workspace). A configured root turns mecak8s into a server-assigned filesystem deployment rooted there: every session is assigned that single root, the filesystem tools and Shell operate on it, and clients have no field with which to select another root. The path must be absolute and clean; a relative value is refused at startup.

This does not change mecak8s's storage-free posture: harness and session state still live in Redis and the Kubernetes API, and the mounted volume holds only agent working files. A root shared across the two default replicas needs a ReadWriteMany volume; a ReadWriteOnce PVC binds to a single node, so scale to one replica or use a per-pod volume if your storage class cannot do RWX. The operator vouches for the mount, so scope it deliberately. Caller identity does not isolate one caller's files from another caller using the same mount.

Redis virtual workspace​

Set redis.filesystem.enabled=true (or pass --redis-filesystem) to provide persistent Read/ListDir/Edit/Write/Copy/Move/Remove/Grep/Glob files without mounting a volume. Files are partitioned by the session owner's exact OIDC issuer/subject pair; same-owner sessions share a namespace, while ownerless sessions share a reserved anonymous namespace. The persisted placement is revalidated on every run. Redis failures, missing namespace markers, and corrupt records fail closed rather than appearing as an empty filesystem.

This mode is deliberately file-lite: it has no Shell, executable-file semantics, git worktrees, or filesystem branch/merge workflow. It is mutually exclusive with workspace. Set redis.readLedger.enabled=true independently to persist each session's read-before-write evidence; deleting a session deletes that ledger but not the principal's shared files. mecak8s sets no TTL on either representation. Redis durability, backups, capacity and eviction policy remain operator concerns.

State topology​

mecak8s keeps session data in Redis and session ownership in the Kubernetes API:

StateService
Session snapshots, retention, and cleanupRedis
Durable event log and resume cursorsRedis Streams
Single-writer session leaseKubernetes coordination.k8s.io Lease

Inspect the event log​

The durable event log at mecatl:events:<SESSION_ID> is a Redis Stream. Read it with:

redis-cli XRANGE "mecatl:events:<SESSION_ID>" - +

Each entry's r field contains the event envelope, including its Actor metadata. Do not modify the sibling cursor key, mecatl:events-gen:<SESSION_ID>, independently of the event log.

Size durable event followers​

Each mecak8s process uses a dedicated Redis client for blocking durable-event followers. This keeps session saves, event appends, and metadata operations on the durability client when watches are idle.

CLI flagHelm valueDefault
--redis-follow-pool-sizeredis.follow.poolSize32
--redis-max-followersredis.follow.maxFollowers32

Both values must be positive integers, and the maximum follower count must not exceed the follow pool size. mecak8s and the Helm chart reject invalid values at startup or render time.

Keep the defaults unless one pod must serve more than 32 concurrent durable watches. Size the follower limit for the expected per-pod watch concurrency, then give the pool at least that many connections. Include every replica and briefly overlapping credential generations in the Redis connection budget.

When the process has admitted the maximum number of followers, a new watch ends with watch_capacity. gRPC reports RESOURCE_EXHAUSTED; HTTP retains its status 200 event stream and sends a terminal event: error frame. The TypeScript SDK reconnects from the last processed cursor with the same filter. This code is separate from watch_lagging, which means a client did not consume the server's bounded delivery buffer quickly enough.

Production Helm chart​

The production chart is published at oci://ghcr.io/stacklok/mecatl/charts/mecak8s; its source is in deploy/helm/mecak8s/. It requires an external Redis endpoint and creates no Redis StatefulSet. Reference a Kubernetes Secret for Redis credentials. The chart's default image tag matches its application and chart versions, so it pulls the corresponding signed ghcr.io/stacklok/mecatl/mecak8s release. Set image.digest to pin an immutable image. It accepts a canonical lowercase SHA-256 digest: sha256: followed by 64 lowercase hexadecimal characters. Set only one of image.tag and image.digest, or clear the tag to use v<chart-version>.

A real-provider deployment (mockProvider: false) must choose one of these security postures:

PostureConfiguration
TLS in the podEnable tls and OIDC.
TLS at an operator-owned edgeSet security.tlsTerminatedUpstream=true, enable OIDC, and optionally keep in-pod TLS for re-encryption.
Unsafe bypassSelect the explicit bypass, which annotates the pod as unsafe.

The chart reserves its security-posture annotations; podAnnotations cannot override them.

Installation telemetry identity​

The chart stores a non-secret installation UUID in a ConfigMap. Live Helm upgrades preserve an automatically generated ID with lookup, but uninstalling the release removes it. Set a canonical lowercase UUID explicitly for GitOps or to preserve the ID across reinstall:

telemetry:
  installationID: 123e4567-e89b-12d3-a456-426614174000

Changing the value rolls the Deployment, and OpenTelemetry exports it as the mecatl.installation.id resource attribute. To rotate an automatically generated ID, make the new UUID explicit during an upgrade:

NEW_ID="$(uuidgen | tr '[:upper:]' '[:lower:]')"
helm upgrade <RELEASE> oci://ghcr.io/stacklok/mecatl/charts/mecak8s \
  --version <VERSION> --namespace <NAMESPACE> --reuse-values \
  --set-string telemetry.installationID="$NEW_ID"

Use workload identity for an LLM gateway​

Edge-terminated h2c sends the caller's bearer token across the pod network in cleartext. Restrict the Service to gateway pods with NetworkPolicy or an mTLS mesh. The chart does not verify the gateway or provide a general NetworkPolicy.

The gateway must forward the original bearer token and expose only the gRPC route. Do not publish /drain, /healthz, or /readyz. The chart creates no Gateway, Route, Certificate, or BackendTLSPolicy; configure those resources or retain in-pod TLS for re-encryption. Move an existing in-pod TLS release to h2c through a blue-green or maintenance cutover.

For an OpenAI-compatible gateway that trusts Kubernetes workload identity, use the chart's existing extraArgs, extraVolumes, and extraVolumeMounts to project a ServiceAccount token and pass an explicit gateway base URL with the bearer file:

extraArgs:
  - --openai-base-url=https://llm-gateway.example.com/v1
  - --openai-bearer-token-file=/var/run/secrets/llm-gateway/token
extraVolumes:
  - name: llm-gateway-token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: api://mecak8s-llm-gateway
            expirationSeconds: 600
extraVolumeMounts:
  - name: llm-gateway-token
    mountPath: /var/run/secrets/llm-gateway
    readOnly: true

mecak8s reads the file before every OpenAI request, so token rotation is automatic. Bearer-file mode requires an explicit, nonempty --openai-base-url and never defaults to api.openai.com. The base URL must use HTTPS unless it targets loopback development, and redirects are refused. --openai-bearer-token-file is mutually exclusive with OPENAI_API_KEY.

Choose the gateway's exact audience instead of the default Kubernetes API audience. Configure the gateway to trust the cluster issuer, that audience, and the exact system:serviceaccount:<namespace>:<serviceaccount> subject.

Session affinity is an infrastructure contract​

Official clients send X-Mecatl-Session-ID on session-bound gRPC and HTTP requests when the ID is printable ASCII without surrounding spaces. Duplicate, malformed, and mismatched values are rejected. Existing clients may omit it. Gateways can use this active-session field for consistent routing, but the Kubernetes session lease remains the ownership authority.

Outbound model requests also carry X-Mecatl-Root-Session-ID. The root field stays constant across a main run's subagents, Parallel branches, team members, and delegated-model routing, while X-Mecatl-Session-ID identifies the active child session. Provider logs can therefore group delegated work by its main conversation without losing child-level attribution. The root field is outbound only and must not be used for gateway affinity.

Both fields are correlation metadata. They grant no authentication, authorization, ownership, fencing, idempotency, or cache authority. Lease loss blocks new state mutations on the stale pod, although an already admitted external call can finish. A pending approval stays durable for the successor.

Closing a live running or awaiting session fails precondition and does not release its lease. During shutdown, Mecatl stops admission, preserves pending approvals, settles runs, then releases ownership. If the deadline expires, the lease remains until process death or TTL expiry. After a handoff, the client retries; external model and tool effects are not exactly once.

The chart does not create gateway or session-affinity resources. Its affinity value controls Kubernetes pod scheduling only.

The separate infrastructure PR has a blocking prerequisite before affinity rollout: live validation must show authenticated admission, request and header-size bounds, and client, IP, and principal rate limits apply before or independently of routing on attacker-controlled session IDs.

Connect global MCP servers​

Use mcp.servers for global streaming-HTTP connections. The chart supports no authentication, a bearer from a Kubernetes Secret, or an OAuth profile. It does not accept inline credentials, arbitrary headers, stdio/SSE transports, browser credentials, or a writable credential store.

Global OAuth profiles enforce an exact-origin network policy. Broker OAuth must set additionalOrigins: [], privateOrigins: [], and maxRedirects: 0. Startup performs the final URL, origin, and loopback validation after Helm has validated the values and Secret references.

mcp:
  servers:
    - name: public
      url: https://public-mcp.example/mcp
      auth: { mode: none }
    - name: github
      url: https://github-mcp.example/mcp
      auth:
        mode: staticBearer
        staticBearer:
          secretKeyRef: { name: mecak8s-mcp, key: github-token }

The static token is projected as MCP_GITHUB_TOKEN; it never appears in Helm values, arguments, or a ConfigMap. For a browser-based GitHub OAuth App, use auth.mode: oauth with upstream: {mode: oauth2, oauth2: {authorizationEndpoint, tokenEndpoint}} instead of issuer, and optionally declare a static tools catalog. OAuth uses the per-session broker. One enrollment can cover several protected upstreams; each token is sent only to its configured backend.

Set mcp.broker.callbackURL to the final public HTTPS callback. Route the full /v1/mcp/broker/ prefix to the mecak8s HTTP listener. Broker mode requires OIDC caller identity.

Declared protected tools appear as placeholders before enrollment. Successful enrollment discovers every protected backend and atomically replaces the placeholders with a frozen per-session catalog. Failure exposes no partial catalog. Keep preregistered client secrets in SecretKeyRef; broker metadata and profiles remain non-secret ConfigMap data.

Broker mode is single-replica

Broker sessions and OAuth state are process-local. The chart requires replicaCount: 1 and uses the Recreate strategy when mcp.broker.callbackURL is set. Broker mode does not provide high availability or zero-downtime rollouts.

For a preregistered OAuth client, add this shape to the server entry:

auth:
  mode: oauth
  oauth:
    issuer: https://issuer.example
    client:
      mode: preregistered
      preregistered:
        id: mecak8s
        secretKeyRef: { name: mecak8s-mcp-oauth, key: client-secret }
    scopes: [mcp.read]
    requestRefreshToken: true
    network:
      additionalOrigins: []
      privateOrigins: []
      maxRedirects: 0

Set client.mode: cimd with cimd.documentURL for client ID metadata, or client.mode: dcr with an HTTPS RFC 8414 discovery URL for dynamic client registration. A plain OAuth2 upstream uses explicit authorizationEndpoint and tokenEndpoint values instead of issuer.

Inspect the broker catalogue in mecatui​

On a broker-only mecak8s connection, /mcp shows the owned session's local broker catalogue: enrollment state, connector names, catalogue state, and tool counts. Opening and refresh read only that local state; they do not probe an upstream, refresh credentials, or enroll connectors. When the owner-authorized session is stably idle and the enrollment controller is wired, /mcp (or Ctrl+O) offers c connect tools after a fresh session, completed turns, a prior connection, a failed/terminal attempt, or a broker-process restart. A persisted name or inventory row never proves live connectivity: after restart the panel truthfully reports broker state unavailable and protected tools remain unavailable until the owner explicitly refreshes this same session. Pending setup shows “Setup in progress” and x cancel setup; prompts and a second refresh are blocked until the existing operation settles. The existing browser flow continues without a reopen-browser action. A running or awaiting session does not offer refresh. Setup is destructive and bundle-wide: starting it withdraws broker tools, and cancellation or failure leaves them unavailable. Use /mcp-refresh for this broker flow. /tools-connect remains a deprecated broker-only alias, and /tools-cancel cancels pending setup.

ToolHive remains the sole custodian of upstream OAuth presentation, callback state, credentials, tokens, refresh, and any grant reuse; Mecatl exposes only its opaque enrollment control. Broker OAuth mode remains constrained to one Helm replica (replicaCount: 1); this refresh path is explicit same-session recovery, not automatic recovery, high availability, or multi-replica routing. The panel still requires the existing authenticated verified principal and a matching owned session; broker-only and direct-MCP compositions remain mutually exclusive, so broker-only sessions do not offer direct resources, prompts, or groups.

The panel is not an upstream health check. It requires the authenticated owner of the session. Broker-only sessions do not expose direct MCP resources, prompts, or groups.

Keep MCP and OAuth endpoints on HTTPS and permit their egress through your NetworkPolicy or mesh. insecureHTTP: true is accepted only for non-loopback, non-OAuth HTTP servers and allows a bearer to cross the pod network in cleartext. Use it only for an isolated in-cluster endpoint.

OAuth profile changes trigger a rollout. After changing a static bearer or OAuth client Secret, roll the Deployment and keep both credentials valid during the transition. The chart reserves these environment names: MECATL_INSTALLATION_ID, MECATL_DRIVER_AUTH_TOKEN, and authentication names generated by mcp.servers. Rendering fails when extraEnv collides with one.

Broker authorization storage uses a separate Redis connection that does not reload Redis CA or ACL files. Restart the pod after rotating those files. The main session-store connection can remain healthy, so /readyz does not detect a stale broker connection.

Configure logging​

The chart exposes the process logging threshold separately from metrics and OTLP:

logging:
  level: debug

This renders --log-level=debug, including the embedded ToolHive/authserver/vMCP slog records in the mecak8s container logs. The default empty value preserves the binary's INFO default. Supported values are debug, info, warn, and error. extraArgs remains available for flags that are not modeled by the chart; if it also contains --log-level, its later argument takes precedence.

Configure model recovery​

Use extraArgs to set the server-owned precommit recovery policy. The defaults are --llm-recovery-budget=30m and --llm-max-attempts=60; command-line values override the command defaults.

extraArgs:
  - --llm-recovery-budget=10m
  - --llm-max-attempts=12

The server retries a model step only before semantic output is visible. A visible failure remains terminal, so a model does not rerun completed tool calls or visible assistant text. See provider recovery limits and cost for the per-step billing implications, and LLM resilience for the remaining resilience flags.

Mount trusted skills, agents, and rules​

Set XDG_CONFIG_HOME to the mount root, put files below <root>/mecatl/{skills,agents,rules}, and set skills.autoDiscover: true (default false) to discover skills from the standard XDG locations ($XDG_CONFIG_HOME/mecatl/skills or ~/.config/mecatl/skills, plus ~/.claude/skills) and, when the workspace is trusted, <workspace>/.mecatl/skills and <workspace>/.claude/skills.

Create the referenced ConfigMap in the release namespace first:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mecatl-config-v1
  namespace: mecatl
immutable: true
data:
  review-skill: |
    ---
    name: review
    ---
    Review changes for correctness and security.
  reviewer-agent: |
    ---
    name: reviewer
    ---
    Review the supplied change and report actionable findings.
  base-rules: |
    Keep responses concise and explain material risks.

Then provide the matching Helm values:

extraEnv:
  - name: XDG_CONFIG_HOME
    value: /etc/mecatl-config
skills:
  autoDiscover: true
extraArgs: [--no-user-model]
extraVolumeMounts:
  - { name: mecatl-config, mountPath: /etc/mecatl-config, readOnly: true }
extraVolumes:
  - name: mecatl-config
    configMap:
      name: mecatl-config-v1
      items:
        - { key: review-skill, path: mecatl/skills/review/SKILL.md }
        - { key: reviewer-agent, path: mecatl/agents/reviewer.md }
        - { key: base-rules, path: mecatl/rules/base.md }

Use a read-only mount and preferably an immutable ConfigMap. An immutable ConfigMap cannot be updated: create a new versioned ConfigMap, update its content and the Helm configMap.name reference, then run helm upgrade. Mutable ConfigMap updates also require a Deployment rollout because discovery is snapshotted at startup. --no-user-model is required when this XDG root is read-only because the user model is writable. Setting XDG_CONFIG_HOME also relocates mecatl/settings.yaml, mecatl/soul.md, and mecatl/auth.yaml lookup, so account for those files explicitly.

On Kubernetes 1.36 or newer, a ToolHive-packaged skill can instead be mounted straight from an OCI artifact. Its SKILL.md must be at the artifact root. Mount each artifact at <XDG_CONFIG_HOME>/mecatl/skills/<skill-name> and keep skills.autoDiscover: true:

extraEnv:
  - { name: XDG_CONFIG_HOME, value: /etc/mecatl-config }
skills:
  autoDiscover: true
extraArgs: [--no-user-model]
extraVolumeMounts:
  - name: review-skill
    mountPath: /etc/mecatl-config/mecatl/skills/review
    readOnly: true
extraVolumes:
  - name: review-skill
    image:
      reference: registry.example/skills/review@sha256:<digest>
      pullPolicy: IfNotPresent

Image volumes are inherently read-only, and the pod's imagePullSecrets apply to artifact pulls normally. Use a digest-pinned reference in production; it remains immutable and IfNotPresent may safely use the node cache. A mutable tag with IfNotPresent may also reuse cached content; use Always if every pod start must resolve that tag from the registry. Mecatl snapshots skill metadata and body at startup, so any artifact change requires pod recreation.

Verify a mounted skill in a real session; GET /v1/skills proves only metadata discovery. A test skill that writes and reads a unique marker should produce Skill, Write, and Read calls in that order, followed by the marker.

Use Helm 3.16 or newer when adding an image volume to an existing release. Older clients can render the YAML but may not know the image field when they calculate an upgrade patch.

Server TLS​

Server TLS is independent of Redis TLS. It is off by default and uses an operator-created, same-namespace kubernetes.io/tls Secret. The quick start creates this Secret and enables these values:

tls:
  enabled: true
  secretName: mecak8s-tls

The chart creates no Secret. With tls.enabled=true, it projects only tls.certKey and tls.keyKey from that Secret, read-only with mode 0440. They default to the standard tls.crt and tls.key data keys; set the values when your Secret uses different PEM key names. The container receives the fixed mounted paths /var/run/secrets/tls/<certKey> and /var/run/secrets/tls/<keyKey> as --tls-cert and --tls-key, enabling TLS for both gRPC and HTTP/SSE. The chart also changes health and readiness requests to HTTPS; the Pod-only drain listener remains plaintext HTTP on port 8082. This is the in-pod TLS + OIDC secure real-provider posture; include the OIDC values shown above for a real provider. For an operator-owned edge TLS boundary instead, set security.tlsTerminatedUpstream=true with OIDC. Keeping tls.enabled=true is valid re-encryption and preserves that upstream attestation; setting it false selects the ClusterIP-only plaintext h2c backend, which must be reachable only from the gateway or mesh. Rotated certificate/key pairs are loaded transactionally for new handshakes without a rollout; invalid candidates retain the last valid generation.

Rotate server TLS certificates​

When --tls-cert and --tls-key point into a Kubernetes projected Secret, mecak8s watches their parent directories and reloads a complete matching pair after the projection settles. A malformed or mismatched intermediate generation is rejected and the last valid certificate continues serving. Existing connections are unaffected; new TLS handshakes use the replacement without a pod restart. A fixed internal observer emits a bounded warning once for a certificate generation that becomes expiring or expired; observation does not disable the last-valid certificate. --client-ca is intentionally static and still requires a pod restart to change the trusted client identities.

Secure Redis credentials and TLS​

Redis credentials reach mecak8s as paths to files projected from a Kubernetes Secret volume — never as container arguments, environment variables, or ConfigMap entries. Mount the Secret read-only (defaultMode: 0440 is a good default), then point the flags at the mounted paths:

--redis-url=redis.example.internal:6379                     # bare host:port, never a redis:// URL
--redis-username-file=/var/run/secrets/redis/username       # optional ACL username
--redis-password-file=/var/run/secrets/redis/password       # optional ACL password
--redis-tls-ca=/var/run/secrets/redis/ca.pem                # private CA...
--redis-tls                                                 # ...or verify against the system trust store

Every credential requires verified TLS, from one of two sources: the host's system trust store (--redis-tls) for a managed Redis whose certificate chains to a public CA, or a mounted PEM CA bundle (--redis-tls-ca) for a private one. --redis-tls-ca replaces the system trust store rather than adding to it. Both modes verify the server certificate against the hostname in --redis-url (including IP SAN rules); hostname verification is never disabled and TLS 1.2 or newer is required. TLS with no ACL is valid. ACL is optional: a password without a username uses Redis's default ACL user, while a username requires a password.

For a managed Redis certificate that chains to a public CA, set redis.caKey: "". The chart then uses the system trust store and does not mount a Redis Secret volume when the username and password keys are also empty.

mecak8s watches the lexical parent directories of every configured Redis CA, username, and password file, so Kubernetes projected-Secret ..data swaps are observed. One coalesced event re-reads the complete configured file set. The process builds fresh durability and follow clients through the same validation and verified-TLS path, and publishes the pair only after bounded successful PING/TLS/auth probes. Invalid or partially projected material leaves the last valid client active; bounded single-flight retries cover the window where the Secret projection and Redis-side ACL/trust update settle in different orders. New operations use the replacement pair, while in-flight operations and migration locks finish on their original durability client before it closes. No Redis files configured means no reload watcher. Credential files may end in one newline, as Kubernetes Secret projections commonly do; other whitespace remains part of the credential.

--redis-url takes a bare host:port. A redis:// or rediss:// URL is rejected on every path, plaintext included, and the rejection never repeats the address back — a URL's userinfo can carry a password, and these errors land in the operator's log.

Client-certificate (mTLS) authentication is not supported: the shared toolhive-core/redisconn connection layer cannot express it (ADR 0233), and it is tracked upstream at toolhive-core#240.

Plaintext Redis is fixture-only

An address-only --redis-url is plaintext and unauthenticated, and is rejected at startup unless you also pass --redis-allow-plaintext. That opt-in exists for the disposable values-kind.yaml profile only (its in-chart Redis fixture, with the mock provider and no auth). Production installs must not set it: use verified TLS, and add ACL credentials when the managed Redis service requires them.

The Helm chart's topology​

The chart creates these resources:

ResourceBehavior
ServiceAccount, Role, and RoleBindingGrants only the required Lease verbs.
DeploymentRuns two storage-free replicas by default; one replica is supported.
ConfigMapStores the non-secret installation UUID for telemetry.
ClusterIP ServiceExposes gRPC on 8080 and HTTP/SSE on 8081.
PodDisruptionBudgetUses maxUnavailable: 1 for two or more replicas and is omitted for one.
Raw-driver NetworkPolicyCreated only with OIDC and limits raw-driver ingress to agent pods.
Local Redis fixtureCreated only by the disposable values-kind.yaml profile.

The chart creates no namespace or general NetworkPolicy. Supply network policy for your provider, MCP, Redis, identity-provider, and Kubernetes API traffic.

Deployment details:

  • Two replicas use RollingUpdate, maxSurge: 1, and maxUnavailable: 0.
  • Pods prefer separate nodes through a soft hostname topology-spread constraint; single-node clusters remain schedulable.
  • terminationGracePeriodSeconds defaults to 60 seconds. The schema requires at least 44 seconds, the first whole second above the 43-second default shutdown budget.
  • No PVC, no --store-dir. The only volumeMount is /tmp for the Go runtime and SSE buffering under readOnlyRootFilesystem: true.
  • A preStop hook calls the Pod-only plaintext drain listener on port 8082 and waits about three seconds for endpoint propagation. Restrict direct Pod-IP access to that port.
  • Pods run as non-root with privilege escalation disabled, all capabilities dropped, and RuntimeDefault seccomp.

With replicaCount: 1, a voluntary disruption can evict the only pod. Redis preserves persisted sessions but cannot preserve availability.

Readiness and health​

mecak8s exposes two unauthenticated probe endpoints on the HTTP port (default 0.0.0.0:8081). A separate plaintext, Pod-only drain listener defaults to 0.0.0.0:8082 and serves only GET /drain:

EndpointPurpose
GET /healthzLiveness — returns 200 unless the process is hung
GET /readyzReadiness — returns 200 only when !draining && redisOK; flips to 503 on drain or Redis failure
GET /drain on port 8082preStop hook target — arms the drain gate, blocks ~3s for endpoint propagation, returns 200

The Service exposes only ports 8080 and 8081, so normal Service/gateway API traffic cannot invoke /drain. Direct Pod-IP access to 8082 remains an operator network-isolation responsibility. The readyz probe is dynamic: it calls svc.StorageReady, which pings the Redis store with a 2-second timeout. A Redis failure shows up as not-ready and removes the pod from Service endpoints without a restart. The startup and readiness probes use a 3-second kubelet timeout so the 2-second Redis bound can complete before Kubernetes abandons the request.

Graceful shutdown​

The shutdown sequence on SIGTERM (or when the kubelet calls GET /drain via the preStop hook) is:

The shutdown budget is 43 seconds: three seconds for endpoint propagation, 15 for run drain, 10 for gRPC, and five each for HTTP, resource closure, and telemetry. The Helm terminationGracePeriodSeconds default is 60. Increase it when you increase any runtime bound; the schema's 44-second minimum covers only the defaults. If a bound expires, the server cancels in-flight runs; a successor can recover them from Redis.

A surviving pod can acquire a gracefully released lease immediately. After a hard stop, it must wait for the 30-second default lease TTL.

Long-running model calls are cancelled when the shutdown bound expires. The successor recovers the session and previous allow-always decisions from Redis.

Verify session failover​

Use the disposable cluster from Try Mecatl on Kubernetes, which runs two ready replicas without production authentication. Get both pod names and forward the first pod's HTTP port:

POD_A=$(kubectl get pods -n mecatl -l app.kubernetes.io/component=agent -o jsonpath='{.items[0].metadata.name}')
POD_B=$(kubectl get pods -n mecatl -l app.kubernetes.io/component=agent -o jsonpath='{.items[1].metadata.name}')

kubectl port-forward -n mecatl "pod/$POD_A" 8081:8081 &

Create a session and send a prompt through pod A:

SESSION_ID=$(curl -s -X POST http://127.0.0.1:8081/v1/sessions \
  -d '{"mode":"default"}' | jq -r .session_id)

curl -s -X POST "http://127.0.0.1:8081/v1/sessions/$SESSION_ID/prompt" \
  -H 'Accept: text/event-stream' -d '{"text":"say hello"}' > /dev/null

Run this command before deleting pod A, then again after pod B returns 200, to see the Lease holder change:

kubectl get lease --namespace mecatl --output wide

Delete pod A gracefully, then send another prompt through pod B:

kubectl delete pod -n mecatl "$POD_A"
kubectl port-forward -n mecatl "pod/$POD_B" 8082:8081 &

curl -s -X POST "http://127.0.0.1:8082/v1/sessions/$SESSION_ID/prompt" \
  -H 'Accept: text/event-stream' -d '{"text":"are you still there?"}'

The request returns 200 with the previous conversation intact.

For comparison, select the replacement pod, forward it, and then force-delete the current holder:

POD_C=$(kubectl get pods -n mecatl \
  -l app.kubernetes.io/component=agent \
  --field-selector="metadata.name!=$POD_B,status.phase=Running" \
  -o jsonpath='{.items[0].metadata.name}')
kubectl wait -n mecatl --for=condition=Ready "pod/$POD_C" --timeout=120s
kubectl port-forward -n mecatl "pod/$POD_C" 8083:8081 &

kubectl delete pod -n mecatl "$POD_B" --force --grace-period=0

curl -i -X POST "http://127.0.0.1:8083/v1/sessions/$SESSION_ID/prompt" \
  -H 'Accept: text/event-stream' -d '{"text":"retry after hard failure"}'

Pod C returns 409 with "leased by another process" until the 30-second default lease TTL expires. Retry the request after the TTL to confirm that it returns 200 with the previous conversation. Run task e2e:k8s for the scripted checks.

Configure caller identity​

Enable OIDC to authenticate every request and isolate sessions, schedules, teams, and memory by the verified (issuer, subject) owner. See Caller identity and OIDC for the shared behavior and client workflows.

Check existing data first​

Enabling OIDC changes access, not just attribution:

  • Callers can act on and list only their own sessions, schedules, teams, memory, and event streams. Foreign resources appear absent.
  • Different callers can use the same memory keys and schedule names without sharing data.
  • Existing ownerless records become unavailable. Export or back them up before enabling ownership; Mecatl does not adopt them.

OIDC does not isolate the shared pod filesystem. Isolate caller workspaces at the deployment layer. Raw storage drivers also remain trusted infrastructure, so keep them inaccessible to tenants.

Before admitting callers: inventory ownerless records​

Stage the OIDC deployment with tenant traffic blocked and configure one exact storage_management.principals issuer/subject pair for the operator. Call the authenticated GetStorageHealth RPC (or GET /v1/storage/health) as that principal and inspect the ownerless_session_count / ownerless_session_ids and ownerless_schedule_count / ownerless_schedule_names fields. The identifiers are bounded samples (the corresponding *_truncated bit says when the count is larger); no transcript, memory value, or schedule prompt is returned. If either *_available field is false, do not proceed until that backend exposes the required metadata inventory.

After ownership is enabled, retention and scheduling skip ownerless records. Disabling enforcement makes them accessible again but does not assign an owner or replay skipped work.

Validator and bounded signing-key cache​

The production OIDC/JWT validator is a delegated, actively-maintained library — Mecatl never hand-rolls token verification. A bad OIDC configuration, including an unreachable initial key fetch, fails closed at startup rather than serving unauthenticated traffic. After a successful fetch, the last good JWKS can cover a short IdP outage. The chart's default oidc.maxJWKSStaleness sets --oidc-max-jwks-staleness=1h: once keys are older than that, the validator refreshes before deciding and returns 503 Service Unavailable when it cannot obtain current keys. A bad, expired, wrong-issuer, or wrong-audience token remains 401. Set the flag to 0 only to deliberately accept unbounded cached-key availability and its signing-key revocation exposure.

This bounds signing-key revocation exposure during an IdP outage; it does not provide per-token revocation before normal token expiry. The JWKS cache is process-local and unpersisted, so a restarted pod fetches current keys again.

Enable OIDC​

Add the OIDC settings to your Helm values:

oidc:
  enabled: true
  issuer: https://idp.example.com/realms/mecatl
  audience: mecatl
ValueFlagMeaning
oidc.issuer--oidc-issueryour IdP's issuer URL, compared byte-exact against the token's iss
oidc.audience--oidc-audiencethe audience this deployment accepts. Required — an audience-less verifier accepts tokens minted for a different service
oidc.jwksURI--oidc-jwks-urioptional. Pin the signing-key endpoint and skip discovery. Only for an air-gapped or pinned-key deployment; leave it out and the issuer's discovery document is used
oidc.maxJWKSStaleness (default 1h)--oidc-max-jwks-stalenessMaximum time a last-good JWKS remains trusted when refresh cannot reach the IdP. 0 deliberately disables the bound.

Use an HTTPS issuer. The validator rejects JWKS endpoints that resolve to private, loopback, link-local, or metadata addresses unless you set oidc.allowPrivateHTTPSIssuer: true and identify the CA with oidc.caSecret and oidc.caKey.

Enabling OIDC creates a narrow NetworkPolicy for pods labelled as raw storage drivers. It does not create a general workload NetworkPolicy. Keep raw driver endpoints off tenant-facing Services and route callers through the authenticated mecak8s Service.

To let remote mecatui discover login settings, configure the externally reachable RFC 9728 resource URL, public client ID, and optional scopes:

oidc:
  resource: https://mecatl.example.com
  clientID: mecatui
  scopes: [openid, profile, offline_access]

The chart rejects partial profiles and never derives the public resource URL from pod addresses. Metadata discovery is anonymous HTTPS and remains separate from authenticated gRPC transport.

Use it from the TUI​

For a static bearer, pass a token from your IdP with --auth-token or MECATL_AUTH_TOKEN. mecatui does not refresh a static bearer. For managed OIDC, run mecatui login ADDRESS with the issuer, client ID, and audience, then run mecatui connect ADDRESS. Add --tls-ca and --private-issuer for an approved private issuer. Login stores and refreshes the encrypted credential; connect never starts login implicitly.

For a development port-forward, bearer traffic stays on loopback:

kubectl port-forward -n mecatl service/mecak8s-mecak8s 8080:8080 &
export MECATL_AUTH_TOKEN="$(your-oidc-cli print-access-token)"
mecatui connect 127.0.0.1:8080 --auth-token "$MECATL_AUTH_TOKEN"

To verify authentication, remove the environment fallback and submit a prompt:

env -u MECATL_AUTH_TOKEN \
  mecatui connect 127.0.0.1:8080

A gRPC connection can succeed before authentication. The first request must fail before the model responds.

For a non-loopback server, configure the workspace on mecak8s; mecatui rejects --workspace. Use --tls and, for a private server CA, connect --tls-ca. The issuer CA used during login is separate. mecatui rejects a bearer over non-loopback cleartext.

After creating a session in the TUI, verify the saved owner through the HTTP API (port-forward 8081:8081 as well if needed):

curl -s http://127.0.0.1:8081/v1/sessions \
  -H "Authorization: Bearer $MECATL_AUTH_TOKEN"

The returned session row contains the verified owner.

Troubleshooting: start here​

Two IdP misconfigurations account for most first-deployment 401s, and neither produces a helpful error:

  1. The audience is not in the token. Keycloak, for example, puts only account in aud by default; your client id appears only if you attach an Audience protocol mapper. Then --oidc-audience=<your-client-id> never matches and every caller gets 401. Decode a token and check aud before anything else.
  2. The issuer string does not match. iss is compared byte-exact, and an IdP stamps whatever external hostname it is configured to advertise — regardless of how your pods reach it. Take --oidc-issuer from the IdP's /.well-known/openid-configuration, never from the in-cluster Service URL.

Beyond that: a 401 means the credential was rejected; a 503 means a required JWKS refresh could not obtain current keys after the configured staleness bound. Before that bound, a last-good JWKS can keep validation available during a short IdP outage. Authn failures are not currently logged, so neither is visible in the agent's output — you will see the status code and nothing else.

Security boundaries​

Filesystem isolationOwnership does not isolate a shared pod filesystem. Use separate workspaces or deployments for tenants that must not share files.
Raw storage driversDrivers are trusted infrastructure and do not authenticate end users. Restrict them to agent workloads and keep them off public Services.
Existing ownerless dataEnabling ownership makes it unavailable. Export or back it up before enabling OIDC.
Signing-key revocationThe default one-hour JWKS staleness bound permits cached signing keys during a short IdP outage. Tokens remain valid until their own expiry.
Rate limits and quotasmecak8s has no rate-limit flags. Enforce caller limits at the gateway or mesh.
Store confidentialityProduction Redis requires verified TLS. Client-certificate authentication is not supported.
Identity dataThe token's name claim, often an email address, is stored unredacted in session snapshots, schedule records, and durable event actor metadata. Configure retention and deletion accordingly.
Authentication visibilityMecatl emits no authentication metric and does not log authentication failures. The caller sees the 401 or 503 response.

Scaling​

The chart defaults to two replicas for HA-oriented operation, but replicaCount: 1 is supported when lower resource usage and simpler session routing are preferred. In single-replica mode there is no failover and the chart omits the PDB, so voluntary eviction can interrupt service; Redis preserves successfully persisted state, not availability.

Kubernetes Leases enforce one writer per session. A second pod receives HTTP 409 or gRPC FAILED_PRECONDITION. Lease renewal defaults to one-third of --session-lease-ttl.

Session affinity is optional. A client that reaches a pod without the lease receives 409 and retries another replica or waits for the current run.

For two or more replicas, the PodDisruptionBudget defaults to maxUnavailable: 1, so voluntary disruptions remove at most one replica as the Deployment scales. Set podDisruptionBudget.maxUnavailable to a non-negative integer or percentage, or set podDisruptionBudget.enabled: false when another operator owns disruption policy.

The default topologySpreadConstraints softly prefer separate kubernetes.io/hostname values and therefore keep single-node clusters schedulable. Production deployments can replace ScheduleAnyway with DoNotSchedule and add a zone-level constraint when the cluster topology can satisfy them. The chart also supports affinity, nodeSelector, and tolerations values.

Use Redis Sentinel, Redis Cluster, or a managed service for production high availability. The disposable in-cluster Redis fixture has one replica and no persistence.

Differences from mecated​

Unlike mecated, mecak8s requires Redis, enables Kubernetes session leases by default, and exposes --redis-url. It does not include the config, skills promote, perf-mcp, or ACP operator commands.

mecak8s does not support the local ChatGPT Codex subscription entry at providers.openai-codex.oauth. API-key entries in auth.yaml remain supported. Use an API-key provider instead of mounting a local Codex OAuth credential.

What's next​