Skip to content
Documentation
Operating on Frank
Operating on CI/CD Platform
Operating on CI/CD Platform

Operating on CI/CD Platform

Companion operations guide for CI/CD Platform — Gitea, Tekton, Zot, and Cosign.

Quick Health Check 

# All CI/CD pods
kubectl get pods -n gitea -o wide
kubectl get pods -n tekton-pipelines -o wide
kubectl get pods -n zot -o wide

# ArgoCD app status
kubectl get applications -n argocd gitea gitea-extras \
  tekton-pipelines tekton-triggers tekton-dashboard tekton-extras \
  zot zot-extras

# ExternalSecrets sync status
kubectl get externalsecret -n gitea
kubectl get externalsecret -n tekton-pipelines
kubectl get externalsecret -n zot

Healthy state: all pods Running on pc-1, all ArgoCD apps Synced/Healthy, all ExternalSecrets SecretSynced.

$ kubectl get pods -n gitea -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE   NOMINATED NODE   READINESS GATES
gitea-6d7d457c49-t5hsj   1/1     Running   0          7h40m   10.244.7.57   pc-1   <none>           <none>

$ kubectl get pods -n tekton-pipelines -o wide
NAME                                                 READY   STATUS    RESTARTS        AGE     IP              NODE     NOMINATED NODE   READINESS GATES
el-gitea-listener-7d85fd8b75-8w9z6                   1/1     Running   3 (7h41m ago)   7d7h    10.244.7.226    pc-1     <none>           <none>
tekton-dashboard-774bff7cc-92cmn                     1/1     Running   0               22d     10.244.8.175    mini-3   <none>           <none>
tekton-events-controller-5cbc777ccd-7gvq7            1/1     Running   0               22d     10.244.8.144    mini-3   <none>           <none>
tekton-pipelines-controller-7496c46798-kgphw         1/1     Running   0               7d10h   10.244.12.233   mini-1   <none>           <none>
tekton-pipelines-webhook-75cd84877-tgctj             1/1     Running   0               22d     10.244.12.139   mini-1   <none>           <none>
tekton-triggers-controller-66fd74568d-m8zsv          1/1     Running   0               22d     10.244.8.254    mini-3   <none>           <none>
tekton-triggers-core-interceptors-66456f8cf6-blt4s   1/1     Running   0               22d     10.244.8.181    mini-3   <none>           <none>
tekton-triggers-webhook-55c8dd895f-j95ft             1/1     Running   0               22d     10.244.8.187    mini-3   <none>           <none>

$ kubectl get pods -n zot -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP             NODE   NOMINATED NODE   READINESS GATES
zot-68c79b95f9-6vbhh   1/1     Running   0          7h41m   10.244.7.202   pc-1   <none>           <none>

Gitea Operations

Mirror Sync Status 

# Check mirror sync time via API
GITEA_URL="http://192.168.55.209:3000"
curl -s "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{
  mirror: .mirror,
  updated_at: .updated_at,
  mirror_interval: .mirror_interval
}'

Expected: mirror: true, updated_at within the last 10 minutes.

Force Mirror Sync 

# Trigger immediate mirror sync
ADMIN_TOKEN=$(kubectl get secret -n gitea gitea-secrets -o jsonpath='{.data.admin-password}' | base64 -d)
curl -sf -X POST "$GITEA_URL/api/v1/repos/tekton-bot/frank/mirror-sync" \
  -H "Authorization: token $ADMIN_TOKEN"

Gitea Logs 

kubectl logs -n gitea deploy/gitea --tail=50
kubectl logs -n gitea deploy/gitea -f  # Follow

Restart Gitea 

kubectl rollout restart deploy/gitea -n gitea
kubectl rollout status deploy/gitea -n gitea

Note: Gitea uses strategy: Recreate (RWO PVC) — expect a brief downtime window during restart.

Common Gitea Issues

SymptomCauseFix
OIDC login failsAuthentik provider misconfigured or downCheck kubectl logs -n gitea deploy/gitea | grep oauth
Mirror not updatingGitHub PAT expired or rate-limitedVerify GITHUB_MIRROR_TOKEN in Infisical is valid
Webhook delivery failsALLOWED_HOST_LIST missing cluster DNSAdd *.svc.cluster.local to gitea.config.webhook.ALLOWED_HOST_LIST
Pod stuck in PendingPVC can’t mount (pc-1 down)Check kubectl get pv and node status

Tekton Operations

List Recent PipelineRuns 

# All PipelineRuns, most recent first
kubectl get pipelinerun -n tekton-pipelines --sort-by=.metadata.creationTimestamp

# Only failed runs
kubectl get pipelinerun -n tekton-pipelines \
  -o jsonpath='{range .items[?(@.status.conditions[0].status=="False")]}{.metadata.name}{"\t"}{.status.conditions[0].message}{"\n"}{end}'

View Pipeline Logs 

# Latest PipelineRun logs (requires tkn CLI)
tkn pipelinerun logs -n tekton-pipelines --last

# Without tkn — find the pod and read logs per step
kubectl get pods -n tekton-pipelines -l tekton.dev/pipelineRun --sort-by=.metadata.creationTimestamp | tail -5
kubectl logs -n tekton-pipelines <pod-name> -c step-clone
kubectl logs -n tekton-pipelines <pod-name> -c step-test
kubectl logs -n tekton-pipelines <pod-name> -c step-build-and-push
kubectl logs -n tekton-pipelines <pod-name> -c step-sign

Cancel a Running PipelineRun 

kubectl patch pipelinerun -n tekton-pipelines <name> \
  --type=merge -p '{"spec":{"status":"CancelledRunFinally"}}'

Clean Up Old PipelineRuns 

# Delete PipelineRuns older than 7 days
kubectl get pipelinerun -n tekton-pipelines \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}' \
  | awk -v cutoff="$(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ)" '$2 < cutoff {print $1}' \
  | xargs -r kubectl delete pipelinerun -n tekton-pipelines

EventListener Health 

# EventListener pod logs
kubectl logs -n tekton-pipelines -l app.kubernetes.io/managed-by=EventListener --tail=30

# EventListener service reachability (from within cluster)
kubectl run test-curl --rm -it --image=curlimages/curl -- \
  curl -s -o /dev/null -w "%{http_code}" \
  http://el-gitea-listener.tekton-pipelines.svc.cluster.local:8080
# Expect: 200 or 202

Tekton Dashboard

Access at http://192.168.55.217:9097 or https://tekton.cluster.derio.net (Authentik forward-auth).

The dashboard is read-only — it shows PipelineRuns, TaskRuns, and logs. Useful for non-CLI users or quick visual debugging.

Common Tekton Issues

SymptomCauseFix
PipelineRun stuck in PendingPVC can’t provision (pc-1 down or Longhorn unhealthy)Check kubectl get pv and Longhorn UI
ComparisonError in ArgoCDTask YAML uses resources instead of computeResourcesFix the Task manifest and re-sync
report-success not firing$(tasks.status) is "Completed" not "Succeeded"Check when clause accepts both values
Step fails with permission deniedMissing fsGroup: 65534 on PipelineRun pod templateAdd to TriggerTemplate’s taskRunTemplate.podTemplate.securityContext
git config failsHOME=/ is read-only for UID 65534Set HOME=/tekton/home env var on the step
PodSecurity violationMissing securityContext on Task stepsAdd runAsNonRoot, capabilities.drop, seccompProfile
Webhook doesn’t trigger pipelineGitea sends X-Gitea-Event, not X-GitHub-EventUse CEL interceptor, not github interceptor

Zot Registry Operations

Registry Health 

# API check (anonymous read)
curl -sk https://192.168.55.210:5000/v2/
# Expect: {} or {"repositories":[]}

# List all repositories
curl -sk https://192.168.55.210:5000/v2/_catalog
# Expect: {"repositories":["test/alpine","..."]}

# List tags for a repo
curl -sk https://192.168.55.210:5000/v2/test/alpine/tags/list

Test Push 

# Push a test image (requires credentials)
ZOT_PASS=$(kubectl get secret -n tekton-pipelines zot-push-creds \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths["192.168.55.210:5000"].password')

# Using crane (lightweight OCI tool)
crane auth login 192.168.55.210:5000 -u tekton-push -p "$ZOT_PASS" --insecure
crane push /dev/null 192.168.55.210:5000/test/healthcheck:latest --insecure

TLS Certificate Status 

# cert-manager Certificate status
kubectl get certificate -n zot zot-tls
# Expect: Ready=True

# Certificate details
kubectl describe certificate -n zot zot-tls

# Check cert expiry
kubectl get secret -n zot zot-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

Registry Logs 

kubectl logs -n zot deploy/zot --tail=50

Restart Zot 

kubectl rollout restart deploy/zot -n zot
kubectl rollout status deploy/zot -n zot

Like Gitea, Zot uses strategy: Recreate for RWO PVC safety.

Common Zot Issues

SymptomCauseFix
401 Unauthorized on pushWrong password or htpasswd hash staleRegenerate htpasswd if ZOT_PUSH_PASSWORD changed in Infisical
x509: certificate signed by unknown authoritySelf-signed cert not trusted by clientUse --insecure flag or add cert to trust store
Containerd pull fails on nodesTalos mirror patch not appliedApply patches/phase06-cicd/06-cluster-zot-registry.yaml via Omni
Pod stuck (PVC)pc-1 down or Longhorn unhealthyCheck node and Longhorn status

Cosign Operations

Verify an Image Signature 

cosign verify --key apps/tekton/cosign.pub \
  --insecure-ignore-tlog --allow-insecure-registry \
  192.168.55.210:5000/<repo>/<image>:<tag>

Expected: Verification for 192.168.55.210:5000/... -- The following checks were performed: ...

Check if an Image Is Signed 

# List signature artifacts for an image
crane ls 192.168.55.210:5000/<repo>/<image> --insecure
# Look for sha256-*.sig tags

Rotate Signing Keys

  1. Generate new key pair: COSIGN_PASSWORD='' cosign generate-key-pair
  2. Store new private key in Infisical as COSIGN_KEY
  3. Commit new cosign.pub to apps/tekton/cosign.pub
  4. Wait for ExternalSecret to refresh (5 minutes) or force sync
  5. New images will be signed with the new key; old signatures remain valid with old public key

End-to-End Webhook Test

To verify the full pipeline chain from Gitea to signed image:

# 1. Trigger a mirror sync (or push directly to Gitea)
curl -sf -X POST "http://192.168.55.209:3000/api/v1/repos/tekton-bot/frank/mirror-sync" \
  -H "Authorization: token $ADMIN_TOKEN"

# 2. Watch for new PipelineRun
kubectl get pipelinerun -n tekton-pipelines -w

# 3. Check Gitea commit status (after pipeline completes)
curl -s "http://192.168.55.209:3000/api/v1/repos/tekton-bot/frank/statuses/<SHA>" \
  -H "Authorization: token $TOKEN" | jq '.[0].state'
# Expect: "success"

References

Operating on VK RelayOperating on ArgoCD Drift