Skip to content
Documentation
Operating on Frank
Operating on Multi-tenancy
Operating on Multi-tenancy

Operating on Multi-tenancy

This is the operational companion to Multi-tenancy — Disposable Kubernetes Clusters with vCluster. That post covers the architecture and template pattern. This one is the day-to-day runbook for creating, connecting to, and troubleshooting virtual clusters.

What “Healthy” Looks Like

Multi-tenancy is healthy when all vCluster StatefulSets are running, each virtual cluster’s API server is responding, and resources are syncing correctly between virtual and host clusters. The ArgoCD applications for each vCluster should show Synced and Healthy.

Observing State

List Virtual Clusters 

# Using vcluster CLI
vcluster list

# Check vCluster pods on the host
kubectl get pods -A -l app=vcluster

# Check ArgoCD status for vCluster apps
argocd app list --port-forward --port-forward-namespace argocd | grep vcluster
$ vcluster list
  
       NAME     |      NAMESPACE       | STATUS  | VERSION | CONNECTED | AGE  
  --------------+----------------------+---------+---------+-----------+------
    experiments | vcluster-experiments | Running | 0.32.1  |           | 39d  
  

$ kubectl get statefulset -A -l app=vcluster
NAMESPACE              NAME          READY   AGE
vcluster-experiments   experiments   1/1     39d

Connect to a Virtual Cluster 

# Connect and switch kubectl context
vcluster connect <name> --namespace <namespace>

# Verify you are in the virtual cluster
kubectl get nodes
kubectl get namespaces

# Disconnect when done
vcluster disconnect

Check Synced Resources 

# From the host cluster, check what the syncer is doing
kubectl logs -n <vcluster-namespace> <vcluster-pod> -c syncer --tail=50

# Check resource sync status
kubectl get pods -n <vcluster-namespace> -l vcluster.loft.sh/managed-by

Routine Operations

Create a New Virtual Cluster

New vClusters follow the template pattern:

  1. Copy the template values:
cp -r apps/vclusters/template/ apps/vclusters/<new-name>/

  1. Customize apps/vclusters/<new-name>/values.yaml — set the name, resource quotas, and any specific configuration.

  2. Add the ArgoCD Application CR in apps/root/templates/vcluster-<new-name>.yaml following the existing pattern.

  3. Commit and push — ArgoCD picks it up automatically.

Delete a Virtual Cluster

Since prune: false, removing a vCluster requires manual cleanup:

# Delete the ArgoCD application
argocd app delete vcluster-<name> --port-forward --port-forward-namespace argocd

# Verify the namespace is cleaned up
kubectl get ns <vcluster-namespace>

Then remove the files from apps/vclusters/<name>/ and apps/root/templates/vcluster-<name>.yaml.

Manage Resource Quotas 

# Check current quotas from inside the virtual cluster
vcluster connect <name> --namespace <namespace>
kubectl get resourcequota -A

# Check host-level resource usage for the vCluster namespace
vcluster disconnect
kubectl top pods -n <vcluster-namespace>

To adjust quotas, update the values in apps/vclusters/<name>/values.yaml and let ArgoCD sync.

Debugging

Virtual API Server Not Responding 

# Check the StatefulSet
kubectl get statefulset -n <vcluster-namespace>
kubectl describe statefulset -n <vcluster-namespace> <vcluster-name>

# Check the PVC for the virtual etcd
kubectl get pvc -n <vcluster-namespace>

# Check pod logs
kubectl logs -n <vcluster-namespace> <vcluster-pod> -c vcluster --tail=100

Common causes:

  • PVC stuck pending (storage class issue)
  • Resource limits too low for the API server
  • Host node where the StatefulSet is scheduled is under pressure

Resources Not Syncing 

# Check syncer logs for errors
kubectl logs -n <vcluster-namespace> <vcluster-pod> -c syncer --tail=100 | grep -i error

# Verify sync configuration in values.yaml
kubectl get configmap -n <vcluster-namespace> -l app=vcluster -o yaml | grep -A 20 sync

Resource Quota Exceeded 

# Connect to the virtual cluster and check
vcluster connect <name> --namespace <namespace>
kubectl describe resourcequota -A

# Check which pods are consuming the most
kubectl top pods -A --sort-by=memory
vcluster disconnect

Network Connectivity Issues 

# Test DNS from inside the virtual cluster
vcluster connect <name> --namespace <namespace>
kubectl run test --image=busybox --rm -it --restart=Never -- nslookup kubernetes.default

# Test connectivity to a host service
kubectl run test --image=busybox --rm -it --restart=Never -- wget -qO- http://<host-service>
vcluster disconnect

Quick Reference

CommandWhat It Does
vcluster listList all virtual clusters
vcluster connect <name> -n <ns>Switch kubectl to virtual cluster
vcluster disconnectReturn to host cluster context
kubectl get pods -A -l app=vclusterCheck vCluster pods on host
kubectl logs -n <ns> <pod> -c syncerCheck resource sync logs
kubectl logs -n <ns> <pod> -c vclusterCheck virtual API server logs
kubectl get pvc -n <ns>Check virtual etcd storage
argocd app list | grep vclusterArgoCD status for vClusters

References

Operating on AuthenticationOperating on Media Generation