AI-Powered Kubernetes Troubleshooting with K8sGPT

Kubernetes clusters are complex, generating vast amounts of logs and events that are difficult to parse manually. k8sgpt empowers SREs and devops engineers by automating root cause analysis through AI, reducing mean-time-to-resolution (MTTR) considerably.

🎞️ Watch Video

Prerequisites:

• Kubernetes cluster
• Helm 3.x 0r 4.x
• kubectl configured

Deploy Ollama LLM in Kubernetes

Create Namespace and Storage

kubectl create namespace ollama

Apply:

kubectl apply -f ollama/ollama-pvc.yaml

Deploy Ollama

kubectl apply -f ollama/ollama-deployment.yaml

# Wait for pod to be ready
kubectl wait --for=condition=ready pod -l app=ollama -n ollama --timeout=300s

Load Model into Ollama

# Get pod name
POD_NAME=$(kubectl get pod -n ollama -l app=ollama -o jsonpath='{.items[0].metadata.name}')

#To load a model into Ollama, simply use
kubectl exec -n ollama $POD_NAME -- ollama pull gemma3:4b

# Verify model is loaded
kubectl exec -n ollama $POD_NAME -- ollama list

Test Ollama Service

kubectl port-forward -n ollama svc/ollama 11434:11434

In another terminal, test the API

curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b",
  "prompt": "Explain crewai in one sentence",
  "stream": false
}'

Install K8sGPT CLI

# Using curl
curl -sSfL https://raw.githubusercontent.com/k8sgpt-ai/k8sgpt/main/install.sh | sh

k8sgpt --help

Configure k8sgpt to use LocalAI using Ollama backend

k8sgpt auth add --backend localai --model mistral --baseurl http://localhost:11434/v1

k8sgpt auth default -p localai

k8sgpt auth list

Install a malicious Pod

kubectl run webapp --image=ngix:latest

Analyze your cluster by using k8s

k8sgpt analyze --explain --backend localai --namespace default --filter Pod
k8sgpt analyze --explain --backend localai --namespace default --filter Service

Deploy K8sGPT Operator

Install the kube-prometheus-stack Helm Chart to use Grafana and Prometheus.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --version 79.8.2 

kubectl get pods -n monitoring -w

Install K8sGPT Operator via Helm

# Add Helm repository
helm repo add k8sgpt https://charts.k8sgpt.ai/
helm repo update
helm install release k8sgpt/k8sgpt-operator -n k8sgpt-operator-system --create-namespace -f k8sgpt-values.yaml

# Verify installation
kubectl get pods -n k8sgpt-operator-system

Configure K8sGPT to Use Local Ollama

# k8sgpt-config.yaml
cat <<EOF | kubectl apply -f -
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    enabled: true
    backend: localai
    model: mistral
    baseUrl: http://ollama.ollama.svc.cluster.local:11434/v1
  noCache: false
  repository: ghcr.io/k8sgpt-ai/k8sgpt
  version: v0.4.1
EOF

Apply configuration:

kubectl apply -f k8sgpt-config.yaml

Check K8sGPT status

kubectl get k8sgpt -n k8sgpt-operator-system
kubectl describe k8sgpt k8sgpt-local -n k8sgpt-operator-system

View K8sGPT Results

# Get results as custom resources
kubectl get results -n k8sgpt-operator-system

# View detailed analysis
kubectl get results -n k8sgpt-operator-system -o yaml

# Watch for new results
kubectl get results -n k8sgpt-operator-system -w

# Access prometheus
kubectl port-forward service/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090

# Access Grafana
kubectl port-forward service/prometheus-grafana -n monitoring 3000:80

🔗 Reference Links:

• K8sgpt Docs

• GitHub-Repo

• Ollama