Stage 5 · Platform
Production Troubleshooting & Operations
kubectl Debug Toolkit
kubectl describe, events, logs, exec, debug containers, and netshoot troubleshooting pods.
kubectl describe
kubectl describe is the first tool for debugging. It shows: resource details, conditions, events, and annotations. Events reveal scheduling failures, image pull errors, probe failures, and OOM kills.
# Describe a pod (shows events at the bottom)
kubectl describe pod my-pod -n production
# Describe a node
kubectl describe node worker-1
# Describe a service (shows endpoints)
kubectl describe svc my-service -n production
# Describe a deployment
kubectl describe deployment my-app -n production
# Filter events
kubectl describe pod my-pod -n production | grep -A 5 EventsThe Events section at the bottom of describe output is the most valuable for debugging. It shows the last several events in chronological order. Look for Warning events — they indicate problems.
kubectl logs
kubectl logs shows container output. Use --previous for crashed containers, --follow for real-time streaming, and --timestamps for time correlation. For multi-container pods, specify the container name.
# Basic logs
kubectl logs my-pod -n production
# Logs from previous container (crash)
kubectl logs my-pod -n production --previous
# Follow logs in real-time
kubectl logs my-pod -n production -f
# Logs with timestamps
kubectl logs my-pod -n production --timestamps
# Specific container in multi-container pod
kubectl logs my-pod -n production -c sidecar
# Logs from all pods matching a label
kubectl logs -l app=my-app -n production --all-containers
# Last 100 lines
kubectl logs my-pod -n production --tail=100
# Logs since a specific time
kubectl logs my-pod -n production --since=2h--previous shows logs from the last container instance (before crash). -f follows new log lines. --tail limits output. --since filters by time. These flags combine for precise log filtering.
kubectl exec
kubectl exec runs commands inside a container. Use it for live debugging: checking configuration, testing connectivity, and inspecting processes. The container must have a shell (or use debug containers).
# Interactive shell
kubectl exec -it my-pod -n production -- /bin/sh
# Run a specific command
kubectl exec my-pod -n production -- cat /etc/config/config.yaml
# Specific container
kubectl exec -it my-pod -n production -c app -- /bin/bash
# Check environment variables
kubectl exec my-pod -n production -- env
# Test network connectivity
kubectl exec my-pod -n production -- curl -s http://api-service:80
# Check DNS resolution
kubectl exec my-pod -n production -- nslookup api-serviceexec runs a single command. -it provides interactive TTY. If the container doesn't have a shell, use kubectl debug to add an ephemeral container with debugging tools.
kubectl debug
kubectl debug adds ephemeral debugging containers to running pods. These containers share the process namespace with the target container, providing debugging tools without restarting the pod.
# Add debug container to running pod
kubectl debug -it my-pod -n production --image=busybox:1.36 --target=app
# Debug using the node's process namespace
kubectl debug node/worker-1 -it --image=busybox:1.36
# Debug with specific profile
kubectl debug -it my-pod -n production --image=busybox:1.36 \
--profile=netadmin
# List existing ephemeral containers
kubectl get pod my-pod -n production -o jsonpath='{.spec.ephemeralContainers}'--target shares the process namespace with the target container. You can see its processes, network connections, and files. --profile adds Linux capabilities for advanced debugging.
Netshoot Troubleshooting
Netshoot is a debugging pod with networking tools: curl, dig, tcpdump, nmap, iperf, and more. Deploy it for network debugging in any namespace.
# Deploy netshoot
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- /bin/bash
# Inside netshoot:
# Test DNS
dig api-service.production.svc.cluster.local
# Test HTTP connectivity
curl -v http://api-service:8080/health
# Capture packets
tcpdump -i eth0 -nn port 8080
# Check network connectivity
ping api-service
# Traceroute
traceroute api-service
# Check listening ports
ss -tlnpnetshoot provides all networking tools in one pod. It shares the host network by default, so it can reach all pods. Deploy it temporarily for debugging and delete when done.
JSONPath Queries
# Get all pod names
kubectl get pods -o jsonpath='{.items[*].metadata.name}'
# Get pods with their IPs
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'
# Get containers with image tags
kubectl get pods -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}{end}'
# Get events filtered by type
kubectl get events -o jsonpath='{.items[?(@.type=="Warning")].message}'JSONPath extracts specific fields from JSON output. range iterates over arrays. ?(@.type==\"Warning\") filters by condition. This is useful for scripting and automation.
When debugging a pod: 1) kubectl describe (events), 2) kubectl logs (output), 3) kubectl logs --previous (crash logs), 4) kubectl exec (live debugging), 5) kubectl debug (ephemeral container). Follow this order for efficient debugging.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.