Stage 5 · Platform
Production Troubleshooting & Operations
DNS & Service Debugging
CoreDNS logs, EndpointSlices, kube-proxy rules, conntrack, and tcpdump packet captures.
DNS Debugging
DNS issues are common in Kubernetes. Symptoms: service name not resolving, intermittent DNS failures, slow DNS lookups. Debug with dig/nslookup from inside pods, check CoreDNS logs, and verify DNS configuration.
# Test DNS resolution
kubectl exec -it test-pod -- nslookup api-service.production.svc.cluster.local
# Check pod DNS configuration
kubectl exec my-pod -- cat /etc/resolv.conf
# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Check CoreDNS config
kubectl get configmap coredns -n kube-system -o yamlDNS debugging: 1) Check /etc/resolv.conf (nameserver should be cluster DNS IP), 2) Test resolution with dig/nslookup, 3) Check CoreDNS pods are running, 4) Check CoreDNS logs for errors.
EndpointSlice Debugging
Endpointslices track which pods are ready to receive traffic. If a service has no ready endpoints, traffic cannot reach pods. Common issues: selector mismatch, pods not ready, and namespace mismatch.
# Check endpoints for a service
kubectl get endpointslice -l kubernetes.io/service-name=api-service -o wide
# Check service selector
kubectl get svc api-service -o yaml | grep -A 5 selector
# Check pod labels
kubectl get pods --show-labels -l app=api
# Check pod readiness
kubectl get pods -o wide | grep api
# Check readiness probe status
kubectl describe pod api-pod | grep -A 5 "Readiness"EndpointSlice issues: 1) No endpoints means no pods match the selector, 2) Endpoints exist but not ready means readiness probe is failing, 3) Wrong namespace means the service and pods are in different namespaces.
kube-proxy Rules
kube-proxy maintains iptables or IPVS rules for Services. If rules are missing or incorrect, Service traffic fails. Check iptables rules, IPVS configuration, and kube-proxy logs.
# Check kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
# Check kube-proxy pods
kubectl get pods -n kube-system -l k8s-app=kube-proxy
# Check kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50
# Check iptables rules (on node)
ssh worker-1 "iptables -t nat -L KUBE-SERVICES | head -20"
# Check IPVS rules (if using IPVS mode)
ssh worker-1 "ipvsadm -Ln | head -20"kube-proxy rules translate Service ClusterIPs to pod IPs. Check: kube-proxy is running, rules are present, and rules are correct. If using IPVS, check IPVS tables with ipvsadm.
Conntrack Issues
Conntrack tracks network connections. Issues: conntrack table full (dropped connections), stale entries (routing to terminated pods), and hash collisions (slow lookups). These cause intermittent connectivity failures.
# Check conntrack table size
ssh worker-1 "cat /proc/sys/net/netfilter/nf_conntrack_count"
ssh worker-1 "cat /proc/sys/net/netfilter/nf_conntrack_max"
# Check conntrack entries for a specific service
ssh worker-1 "conntrack -L -p tcp --dport 8080"
# Check for stale entries (terminated pod IPs)
ssh worker-1 "conntrack -L | grep 10.244.1.5"
# Flush conntrack cache (causes brief disruption)
ssh worker-1 "conntrack -F"conntrack_count approaching conntrack_max indicates table pressure. Stale entries route to terminated pods. Flush the table to clear stale entries. Increase conntrack_max for high-connection workloads.
Packet Captures
tcpdump captures network packets for deep debugging. Use it to see actual traffic flow, identify dropped packets, and debug TLS issues. Deploy tcpdump in a pod or use netshoot.
# Deploy netshoot for packet capture
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- /bin/bash
# Capture traffic to a service
tcpdump -i eth0 -nn host api-service and port 8080
# Capture DNS traffic
tcpdump -i eth0 -nn port 53
# Capture TLS traffic (won't show decrypted content)
tcpdump -i eth0 -nn port 443
# Write capture to file for Wireshark
tcpdump -i eth0 -nn -w /tmp/capture.pcap port 8080
# Check for retransmissions (sign of packet loss)
tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0' | wc -ltcpdump shows raw packet flow. Filter by host, port, or protocol. Write to .pcap for Wireshark analysis. Retransmissions indicate packet loss or network issues.
Service Connectivity
# Test service connectivity
kubectl exec -it test-pod -- curl -v http://api-service:80
# Test with specific headers
kubectl exec -it test-pod -- curl -H "Host: api.example.com" http://api-service:80
# Test TCP connectivity
kubectl exec -it test-pod -- nc -zv api-service 8080
# Check service ports
kubectl get svc api-service -o yaml | grep -A 5 ports
# Check if service has endpoints
kubectl get endpoints api-serviceService connectivity debugging: 1) Test with curl/nc, 2) Check service ports match container ports, 3) Verify endpoints exist and are ready, 4) Check NetworkPolicy allows traffic, 5) Check kube-proxy rules.
Set ndots:5 (default) for general use. For pods that mostly access external services, set ndots:2 to reduce search path iterations. Use dnsConfig to override per-pod. Monitor CoreDNS query latency.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.