Stage 3 · Build
Network Debugging in Production
Network Debug Methodology
Divide-and-conquer — isolate layer by layer from application to wire.
Systematic Debugging
Network debugging is about ruling out possibilities systematically. Start with what you know, form a hypothesis, test it, and narrow down the problem. Random commands without a plan waste time and miss the issue.
1. OBSERVE: What is the exact symptom?
- Error message? Timeout? Connection refused?
- When did it start? All clients or some?
2. ISOLATE: Where is the problem?
- Which layer? L1/L2/L3/L4/L7?
- Which component? Client, network, server?
3. HYPOTHESIZE: What could cause this?
- DNS? Firewall? Certificate? Service down?
4. TEST: Verify the hypothesis
- Run a specific command that confirms/denies
5. FIX: Apply the solution
6. VERIFY: Confirm the fix worksThe key is step 2 — isolation. Before fixing, you must know which layer and which component is failing. A systematic approach prevents you from fixing the wrong thing.
Layer-by-Layer Approach
# Layer 1/2 - Link status
ip link show # Is the interface up?
ethtool eth0 # Link speed, duplex, errors
# Layer 3 - Network reachability
ip route show # Routing table
ping <gateway> # Can we reach the gateway?
ping <destination> # End-to-end reachability
traceroute <destination> # Path to destination
# Layer 4 - Transport
ss -tan # TCP connection states
nc -zv <host> <port> # Port open/closed?
telnet <host> <port> # Connection test
# Layer 7 - Application
curl -v <url> # Full HTTP request/response
dig <domain> # DNS resolution
openssl s_client -connect <host>:443 # TLS verificationAlways start from the bottom. If the link is down (L1), nothing else matters. If you cannot reach the gateway (L3), DNS and HTTP are irrelevant. Fix layer by layer, bottom up.
Tools for Each Layer
| Layer | Tool | What It Shows |
|---|---|---|
| L1/L2 | ip link, ethtool | Interface status, speed, errors |
| L3 | ping, traceroute, ip route | Reachability, path, routing |
| L4 | ss, nc, telnet, tcpdump | Connections, ports, packets |
| L7 | curl, dig, openssl | HTTP, DNS, TLS |
| All | Wireshark | Full packet capture and analysis |
Debugging Checklists
1. DNS resolution working?
$ dig example.com +short
2. IP reachability?
$ ping <ip>
3. Port open?
$ nc -zv <ip> <port>
4. TLS working?
$ openssl s_client -connect <host>:443
5. HTTP response?
$ curl -v https://example.com
6. Server logs?
$ kubectl logs <pod> --tail=50
7. Firewall rules?
$ sudo iptables -L -n
8. Service discovery?
$ kubectl get svc, endpointsPrint this checklist and keep it at your desk. When a connection fails, go through each step. Most issues are caught at steps 1-4. If all those pass, the issue is in the application.
Common Debugging Anti-Patterns
Restarting services without understanding the root cause is not debugging. The problem will come back. Take the time to find the actual cause. Restarting masks the symptom and wastes time.
Other anti-patterns: assuming the problem is where the symptom appears (the root cause is often elsewhere), changing multiple things at once (you will not know which change fixed it), and skipping the basics (did you check if the service is running?).
When debugging a chain of services, binary search for the failing component. If A -> B -> C -> D and D is failing, test B -> C directly. If that works, the problem is between C and D. Halve the search space each time.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.