A Practitioner's Checklist for Troubleshooting Common Network Protocol Issues

When a critical service stops responding, the first instinct is often to restart something—a router, a server, a firewall. That might buy time, but it rarely teaches you what broke. Network protocol issues demand a repeatable, evidence-based process. This checklist is built for practitioners who need to isolate faults quickly, whether the symptom is a dropped connection, a slow page load, or a routing loop. We focus on the protocols you encounter daily: TCP, DNS, DHCP, HTTP/HTTPS, and BGP. Each section gives you a concrete set of checks, not abstract advice.

1. Building Your Troubleshooting Foundation: Tools and Mindset

Before you touch any configuration, you need a clear picture of what is actually happening on the wire. The most common mistake is jumping to a hypothesis—'it must be DNS'—and spending an hour chasing a red herring. Start by collecting data without prejudice.

Essential Tools for Every Kit

Wireshark or tcpdump should be your first stop. A packet capture at the affected endpoint (or a mirrored port) shows you exactly which packets arrive, which are retransmitted, and where delays occur. Complement this with ping and traceroute for basic connectivity checks, but remember that ICMP can be blocked or rate-limited. For DNS, dig or nslookup give you query timing and response details. For HTTP, curl with verbose output reveals headers, redirects, and TLS handshake steps. For BGP, tools like bird or bgpdump help inspect routing tables.

Equally important is a baseline. If you don't know the normal latency, packet loss rate, or DNS resolution time for your environment, you cannot identify anomalies. Spend a few minutes recording these metrics during stable periods. That baseline becomes your reference when something goes wrong.

The Mindset: Isolate, Don't Guess

Work in layers. Start at the physical and data-link layers: are cables plugged in? Are link lights on? Is there duplex mismatch? Then move up to the network layer (IP connectivity, routing), transport layer (TCP handshake, retransmissions), and application layer (protocol-specific behavior). Each layer narrows the possibilities. Keep a running log of what you test and what you observe. This prevents repeating steps and helps you spot patterns.

One team I read about spent three hours replacing cables and switches for a 'link down' issue that turned out to be a misconfigured VLAN on a single port. A systematic layer-by-layer check would have found it in ten minutes. The checklist below follows this layered approach.

2. TCP Troubleshooting: Handshake Failures, Retransmissions, and Window Issues

TCP is the backbone of most application traffic, and its behavior gives rich clues about network health. When a connection fails or performs poorly, the TCP handshake and subsequent data flow often reveal the root cause.

Handshake Problems

If a TCP connection never establishes, capture the three-way handshake. Look for the SYN, SYN-ACK, ACK sequence. Missing SYN-ACK usually means the destination host did not respond—check if the service is listening, if a firewall dropped the SYN, or if the destination IP is unreachable. A SYN-ACK that comes from a different IP than the one you sent the SYN to indicates a spoofing or NAT issue. If the handshake completes but the application still fails, the problem is above TCP—often TLS or application logic.

Retransmissions and Duplicate ACKs

Retransmissions are the most visible sign of packet loss. In Wireshark, filter for 'tcp.analysis.retransmission'. A few retransmissions during a transfer may be normal on a lossy link, but sustained retransmissions point to congestion, faulty hardware, or a duplex mismatch. Duplicate ACKs (filter: 'tcp.analysis.duplicate_ack') indicate out-of-order delivery or loss. If you see many duplicate ACKs followed by a fast retransmission, the network is dropping packets. Check interface errors on switches and routers: CRC errors, collisions, or excessive drops.

TCP Window and Buffer Issues

A zero-window advertisement means the receiver cannot accept more data—often because the application is not reading fast enough. This is an application performance issue, not a network fault. On the sender side, a small send buffer can limit throughput even if the network has capacity. Use tools like ss or netstat on Linux to check buffer sizes and window scaling. If window scaling is not negotiated (look for the WS option in the SYN), the effective window is limited to 65,535 bytes, capping throughput on high-latency links.

One common scenario: a file transfer between data centers is slow. Wireshark shows the window is never fully used, and retransmissions are rare. The culprit is often a low TCP send buffer on the sender or a low receive buffer on the receiver. Tuning these buffers (e.g., net.core.rmem_max and wmem_max on Linux) can dramatically improve throughput without any network change.

3. DNS Resolution: When Queries Go Wrong

DNS is the most common scapegoat for network problems, and sometimes it really is the culprit. A failing DNS lookup can make everything feel broken, even when the network is healthy. The key is to isolate whether the problem is local (the resolver), the recursive server, or the authoritative server.

Step 1: Test Resolution Locally

Use dig +trace example.com to see the full resolution path. If the first response times out, your resolver may be unreachable. Check your /etc/resolv.conf (or DHCP-assigned DNS servers) and ping the resolver IP. If the resolver responds but the trace stops at a particular nameserver, that nameserver may be failing or misconfigured. Look for SERVFAIL responses, which often indicate a problem at the authoritative side—maybe the zone is misconfigured or the server is overloaded.

Step 2: Check Caching and TTLs

Stale DNS records cause intermittent failures. If a record's TTL is very short (e.g., 60 seconds), clients may query frequently, increasing load. If the TTL is very long (e.g., 86400 seconds), a change takes a day to propagate. Use dig to inspect the TTL of the response. If you recently changed an A record but clients still see the old IP, the resolver may be caching aggressively. Flush the cache on the resolver or wait for the TTL to expire.

Step 3: DNSSEC and EDNS0 Issues

Some networks block or mishandle large DNS responses (over 512 bytes) that include DNSSEC signatures or EDNS0 options. If a query works with +nodnssec but fails with DNSSEC enabled, the resolver or firewall may be dropping the larger packet. Check if your resolver supports EDNS0 (most do) and if the firewall allows UDP responses larger than 512 bytes. In some cases, switching to TCP for DNS (via configuration) bypasses the size limitation.

A typical pitfall: a DNS query returns NXDOMAIN for a valid hostname because the resolver is configured to append an incorrect search domain. Use dig with a trailing dot (dig example.com.) to force an absolute query and bypass search domains. If that works, the issue is client-side search domain configuration.

4. DHCP: Address Allocation and Renewal Problems

DHCP failures often manifest as 'no network access' on a client that previously worked. The client gets an APIPA address (169.254.x.x) or an old lease that no longer matches the subnet. Troubleshooting DHCP requires understanding the DORA process (Discover, Offer, Request, Acknowledge).

Capturing the DORA Exchange

Run a packet capture on the client during a DHCP renewal. Filter for 'bootp' or 'dhcp'. If you see only Discover messages and no Offer, the DHCP server is not responding. Common causes: the server is down, the relay agent (if the client is on a different subnet) is misconfigured, or a firewall is blocking UDP ports 67/68. If Offers arrive but no Acknowledge, the server may be out of addresses or the client's MAC is blacklisted. Check the DHCP server logs for 'no free leases' or 'unknown client' entries.

Lease Time and Renewal Timing

Short lease times (e.g., 5 minutes) increase network traffic and the risk of renewal failures. Long lease times (e.g., 7 days) mean a client can hold an address that conflicts with a new device. If a client fails to renew before the lease expires, it loses its IP. Use ipconfig /all (Windows) or dhclient -v (Linux) to see the lease expiry. If the client cannot reach the DHCP server at renewal (e.g., the network was down for a day), it will eventually get a new address—but that address may be different, breaking any static mappings or firewall rules that relied on the old IP.

Vendor-Specific Options and Relay Agents

Some DHCP servers use option 82 (relay agent information) to assign addresses based on the switch port. If the relay agent is not configured correctly, clients may get addresses from the wrong pool or no address at all. Check the relay agent configuration on the switch or router: it must have the correct DHCP server IP and the interface toward the clients must be in the correct VLAN. A misconfigured relay can cause the server to see the request as coming from the wrong subnet, leading to an incorrect address range.

One scenario: a new office subnet was added, but the DHCP relay on the router was not updated. Clients in that subnet sent Discover broadcasts, the router relayed them to the server with the wrong gateway interface, and the server offered addresses from a different pool. The clients accepted, but they could not reach the internet because their default gateway was wrong. The fix was to update the relay configuration to specify the correct subnet for that VLAN.

5. HTTP/HTTPS: Troubleshooting Application Layer Issues

HTTP and HTTPS failures are often blamed on the network, but many originate at the application or TLS layer. The key is to isolate whether the problem is connectivity, TLS handshake, or the application itself.

Step 1: Check HTTP Status Codes

Use curl -v http://example.com to see the full request and response. A 200 OK means the server received the request and returned content. A 3xx redirect may indicate a misconfigured virtual host or a missing trailing slash. 4xx errors (client errors) point to issues with the request: 401 Unauthorized (authentication), 403 Forbidden (permissions), 404 Not Found (URL path), 429 Too Many Requests (rate limiting). 5xx errors (server errors) indicate the server failed to fulfill a valid request: 500 Internal Server Error, 502 Bad Gateway (upstream server issue), 503 Service Unavailable (overload), 504 Gateway Timeout.

Step 2: TLS Handshake Analysis

If the connection fails with a TLS error, capture the handshake. Common issues: the server certificate is expired or self-signed (curl -k bypasses verification for testing), the client does not trust the CA, or the cipher suite is incompatible. Use openssl s_client -connect example.com:443 to see the certificate chain and handshake details. If the handshake fails after the ServerHello, the server may be requesting a client certificate that the client does not have. If it fails immediately, check if the port is actually TLS (some services use STARTTLS on port 25 or 587).

Step 3: Latency and Connection Pooling

Slow page loads are often due to many small HTTP requests, each requiring a new TCP connection. HTTP/2 and connection reuse reduce this overhead. Check if the server supports HTTP/2 (curl --http2). If not, consider enabling it. Also look at the Time to First Byte (TTFB): a high TTFB (>500 ms) suggests the server is slow to generate the response, not a network issue. Use curl -w '%{time_total}' to measure total time, and break it down with -w 'time_namelookup, time_connect, time_starttransfer, time_total'. If name lookup or connect time is high, the problem is DNS or TCP handshake latency. If starttransfer is high, the server is slow.

A common pitfall: a web application behind a reverse proxy returns 502 errors. The reverse proxy cannot reach the upstream server. Check if the upstream is listening on the correct port and if the proxy's network path to the upstream is functional. A packet capture between the proxy and upstream often reveals the issue quickly.

6. BGP: Routing Loops, Prefix Flapping, and Path Selection

BGP issues affect entire networks, causing traffic to take suboptimal paths or disappear entirely. Troubleshooting BGP requires understanding the routing table, the BGP session state, and the path attributes.

Session States and Keepalives

Check the BGP session state: Idle, Connect, Active, OpenSent, OpenConfirm, or Established. Any state other than Established indicates a problem. Common causes: the neighbor IP is unreachable (check ping), the TCP port 179 is blocked by a firewall, the BGP authentication (MD5 or TCP-AO) is mismatched, or the AS number or hold timer is misconfigured. Use show bgp summary (Cisco) or birdc show protocols (BIRD) to see the state and uptime. If the session flaps between Established and Idle, check the keepalive timer and the number of prefixes exchanged—a session with too many prefixes may time out.

Prefix Advertisement and Withdrawal

If a route is missing from the global table, check if the prefix is being advertised by the origin AS. Use looking glass servers (e.g., routeviews.org) to verify that the prefix is visible from multiple vantage points. If it is visible but your router does not receive it, the issue could be a filter on your upstream provider or a missing route-map. Use show ip bgp prefix (Cisco) to see if the route is in the table and which attributes it has. If the route is present but not selected, check the local preference, AS path length, and MED. A common mistake: a route with a longer AS path is preferred because the local preference is higher on one path—check the configuration.

Routing Loops and Blackholes

A routing loop occurs when two routers point to each other for the same prefix. Traceroute shows the packet bouncing between the same two hops. This often happens when a route is redistributed incorrectly between IGP and BGP, or when a static route overrides a BGP-learned route. Use show ip route for the prefix to see the next hop. If the next hop points to a router that points back, you have a loop. Fix by removing the conflicting route or adjusting the administrative distance.

Another scenario: a prefix is withdrawn by the origin but still present in the table because a stale route is not removed. This can cause a blackhole where traffic is sent to a router that no longer has a path. Use clear ip bgp * soft to force a refresh without tearing down sessions. If that does not help, check the BGP table for 'valid' vs 'invalid' routes—invalid routes are often due to missing next hops.

7. Mini-FAQ: Common Protocol Questions

Why does my TCP connection hang after the handshake?

This is often caused by a firewall or load balancer that intercepts the connection but does not properly forward data. Check for TCP sequence number issues or a device that sends a RST after a few packets. A packet capture will show if the server receives the data but does not respond—then it is an application issue.

How do I find the cause of packet loss?

Use tools like mtr (combined ping and traceroute) to see where loss occurs. If loss appears on multiple hops, it may be due to ICMP rate limiting on intermediate routers—not actual data loss. Compare with TCP retransmission rates. If retransmissions are low, the loss may be cosmetic. If they are high, focus on the first hop that shows loss in mtr.

What is asymmetric routing and why does it cause problems?

Asymmetric routing means packets take different paths in each direction. This is normal on the internet, but it can confuse stateful firewalls that expect to see both directions of a flow. If a firewall sees only one direction, it may drop packets. Solutions include using stateless ACLs or ensuring that the firewall sees both paths (e.g., by routing both directions through it).

How do I test if MTU is causing issues?

Send pings with the DF (Don't Fragment) flag set and gradually increase the packet size until you get a 'Fragmentation Needed' ICMP message. The largest size that works is the path MTU. If it is lower than the interface MTU, you need to adjust the MTU on the endpoints or enable PMTUD (Path MTU Discovery). Some firewalls block ICMP unreachable messages, breaking PMTUD—check that ICMP type 3, code 4 is allowed.

Why does DHCP work on one VLAN but not another?

Most likely the DHCP relay agent is not configured for the second VLAN. Ensure the router or switch interface in that VLAN has the 'ip helper-address' (Cisco) or 'dhcp-relay' (other vendors) pointing to the DHCP server. Also check that the VLAN interface has an IP address and is up.

After you resolve an issue, document what you found and how you fixed it. The next time a similar symptom appears, you will have a reference. That is the real value of a checklist: it turns troubleshooting from a reactive scramble into a repeatable process.