Systematic Network Troubleshooting Capstone

Learning objective

Use a repeatable, evidence-first workflow to scope an outage, isolate the failing layer, restore the whole service, and communicate what happened.

Scope: define the incident before touching the network

A useful incident statement says who, what, where, when, and how consistently. “The network is down” is not a scope. “VLAN 20 users at the branch cannot open the portal, while other sites work, beginning after the morning change” gives you boundaries that tests can challenge.

Record the last known good time, recent changes, affected users, unaffected controls, and business impact. Separate a symptom from a cause: a browser timeout proves only that a transaction did not finish.

Hypothesis: make a falsifiable explanation

A hypothesis links the observed symptom to one possible failure. Before running a command, write a predicted result: “If the access VLAN is wrong, the configured VLAN will disagree with the client subnet.” A prediction protects you from interpreting every output as confirmation.

Keep several plausible explanations alive. Rank them by symptom fit, probability, test cost, and risk—not by which command you remember first.

Evidence: test from least destructive to most invasive

Start with observation and read-only commands. Confirm addressing, neighbor state, switchport membership, route selection, name resolution, TCP establishment, TLS negotiation, and application response as separate layers. Prefer the least-destructive test that can distinguish two hypotheses.

For every test, record the command, device or interface, timestamp, expected result, actual evidence, and interpretation. A packet capture is not a conclusion: flags, tuples, timing, retransmissions, and capture position must support the claim.

Isolation: change one variable at a time

Compare failing and working paths. Move from broad boundaries toward the smallest component that explains all observations. When evidence contradicts your hypothesis, revise it openly. Do not stack speculative changes; each uncontrolled change destroys information and creates rollback risk.

Before remediation, capture the faulty state and define the expected post-change result. Use a reversible change, a rollback condition, and an owner whenever the environment permits.

Restore: verify the complete transaction

Restoration is broader than ping. Recheck addressing, ARP or neighbor discovery, VLAN membership, forward and return routing, DNS, TCP, TLS, and HTTP or the actual application protocol. Confirm both an affected user and an unaffected control. Watch long enough to detect cache, timeout, or failover effects.

If safe authority, evidence, or specialist ownership is missing, escalation is a valid troubleshooting action. Send a compact handoff containing scope, timeline, hypotheses tested, evidence, changes, rollback state, and the precise help required.

Report: preserve operational learning

A strong incident report distinguishes impact, evidence, root cause, contributing conditions, correction, restoration proof, and prevention. It does not rewrite the timeline to make the final answer seem obvious.

Reflection: What evidence would make you abandon your leading hypothesis? Which end-to-end checks would prove the user experience is restored? What threshold would make you escalate rather than continue alone?

Guided branch incident

The branch portal has several sequential faults. State a hypothesis, predict what a safe test should show, collect evidence, and remediate only when the evidence supports the change. The next fault becomes visible only after the current layer is corrected.

Systematic troubleshooting capstone

Restore the branch portal

0 simulated minutes · 0/3 faults corrected

Current network path

CBranch clientclient

SAccess switchswitch

RBranch routerrouter

FEdge firewallfirewall

RDNS resolverresolver

SPortal serverserver

Branch clienteth0Gi1/0/18Access switch

Access switchGi1/0/48Gi0/0Branch router

Branch routerGi0/1insideEdge firewall

Edge firewalldmzeth0DNS resolver

Edge firewalloutsideeth0Portal server

• Branch client eth0 to Access switch Gi1/0/18
• Access switch Gi1/0/48 to Branch router Gi0/0
• Branch router Gi0/1 to Edge firewall inside
• Edge firewall dmz to DNS resolver eth0
• Edge firewall outside to Portal server eth0

Confirm incident scope

Hypothesis journal

What could explain the evidence?

The access port is in the wrong VLANA more-specific route overrides the correct pathThe resolver has stale portal data

What should the next test show?

The switchport VLAN differs from the client subnet.

Confidence

underconfidentcalibratedoverconfident

Evidence board

Select a safe test to collect evidence.

Form a hypothesis, predict the result, then collect evidence.

Remediation controls

End-to-end restoration

Complete evidence-backed remediation before restoration testing.

Run verification: Addressing is correctRun verification: Gateway adjacency resolvesRun verification: Access VLAN is correctRun verification: Portal route is correctRun verification: DNS returns the active portalRun verification: TLS handshake succeedsRun verification: Portal returns HTTP 200

Close incidentRestart incident

Incident timeline

1. No evidence collected yet.

Guided incident debrief

Review your timeline. Identify where an incorrect attempt still improved the investigation, whether your confidence matched the evidence, and why verifying DNS, TLS, and HTTP added information beyond a successful ping.

Final knowledge check

Answer all three questions correctly to complete this capstone lesson.

Knowledge check: Evidence contradicts your leading troubleshooting hypothesis. What should you do next?

Revise the hypothesis and test the next best explanationKeep changing settings until the symptom disappearsIgnore the evidence because the original hypothesis came first

Check answer

Knowledge check: Which test should normally come first during an incident?

The least-destructive test that distinguishes plausible causesA broad configuration replacementA reboot of every device in the path

Check answer

Knowledge check: A successful ping returns after remediation. What best proves the service is restored?

Verify the complete user transaction, including DNS, transport, security, and application responseTreat ping alone as proof that every layer worksWait for users to report whether the problem returns

Check answer

The Pro capstone combines sparse asymmetric-routing evidence, Wireshark-style packet analysis, and RFC 9293 validation in one coordinated incident.

Pro incident: sparse evidence and asymmetric routing

Correlate packet, route, firewall, DNS, and interface evidence without guided sequencing.

Systematic troubleshooting capstone

Pro incident: intermittent branch portal

0 simulated minutes · 0/2 faults corrected

Current network path

CBranch clientclient

SAccess switchswitch

RBranch routerrouter

FEdge firewallfirewall

RDNS resolverresolver

SPortal serverserver

Branch clienteth0Gi1/0/18Access switch

Access switchGi1/0/48Gi0/0Branch router

Branch routerGi0/1insideEdge firewall

Edge firewalldmzeth0DNS resolver

Edge firewalloutsideeth0Portal server

• Branch client eth0 to Access switch Gi1/0/18
• Access switch Gi1/0/48 to Branch router Gi0/0
• Branch router Gi0/1 to Edge firewall inside
• Edge firewall dmz to DNS resolver eth0
• Edge firewall outside to Portal server eth0

Confirm incident scopeOpen hypothesis worksheet

Evidence board

Select a safe test to collect evidence.

Form a hypothesis, predict the result, then collect evidence.

Remediation controls

End-to-end restoration

Complete evidence-backed remediation before restoration testing.

Run verification: Forward and return paths are symmetricRun verification: Client and resolver answers agreeRun verification: Portal service is restored

Close incidentRestart incident

Incident timeline

1. No evidence collected yet.

Pro incident report

Write the incident report

Turn observations into an evidence-backed operational record.

Root-cause score

0%

Restoration score

0%

Report score

0%

ImpactEvidenceRoot causesCorrectionRestorationPreventionSubmit incident report

Pro packet and RFC validation

Validate packet evidence and standards

Check Wireshark: repeated SYN without a returnDNS cache timingForward and return route validation

Evidence: The client capture shows SYN retransmission at 1, 3, and 7 seconds; the stateful firewall records zero return bytes.

Which finding best explains this evidence?

The return path bypasses the stateful firewallThe access switch has a duplex mismatchThe server certificate has expired

Check advanced decisionRFC 9293