Wireshark for Security Professionals: Protocol Analysis and Intrusion Detection

Wireshark is the world's most widely used network protocol analyzer, capable of capturing and dissecting traffic at an extraordinarily granular level. For security professionals, it serves dual roles: as an offensive tool for intercepting and analyzing network communications during assessments, and as a defensive tool for incident response, intrusion detection, and forensic investigation of malicious traffic. This guide covers Wireshark from a security operations perspective, focusing on the techniques analysts actually use when investigating real incidents.

Capture Filters vs Display Filters

Understanding the fundamental difference between these two filter types is critical and a source of constant confusion.

Capture Filters (BPF Syntax)

Capture filters use Berkeley Packet Filter (BPF) syntax and are applied before packets enter Wireshark. They reduce storage and processing overhead by discarding unwanted packets at the capture driver level. Once a packet is discarded by a capture filter, it's gone — you cannot retroactively see it.

# Capture only HTTP and HTTPS traffic
tcp port 80 or tcp port 443

# Capture traffic to/from a specific host
host 192.168.1.100

# Capture all traffic except your own management traffic
not host 10.0.0.5

# Capture DNS traffic
udp port 53

# Capture ICMP only (for ping analysis)
icmp

# Capture traffic from a specific subnet
net 192.168.1.0/24

# Capture traffic on a specific port range
tcp portrange 1-1024

# Capture ARP traffic
arp

# NOT destination broadcast
not broadcast and not multicast

# Capture traffic between two specific hosts
host 192.168.1.10 and host 192.168.1.20

# Capture only TCP SYN packets (for port scan detection)
tcp[13] == 0x02

# Capture SYN-ACK packets
tcp[13] == 0x12

# Capture TCP RST packets
tcp[13] & 0x04 != 0

Display Filters (Wireshark Filter Language)

Display filters are applied post-capture to the existing packet set. They use Wireshark's own protocol-aware language and are far more powerful and flexible than BPF.

# Filter by IP address
ip.addr == 192.168.1.100
ip.src == 192.168.1.100
ip.dst == 192.168.1.100

# Filter by subnet
ip.addr == 192.168.1.0/24

# Filter by protocol
http
dns
smb
ftp
ssh
telnet
icmp

# Filter by port
tcp.port == 443
tcp.dstport == 80
udp.port == 53

# HTTP specific
http.request.method == "POST"
http.response.code == 200
http.host contains "malicious"
http.request.uri contains "passwd"

# DNS
dns.qry.name contains "suspicious"
dns.flags.response == 0   # DNS queries only
dns.flags.response == 1   # DNS responses only

# TLS/SSL
tls.handshake.type == 1   # ClientHello
ssl.handshake.type == 2   # ServerHello

# SMB
smb.cmd == 0x72    # SMB Negotiate
smb2.cmd == 0      # SMB2 Negotiate

# Filter by TCP flags
tcp.flags.syn == 1 and tcp.flags.ack == 0   # SYN only
tcp.flags.syn == 1 and tcp.flags.ack == 1   # SYN-ACK
tcp.flags.rst == 1                           # RST
tcp.flags.fin == 1                           # FIN

# Boolean operators
ip.src == 192.168.1.1 and tcp.port == 80
http or dns
not arp

# Contains / matches (regex)
http.request.uri contains "admin"
http.request.uri matches "id=[0-9]+"

# Frame number and time
frame.number == 1234
frame.time_relative > 10.0

# Packet length
frame.len > 1400
frame.len < 60

Protocol Dissectors and Following Streams

Following TCP Streams

One of Wireshark's most powerful features for analysis. Right-click any TCP packet → Follow → TCP Stream to reconstruct the complete conversation in human-readable form.

# Display filter to see all packets in a specific TCP stream
tcp.stream eq 0    # first stream
tcp.stream eq 5    # sixth stream

# Follow UDP stream (DNS, QUIC, etc.)
udp.stream eq 0

# Follow HTTP stream
http.request or http.response

HTTP Analysis

# Find all HTTP POST requests
http.request.method == "POST"

# Find potential credential submissions
http.request.method == "POST" and (http.request.uri contains "login" or http.request.uri contains "auth")

# Find large responses (possible data exfiltration)
http.response and http.content_length > 100000

# Show only HTTP 401/403 (access control issues)
http.response.code == 401 or http.response.code == 403

# View all HTTP objects: File > Export Objects > HTTP
# This extracts all files transferred over HTTP

Detecting Network Attacks

Detecting Port Scans

Port scans generate distinctive traffic patterns — rapid connection attempts across many ports from a single source, with RST responses from closed ports.

# Large number of SYN packets to different ports from one source
ip.src == SCANNER_IP and tcp.flags.syn == 1 and tcp.flags.ack == 0

# RST flood from target (closed ports)
ip.dst == SCANNER_IP and tcp.flags.rst == 1

# UDP scan — ICMP port unreachable responses
icmp.type == 3 and icmp.code == 3

# Statistics > Endpoints → sort by packets to identify active scanners
# Statistics > Conversations → identify unusual connection patterns

# Stealth scan detection (many SYNs, no established connections)
tcp.flags == 0x002 and not tcp.flags == 0x012

Detecting ARP Poisoning / MITM

ARP poisoning floods the network with gratuitous ARP replies to associate the attacker's MAC address with victim IP addresses.

# Filter ARP traffic
arp

# Look for ARP replies that weren't preceded by requests (gratuitous ARP)
arp.opcode == 2   # ARP Reply

# Detect duplicate IP → MAC mappings
# Check if multiple MACs claim the same IP
# Statistics > ARP Statistics

# Red flag: One MAC address claiming to be both the router and multiple hosts
# Display filter: arp.src.hw_mac == MAC_OF_SUSPECT

# ARP storm detection
arp.opcode == 2 and eth.src == ATTACKER_MAC

Detecting Credential Sniffing Opportunities

Identify cleartext protocols transmitting credentials:

# FTP authentication
ftp.request.command == "USER" or ftp.request.command == "PASS"

# Telnet sessions (entire session is cleartext)
telnet

# HTTP Basic Authentication (base64-encoded in Authorization header)
http.authorization

# SMTP AUTH (email credentials)
smtp.req.command == "AUTH"

# POP3 authentication
pop.request.command == "USER" or pop.request.command == "PASS"

# LDAP simple bind (cleartext credentials)
ldap.simple

# HTTP form POST with password field
http.request.method == "POST" and http.file_data contains "password="

# SMBv1 credentials (NTLM challenge-response visible)
ntlmssp

Detecting C2 Beaconing

# Regular, periodic connections to external IPs
# Filter for consistent intervals:
# Statistics > IO Graph — look for regular spikes at fixed intervals

# DNS C2 — unusually long DNS query names
dns.qry.name.len > 50

# DNS tunneling — many DNS TXT queries
dns.qry.type == 16   # TXT record

# Large DNS responses (DNS exfiltration)
dns and frame.len > 512

# High-frequency DNS queries to same domain
dns.qry.name contains "suspicious-c2.com"

# ICMP tunneling — ICMP packets with data payloads
icmp and data.len > 0

TLS/SSL Traffic Decryption

Wireshark can decrypt TLS traffic if you have the pre-master secret or private key.

Method 1: Pre-Master Secret Log (SSLKEYLOGFILE)

This is the most practical method for modern TLS 1.3 where static key decryption doesn't work:

# Set environment variable before launching browser
# Linux/macOS
SSLKEYLOGFILE=/tmp/ssl_keys.log firefox &
SSLKEYLOGFILE=/tmp/ssl_keys.log chromium &

# Windows (cmd)
set SSLKEYLOGFILE=C:\ssl_keys.log
start chrome

# In Wireshark:
# Edit > Preferences > Protocols > TLS
# Set "(Pre)-Master-Secret log filename" to your keylog file
# Encrypted sessions are now decrypted in real-time

Method 2: RSA Private Key (TLS 1.2 and earlier)

# Edit > Preferences > Protocols > TLS > RSA Keys List > Add
# Enter:
# IP address: server IP
# Port: 443
# Protocol: http
# Key file: /path/to/private_key.pem

# Note: Does NOT work with Forward Secrecy (ECDHE/DHE key exchange)
# Only works with RSA key exchange (increasingly rare)

Decrypting with Zeek/tshark

# tshark with keylog file
tshark -r capture.pcap -o "tls.keylog_file:/tmp/ssl_keys.log" -Y http -T fields \
  -e http.host -e http.request.uri -e http.request.method

# Export decrypted HTTP objects
tshark -r capture.pcap -o "tls.keylog_file:/tmp/ssl_keys.log" --export-objects http,/tmp/extracted/

Extracting Files from PCAP

# Wireshark GUI:
# File > Export Objects > HTTP   (HTTP downloads)
# File > Export Objects > SMB    (file transfers over SMB)
# File > Export Objects > FTP    (FTP uploads/downloads)
# File > Export Objects > DICOM  (medical imaging protocol)

# tshark CLI file extraction
tshark -r capture.pcap --export-objects http,/tmp/http_files/
tshark -r capture.pcap --export-objects smb,/tmp/smb_files/

# NetworkMiner (Windows) — automatically extracts and reassembles files
# Zeek (formerly Bro) — automated file extraction with metadata
zeek -r capture.pcap /opt/zeek/share/zeek/policy/frameworks/files/extract-all-files.zeek

tshark CLI Automation

tshark is the command-line version of Wireshark, ideal for scripting, remote capture, and automated analysis pipelines.

Basic tshark Usage

# Live capture
tshark -i eth0

# Capture to file
tshark -i eth0 -w /tmp/capture.pcap

# Capture with filter
tshark -i eth0 -f "tcp port 80" -w /tmp/http.pcap

# Rotate files every 100MB
tshark -i eth0 -b filesize:102400 -w /tmp/capture.pcap

# Read a PCAP file
tshark -r capture.pcap

# Apply display filter
tshark -r capture.pcap -Y "http.request.method == POST"

# Extract specific fields
tshark -r capture.pcap -Y "http" -T fields -e frame.time -e ip.src -e ip.dst -e http.request.uri

# Count conversations
tshark -r capture.pcap -q -z conv,tcp

# Protocol statistics
tshark -r capture.pcap -q -z io,phs

# HTTP request statistics
tshark -r capture.pcap -q -z http,tree

# Find all unique destination IPs
tshark -r capture.pcap -T fields -e ip.dst | sort -u

# Extract all DNS queries
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e dns.qry.name | sort | uniq -c | sort -rn

Security Automation Scripts

# Script: Find potential password leaks in HTTP POST data
tshark -r capture.pcap -Y "http.request.method == POST" \
  -T fields -e ip.src -e http.host -e http.request.uri -e http.file_data \
  2>/dev/null | grep -iE "password|passwd|pwd|secret|token"

# Script: Extract all unique hostnames from HTTPS SNI
tshark -r capture.pcap -Y "tls.handshake.extensions_server_name" \
  -T fields -e ip.dst -e tls.handshake.extensions_server_name \
  | sort -u

# Script: Detect port scan activity
tshark -r capture.pcap -Y "tcp.flags.syn==1 and tcp.flags.ack==0" \
  -T fields -e ip.src -e ip.dst -e tcp.dstport \
  | awk '{src[$1]++} END {for (ip in src) if (src[ip] > 50) print ip, src[ip], "ports scanned"}'

# Script: Count DNS queries per domain (C2 detection)
tshark -r capture.pcap -Y "dns.flags.response==0" \
  -T fields -e dns.qry.name \
  | sed 's/.*\.\([^.]*\.[^.]*\)$/\1/' \
  | sort | uniq -c | sort -rn | head -20

Wireshark Expert Analysis Features

Statistics and IO Graphs

# Statistics > IO Graph
# Plot packet rate over time — spikes indicate scan/attack activity
# Compare multiple filters on the same graph

# Statistics > Conversations
# Identify top talkers and most active connections

# Statistics > Protocol Hierarchy
# See protocol breakdown — unusual protocols indicate tunneling

# Statistics > Follow Up (TCP) Stream Statistics
# View stream reconstruction

# Analyze > Expert Information
# Wireshark flags TCP retransmissions, resets, malformed packets
# Errors = red, Warnings = yellow, Notes = cyan, Chats = blue