Capturing network traffic for troubleshooting
There are certain situations when a problem can only be resolved if the support team can take a closer look at the actual network traffic seen by the applications. This can help identify network issues and bottlenecks, and analyze network and application level protocols.
We recommend using the following network capture tools:
- Wireshark
- Tshark
- Verba Packet Capture
Installing Wireshark and Tshark on Verba servers
You can download and install Wireshark (and the Tshark command line tool with it) on the Verba servers. To learn more about Wireshark and download the installer, visit https://www.wireshark.org/.
Do not remove the Winpcap driver when installing Wireshark. Otherwise, the system components relying on the Wincap drive will not work anymore
Capturing on Recording Servers causes extra load on CPU and disk utilization and can interfere with the recording process which can lead to data loss under critical circumstances. The load of the packet capture on the Recording Servers should be always considered and if possible should only be used during non-busy hours.
Tshark
Tshark is a high performance packet capture application that is part of the Wireshark installation package. It is a command line tool for high performance continuous capturing. It is useful when network traffic is high and/or capturing with Wireshark becomes unstable, and when we need to leave tracing on for a longer period of time (many hours or days).Examples
Get help:
tshark -h
List interfaces:
tshark -D
Start capturing with capture file rotation:
tshark -i 3 -B 96 -b filesize:250000 -b files:100 -w c:\tmp\test.pcap -F pcap
Where:
- -i specifies the interface with the ID retrieved by tshark –D
- -B sets capture buffer size in Mbyte. Default is 2 Mbyte, if there is a large traffic, you should go up to 96 Mbyte
- -b specifies the capture rotation: filesize:xxx max size of a single capture file in Kbytes, files:xxx the number of files after which the oldest one is overwritten. Using file rotation set based on available disk space we can make sure there will be enough space left for the other applications when we leave tracing on for a longer time.
- -w output file
- -F output format (could be pcap or pcapng. We prefer pcap, that can be directly processed by the passive recorder)
- -f can specify capture filter in BPF syntax, see below. Using an appropriate capture filter highly reduces the load on CPU and Disk
BPF filter syntax
For the full syntax, see https://biot.com/capstats/bpf.html or https://www.ibm.com/support/knowledgecenter/SS42VS_7.3.3/com.ibm.qradar.doc/c_forensics_bpf.html.
Important filter examples:
Filtering all (src/dst) IP traffic of given host:
host 10.110.77.200
Filtering inbound/outbound IP traffic of given host:
src host 10.110.77.200 / dst host 10.110.77.200
Filtering for specific subnet:
(src/dst) net 10.110.77.0/24
Filtering for specific TCP or UDP port:
tcp/udp src/dst 443
Logical combination: or/and and grouping with () supported
Filtering for SIP (non secure, 5060 default port):
traffic of 10.110.77.200 CUCM: (udp port 5060 or tcp port 5060) and host 10.110.77.200
Filtering for proxy-filter communication:
tcp port 10201
Filtering for proxy-recorder communication:
tcp port 11112
Filtering for recording director – media recorder communication:
tcp port 10500
Wireshark
Wireshark is a GUI based packet trace analyzer. It can parse the real-time traffic capture or read a network packet capture file. The application can identify encapsulations and interpret and visualizes protocol data at all layers. It uses a different syntax than the BPF capture filter (used by Tshark). Filtering can only be applied to displaying the packets (and not capture filtering). For more information on display filters, see https://wiki.wireshark.org/DisplayFilters.
Display filter examples
Searching for string "xxxx" in the whole packet:
frame contains "xxxx"
Filtering for SIP or Skinny or H.323 call control packets:
sip or sccp or h323
Filtering for HTTP messages:
http
Filtering for IP traffic of 10.110.77.200:
ip.dst=="10.110.77.200" or ip.src=="10.110.77.200"
Filtering for TCP traffic on 5060 port (works with udp as well):
tcp.dstport==5060 or tcp.srcport==5060
Verba packet capture
The system comes with a built-in packet capture tool called Verba Packet Capture. This tool collects and stores network traffic without analyzing it or interfering with the recording progress, similar to Tshark. Verba Packet Capture creates standard PCAP files that can be opened by WireShark.
The tool is especially useful when troubleshooting proxy based recording issues because the tool can take into account the recorder settings and connect to the same proxies as a redundant/2N recorder pair of the recorder. That way it receives exactly the same traffic as the recorder service. The tool should be run on the servers where the Passive Recorder service runs.