Course12:Advanced - Troubleshooting with Wireshark: Difference between revisions
m (Protected "Course12:Advanced - Troubleshooting with Wireshark" [edit=sysop:move=sysop]) |
m (Protected "Course12:Advanced - Troubleshooting with Wireshark" ([Edit=Allow only administrators] (indefinite) [Move=Allow only administrators] (indefinite))) |
||
(One intermediate revision by one other user not shown) | |||
Line 1: | Line 1: | ||
{{#moodlebook: Master Templates / V12 Templates / Advanced | | {{#moodlebook: Master Templates / V12 Templates / Advanced | Troubleshooting with Wireshark | 121 }} |
Latest revision as of 11:52, 12 October 2023
The goal of this topic is to aquire a fundamental knowledge of troublehsooting Innovaphone Problems with Wireshark
Setting up Wireshark
Getting started
Before we can get started to learn about Wireshark, you should download the application from the
You may ask yourself why we are not using the new 2.0 release of Wireshark. After testing the new version we came to the conclusion that it isn't fitting our needs.
The next step to get going with Wireshark is downloading the latest Wireshark dll from our download server. You'll find a 32 bit and a 64 bit version (depending on your PC's architecture) in the tools item of the apps package on the Version 6 download page. For convenience, this is also available on the
After downloading the appropriate dll you need to paste it to the Wireshark plugins folder. Usually you should find the plugin folder here: C:\Program Files\Wireshark\plugins\1.12.9.
This dll decodes innovaphone specific debug messages as well as Q931 and LAPD frames. While Q931 and LAPD will help you in debugging ISDN related problems, innovaphone debug messages will give you an inside in all kind of problems. But since this is the scope of the book, you will hear more about it later.
How to capture
How to capture
Before we can gather packets we have to set up remote PCAP Server at any innovaphone device we want to trace. So go to
Furthermore you should know that you will find any activated logging information as innovaphone debug message in the trace as well. This often helps you in recreating the call flow if you aren't sure how the call is routed.
It's highly recommended to turn off the RPCAP check mark after you created the trace because a vicious attacker might use this open port to gather packets via rpcap for himself. In addition to the security issue any activated trace flag might also cause unnecessary CPU load. So if you want a healthy and secure system we recommend to turn off all check marks after creating the trace.
The next step we have to take in capturing traffic, is setting up a remote interface in our Wireshark. Here is a video tutorial on how this is done. video on setting up a remote interface
If you want to stop the trace you need to press the red square. After you have done that you are free to go and explore all the features of Wireshark.
How Wireshark works
How Wireshark works
After setting up Wireshark we should talk about how the captured packets are transmitted to your Wireshark application. The default Port of Wireshark is TCP/2002 but this port is just used while the connection is established. All Innovaphone debug messages and packets of the RPCAP servers network interface are buffered until the Wireshark connects. As soon as client and server managed to get a connection, two randomly chosen ports are used to exchange data. The buffer at the RPCAP server has just a limited space and as a result it's possible that Wireshark can't read the captured packets as fast as they keep coming in. In that case there will be a buffer overflow and the oldest packets will be dropped. This situations have to be avoided because useful information might be dropped and the trace gets useless. After creating the trace you should check the trace for TRACE-LOST Innovaphone debug messages because they indicate a buffer overflow. There is an easy way to accomplish this just filter the trace for innovaphone contains "LOST". Usually this problem happens if the connection between the RPCAP server and the Wireshark application is not good enough. Creating the trace from right next to it often does the trick to solve the problem.
Sometimes if you have a sporadic problem you will need to set up a unattended long term Wireshark trace but in most cases you don't want one huge trace file to analyse. Wireshark has a good solution to it and you will find it at Capture/Options. The feature "Use multiple files" allows you to automatically create an additional trace file either per time (seconds, minutes, hours) or per bytes (kilobytes, megabytes, gigabytes). In order to maintain overview Wireshark adds a five-digit sequential number and the date and time to the name of the file. As a consequence you can easily find the correct trace if you know the precise time the problem occurred. Furthermore you can use a ring buffer which avoids filling up your hard drive because it automatically deletes the oldest trace file outside the ring buffer range.
Choosing the correct traceflag
Choosing the correct traceflag
It's not always easy to know which trace flag will give us the desired information. As we talked before simply activating all traceflags is not a good idea because this will definitly result in a buffer overflow and consequently the trace will be unreadable. Furthermore you don't want to be to careful in setting traceflags, because this might result in a trace that won't contain the information we are looking for. So let's talk about which trace flag to choose if you want to debug a certain problem. Each of the following six sub chapter will contain a common problem we are facing each day in support.
Poor Audio Quality
Poor Audio Quality
Asume a customer reports an bad voice quality during a call. The first thing we need to know is, if the established call was a pure VoIP call and which codec was used during the communication. It's essential to know that the RTP(audio) connection is endpoint to endpoint by default but if you activate media relay at the PBX, RTP traffic will be passed through the PBX. In order to capture RTP Traffic we will have to create the trace from the endpoints. If you activate the ALL IPv4 TCP/UDP traceflag every Layer 4 Segment/Datagram that's originating or destined for the Innovaphone device will be part of the trace. No other traceflag would capture RTP consequently we need to activate it to debug this problem.
Wireshark has an own tool for debugging RTP Traffic. You'll find it at Telephony/RTP/Show all Streams in your Wireshark. All RTP Stream contained in the trace are displayed in this dialogue. If you know the source IP and port and the destination IP and port you can find the buggy RTP stream. By clicking Analyze you can take a look at the health of the stream. A dropped packet, or a wrong sequence number will be highlighted. If you press Player you can even listen to the stream if it's G.711 encoded and not encrypted.
Sometimes Innovaphone Support will ask you for a DSP trace because the poor audio quality seems to be originating from an Innovaphone device. In that case you should go to http://x.x.x.x/debug.xml of your device and activate the required DSP trace flags. Innovaphone Support will tell which traceflag is required.
One way audio
One way audio
One way audio is often related to a 3rd party interoperability problem, that's why we need to trace the signalling at the interface between Innovaphone and the 3rd party product. In most cases that's the PBX so we need to discuss which trace flag to activate at the PBX.
It's important to know that any internal communication between the PBX part and the gateway part of the innovaphone device will be done via H.323. As a result capturing the H.323 traffic will be wise. If you already capturing All IPv4 TCP/UDP Traffic any H.323 will be part of the trace. If you want to distinguish if the H.323 communication is external or internal, you have to take a look at the IP address. An internal communication will always be between two loopback addresses (for example 127.8.171.1 and 127.8.1.1).
If your 3rd party product is using SIP, you have to check the correct SIP traceflag. Since SIP is an application layer protocol it can work independent of its transport layer. We are talking about SIP if the transport layer is UDP, TSIP if the transport layer is TCP and if SIP/TCP is encrypted by TLS we are talking about SIPS. At the tracing page you'll find a traceflag for each of those SIP methods so you have to check the appropriate check-mark.
If the customer is experiencing one way audio we have to be curious about the RTP traffic as well. There might be a one-sided RTP Stream or the RTP stream contains only silence. As we talked before we have to trace RTP traffic from the endpoint if Media Relay isn't active. You might be tempted to capture the PBX traffic and the phones traffic in one trace file, if you create two RPCAP interfaces at your Wireshark application. However we have to advise against it, because reading the trace becomes hardly possible afterwards. Instead we recommend using two instances of Wireshark to capture the traffic separately on each device.
Update script failure
Update script failure
Debugging the update script can be tedious because you have to know which traceflag to activate. The update mechanism contains of two components, UP1 and UP0 . While UP1 is constantly polling the update script configured at Services/Update, UP0 is executing those commands. Those two components can be easily traced if you go to x.x.x.x/debug.xml and activate the trace flags shown in the picture. Since the Update mechanism is using HTTP, we highly recommend to activate the HTTP client and verbose traceflag while capturing because this will tell you the actual HTTP conversation between Innovaphone device and script storage space.
In case the Update script is using HTTPS don't forget to activate the ALL IPv4 TLS Traffic trace flag because otherwise you won't be able to see the HTTPS communication, since it's encrypted.
ISDN problem
ISDN problem
If the customer is experiencing an ISDN related problem the logging information is often not enough for debugging. The logging page will show you if an interface is going up and down uncontrollably but it won't show us the Layer2 and Layer3 information. You need to activate the according BRI/PRI tracing flag at the tracing page to see the information in your Wireshark. Q.931 is the Layer3 information and LAPD and TEI is the Layer2 information. It's important to know that 127.0.0.1 is the device you are capturing from and 127.0.0.0 is the device on the opposite site of your BRI/PRI Interface. This way we can find out if we are receiving or sending a message. Those messages are decoded by the Innovaphone Wireshark dll consequently you'll need it if you want to debug an ISDN related problem.
Voicemail Error
Voicemail Error
Finding the problem in a malfunctioning voicemail installation can be sometimes a bit frustrating. This comes from the fact that the VM solution is composed from 4 different independent elements.
Troubleshooting Matrix
XML script
To debug this issue, but also as a general procedure for all cases, the Trace Flag in the PBX Voicemail object must be enabled.
HTTP
You can further investigate on this problem by enabling the HTTP - Client trace flag in the Diagnostics menu. A common mistake is the missing HTTP client authentication at a remote WebDAV server. If the WebDAV server is on the same device as the VM object (i.e. PBX), HTTP authentication is not needed.
In case that the e-mail notification is not working as desired, you should look at the DNS and SMTP traffic. There is a hidden Trace flag for both modules. You can add those two Trace flags via your Browser if you enter those commands and write it to the config.
x.x.x.x/!config add DNS0 /trace
x.x.x.x/!config add SMTP0 /trace
x.x.x.x/!config write
x.x.x.x/!config activate
If you want to delete those traceflags you have to enter the following.
x.x.x.x/!config rem DNS0 /trace
x.x.x.x/!config rem SMTP0 /trace
x.x.x.x/!config write
x.x.x.x/!config activate
Audio or MWI
In order to solve this types of problems, you must look at the signalling information(H.323 or SIP) between the calling phone and the PBX. Also check for exclusive codec settings at the devices in use, since this might cause media negotiation problems.
LDAP search error
LDAP search error
Another common problem is that the name resolution isn't working as it should. For tracing LDAP just activate the All IPv4 TCP/UDP Traffic at the PBX if you are facing a problem with myPBX and in case the LDAP query is originating from the phone you have to trace at the phone. If the external LDAP server isn't working at the default port 389 you won't find the traffic by filtering the capture for "ldap". You have to filter the capture for the configured TCP port (tcp.dstport == xxx) . If you found those packets you have to right click and then press decode as . Another window will open which allows you to search for LDAP (Transport). If you click OK, the TCP message will be decoded as LDAP messages. LDAP data is transmitted as cleartext unless you are using LDAPS.In case you are using LDAPS, the traffic will be encrypted by TLS consequently you have to activate the ALL IPv4 TLS Traffic trace flag.
Debugging the registraton process
Debugging the registraton process
Debugging a VOIP call
Debugging a VOIP call
To debug the trace we should first look at the VoIP Calls feature of Wireshark to find the call in the trace. If you can't find the call in there, signalling traffic will be missing.
As soon as ICE candidates are exchanged both phones are trying to find the best way to reach the opposite side. ICE is sending a STUN packet the same way a RTP packet would take to check if a connection can be established. This video illustrates how you are able to find the information in a Wireshark trace. In the end ICE should find a way by checking every possible connection between the candidates.
Debugging a call to the SIP Provider
Debugging a call to the SIP Provider
To make things more interesting we extended our setup with a SIP provider simulation at 172.16.84.10 and redesigned the internal network to a 192.168.1.0/24 network. This trace was created from the PBX and All IPv4 TCP/UDP traffic and the SIP/UDP Trace flag at x.x.x.x/debug.xml was active during the capture process. The SIP trace flag at the debug page is a very good tool because it gathers all SIP as well as the ICE related debug messages. If you don't know how SIP signalling works, I recommend reading the VoIP Protocols book before we go through the trace together.
Once again the starting point to analyse the trace is checking the VoIP calls feature of Wireshark to see if the call is actually in the trace. The call has been made from UserA and extension number 100 to the destination number 000123456. Since we talked about the internal communication and the exchange of candidates in H.323 before, we should focus at the SIP signalling now. By clicking at the INVITE in the call flow graph, we will automatically jump to the INVITE in the list of traced packets. Since a packet is encapsulated and SIP is a application Layer protocol, youll find the SDP information if you expand the SIP tree and Message Body. The codec information is exchanged in the Media Description (m-line), while the Media Attributes (a-line) just describe the codec information in more detail. In the past the RTP endpoint address was exchanged in the connection information as part of the media description or the session description. Nowadays the connection information has to be part of the Media Description because Audio and Video demand two different RTP channels, but it's not really used because ICE provides a much better service in negotiating the best way to send and receive RTP packets. The ICE candidates are transmitted as media attributes.
You'll find the answer from the provider in the OK message. You'll see several numbers in the m-line. Those numbers define the Payload of the RTP traffic. Since 101 is used to allow the transmission of DTMF tones and 13 is used for comfort noise, we know that g711a was used as speech codec because 8 was defined by IETF as Profile for g711a.You'll find a detailed description of the Audio/Video Profiles in RFC-3551. In the end candidates are exchanged the same way as we saw in the INVITE.
For debugging ICE we can filter the trace for innovaphone contains ICE again and search for the Checklist, Candidates and Conclusion innovaphone debug message. This will tell us if the connection was successfully and which candidate was nominated for the RTP traffic. In our trace the connection was established directly via the NAT Router. The SIP provider simulation will send its RTP traffic to the received Server Reflex address.