Course12:Advanced - Troubleshooting with Wireshark

From innovaphone wiki
Jump to navigation Jump to search

The goal of this topic is to aquire a fundamental knowledge of troublehsooting Innovaphone Problems with Wireshark

Setting up Wireshark

Getting started

Before we can get started to learn about Wireshark, you should download the application from the https://class.innovaphone.com/moodle2/mod/resource/icon.gif Recommended Tools and Firmware for this Course. All videos were created using version 1.12.9, so we recommend downloading and using this version. Wireshark will ask you to update to the new version - don't do it!

You may ask yourself why we are not using the new 2.0 release of Wireshark. After testing the new version we came to the conclusion that it isn't fitting our needs.

The next step to get going with Wireshark is downloading the latest Wireshark dll from our download server. You'll find a 32 bit and a 64 bit version (depending on your PC's architecture) in the tools item of the download.png apps package on the Version 6 download page. For convenience, this is also available on the https://class.innovaphone.com/moodle2/mod/resource/icon.gif Recommended Tools and Firmware for this Course (Tools Package).

After downloading the appropriate dll you need to paste it to the Wireshark plugins folder. Usually you should find the plugin folder here: C:\Program Files\Wireshark\plugins\1.12.9.

This dll decodes innovaphone specific debug messages as well as Q931 and LAPD frames. While Q931 and LAPD will help you in debugging ISDN related problems, innovaphone debug messages will give you an inside in all kind of problems. But since this is the scope of the book, you will hear more about it later.


How to capture

How to capture

Before we can gather packets we have to set up remote PCAP Server at any innovaphone device we want to trace. So go to Maintenance / Diagnostics / Tracing and activate the Enable RPCAP check mark and any further check mark you are interested in. Usually you will tick All IPv4 TCP/UDP Traffic so that Wireshark will capture all network traffic from and to this device. Additionally you want to tick the All IPv4 TLS Traffic check mark because most signalling traffic is encrypted nowadays. This trace flag allows Wireshark to gather the decrypted information because otherwise the trace is useless.

Furthermore you should know that you will find any activated logging information as innovaphone debug message in the trace as well. This often helps you in recreating the call flow if you aren't sure how the call is routed.

It's highly recommended to turn off the RPCAP check mark after you created the trace because a vicious attacker might use this open port to gather packets via rpcap for himself. In addition to the security issue any activated trace flag might also cause unnecessary CPU load. So if you want a healthy and secure system we recommend to turn off all check marks after creating the trace.

The next step we have to take in capturing traffic, is setting up a remote interface in our Wireshark. Here is a video tutorial on how this is done. video on setting up a remote interface

If you want to stop the trace you need to press the screenshot.png red square. After you have done that you are free to go and explore all the features of Wireshark.


How Wireshark works

How Wireshark works

After setting up Wireshark we should talk aboutscreenshot.png how the captured packets are transmitted to your Wireshark application. The default Port of Wireshark is TCP/2002 but this port is just used while the connection is established. All Innovaphone debug messages and packets of the RPCAP servers network interface are buffered until the Wireshark connects. As soon as client and server managed to get a connection, two randomly chosen ports are used to exchange data. The buffer at the RPCAP server has just a limited space and as a result it's possible that Wireshark can't read the captured packets as fast as they keep coming in. In that case there will be a buffer overflow and the oldest packets will be dropped. This situations have to be avoided because useful information might be dropped and the trace gets useless. After creating the trace you should check the trace for TRACE-LOST Innovaphone debug messages because they indicate a buffer overflow. There is an easy way to accomplish this just filter the trace for innovaphone contains "LOST". Usually this problem happens if the connection between the RPCAP server and the Wireshark application is not good enough. Creating the trace from right next to it often does the trick to solve the problem.

Sometimes if you have a sporadic problem you will need to set up a unattended long term Wireshark trace but in most cases you don't want one huge trace file to analyse. Wireshark has a good solution to it and you will find it at screenshot.png Capture/Options. The feature "Use multiple files" allows you to automatically create an additional trace file either per time (seconds, minutes, hours) or per bytes (kilobytes, megabytes, gigabytes). In order to maintain overview Wireshark adds a five-digit sequential number and the date and time to the name of the file. As a consequence you can easily find the correct trace if you know the precise time the problem occurred. Furthermore you can use a ring buffer which avoids filling up your hard drive because it automatically deletes the oldest trace file outside the ring buffer range.

Choosing the correct traceflag

Choosing the correct traceflag


It's not always easy to know which trace flag will give us the desired information. As we talked before simply activating all traceflags is not a good idea because this will definitly result in a buffer overflow and consequently the trace will be unreadable. Furthermore you don't want to be to careful in setting traceflags, because this might result in a trace that won't contain the information we are looking for. So let's talk about which trace flag to choose if you want to debug a certain problem. Each of the following six sub chapter will contain a common problem we are facing each day in support.

      Poor Audio Quality

      Poor Audio Quality


      Asume a customer reports an bad voice quality during a call. The first thing we need to know is, if the established call was a pure VoIP call and which codec was used during the communication. It's essential to know that the RTP(audio) connection is endpoint to endpoint by default but if you activate media relay at the PBX, RTP traffic will be passed through the PBX. In order to capture RTP Traffic we will have to create the trace from the endpoints. If you activate the ALL IPv4 TCP/UDP traceflag every Layer 4 Segment/Datagram that's originating or destined for the Innovaphone device will be part of the trace. No other traceflag would capture RTP consequently we need to activate it to debug this problem.

      Wireshark has an own tool for debugging RTP Traffic. You'll find it at screenshot.png Telephony/RTP/Show all Streams in your Wireshark. All RTP Stream contained in the trace are displayed in this dialogue. If you know the source IP and port and the destination IP and port you can find the buggy RTP stream. By clicking screenshot.png Analyze you can take a look at the health of the stream. A dropped packet, or a wrong sequence number will be highlighted. If you press screenshot.png Player you can even listen to the stream if it's G.711 encoded and not encrypted.

      Sometimes Innovaphone Support will ask you for a DSP trace because the poor audio quality seems to be originating from an Innovaphone device. In that case you should go to screenshot.png http://x.x.x.x/debug.xml of your device and activate the required DSP trace flags. Innovaphone Support will tell which traceflag is required.

      One way audio

      One way audio


      One way audio is often related to a 3rd party interoperability problem, that's why we need to trace the signalling at the interface between Innovaphone and the 3rd party product. In most cases that's the PBX so we need to discuss which trace flag to activate at the PBX.

      It's important to know that any internal communication between the PBX part and the gateway part of the innovaphone device will be done via H.323. As a result capturing the H.323 traffic will be wise. If you already capturing All IPv4 TCP/UDP Traffic any H.323 will be part of the trace. If you want to distinguish if the H.323 communication is external or internal, you have to take a look at the IP address. An internal communication will always be between two loopback addresses (for example 127.8.171.1 and 127.8.1.1).

      If your 3rd party product is using SIP, you have to check the correct SIP traceflag. Since SIP is an application layer protocol it can work independent of its transport layer. We are talking about SIP if the transport layer is UDP, TSIP if the transport layer is TCP and if SIP/TCP is encrypted by TLS we are talking about SIPS. screenshot.png At the tracing page you'll find a traceflag for each of those SIP methods so you have to check the appropriate check-mark.

      If the customer is experiencing one way audio we have to be curious about the RTP traffic as well. There might be a one-sided RTP Stream or the RTP stream contains only silence. As we talked before we have to trace RTP traffic from the endpoint if Media Relay isn't active. You might be tempted to capture the PBX traffic and the phones traffic in one trace file, if you create two RPCAP interfaces at your Wireshark application. However we have to advise against it, because reading the trace becomes hardly possible afterwards. Instead we recommend using two instances of Wireshark to capture the traffic separately on each device.

      Update script failure

      Update script failure


      Debugging the update script can be tedious because you have to know which traceflag to activate. The update mechanism contains of screenshot.png two components, UP1 and UP0 . While UP1 is constantly polling the update script configured at fish-help.png Services/Update, UP0 is executing those commands. Those two components can be easily traced if you go to screenshot.png x.x.x.x/debug.xml and activate the trace flags shown in the picture. Since the Update mechanism is using HTTP, we highly recommend to activate the HTTP client and verbose traceflag while capturing because this will tell you the actual HTTP conversation between Innovaphone device and script storage space.

      In case the Update script is using HTTPS don't forget to activate the ALL IPv4 TLS Traffic trace flag because otherwise you won't be able to see the HTTPS communication, since it's encrypted.

      ISDN problem

      ISDN problem


      If the customer is experiencing an ISDN related problem the logging information is often not enough for debugging. The logging page will show you if an interface is going up and down uncontrollably but it won't show us the Layer2 and Layer3 information. You need to activate the according BRI/PRI tracing flag at the tracing page to see the information in your Wireshark. Q.931 is the Layer3 information and LAPD and TEI is the Layer2 information. It's important to know that 127.0.0.1 is the device you are capturing from and 127.0.0.0 is the device on the opposite site of your BRI/PRI Interface. This way we can find out if we are receiving or sending a message. Those messages are decoded by the Innovaphone Wireshark dll consequently you'll need it if you want to debug an ISDN related problem.

      Voicemail Error

      Voicemail Error


      Finding the problem in a malfunctioning voicemail installation can be sometimes a bit frustrating. This comes from the fact that the VM solution is composed from 4 different independent elements.

      screenshot.png Troubleshooting Matrix

      XML script


      To debug this issue, but also as a general procedure for all cases, the Trace Flag in the screenshot.png PBX Voicemail object must be enabled.

      HTTP


      You can further investigate on this problem by enabling the HTTP - Client trace flag in the Diagnostics menu. A common mistake is the missing HTTP client authentication at a remote WebDAV server. If the WebDAV server is on the same device as the VM object (i.e. PBX), HTTP authentication is not needed.

      E-mail


      In case that the e-mail notification is not working as desired, you should look at the DNS and SMTP traffic. There is a hidden Trace flag for both modules. You can add those two Trace flags via your Browser if you enter those commands and write it to the config.

      x.x.x.x/!config add DNS0 /trace
      x.x.x.x/!config add SMTP0 /trace
      x.x.x.x/!config write
      x.x.x.x/!config activate

      If you want to delete those traceflags you have to enter the following.

      x.x.x.x/!config rem DNS0 /trace
      x.x.x.x/!config rem SMTP0 /trace
      x.x.x.x/!config write
      x.x.x.x/!config activate

      Audio or MWI


      In order to solve this types of problems, you must look at the signalling information(H.323 or SIP) between the calling phone and the PBX. Also check for exclusive codec settings at the devices in use, since this might cause media negotiation problems.

      LDAP search error

      LDAP search error


      Another common problem is that the name resolution isn't working as it should. screenshot.png For tracing LDAP just activate the All IPv4 TCP/UDP Traffic at the PBX if you are facing a problem with myPBX and in case the LDAP query is originating from the phone you have to trace at the phone. If the external LDAP server isn't working at the default port 389 you won't find the traffic by filtering the capture for "ldap". You have to filter the capture for the configured TCP port screenshot.png (tcp.dstport == xxx) . If you found those packets you have to right click and then press screenshot.png decode as . Another window will open which allows you screenshot.png to search for LDAP (Transport). If you click OK, the TCP message will be decoded as LDAP messages. LDAP data is transmitted as cleartext unless you are using LDAPS.In case you are using LDAPS, the traffic will be encrypted by TLS consequently you have to activate the ALL IPv4 TLS Traffic trace flag.

      Debugging the registraton process

      Debugging the registraton process



      Before we can start debugging calls we have to talk about the registration process. I strongly recommend to read the H.323 chapter in our VOIP protocols book before we go on. An Innovaphone phone usually uses H.323 to register at the PBX. H.225 RAS (Registration, Admission and Status ) is used to perform a registration process between a phone and its PBX or a slave PBX to its Master PBX. For establishing, controlling and ending of a call H.323 uses H.225 CS (call signalling) messages which are TCP based.

      Nowadays RAS messages are independent from its transport layer. Originally RAS messages were designed to be sent via UDP but this isn’t true anymore. You can choose to tunnel RAS messages into H.225 CS messages which are sent via TCP. You can even choose to encrypt those H.225 CS messages for security reasons via TLS. That’s why you have the possibility screenshot.png to choose the registration method at an innovaphone device.

      So let's have a look into a Wireshark trace of those two registration methods. Please download the trace file https://class.innovaphone.com/moodle2/pix/f/web.gif here. In order to find the appropriate information we should filter the traffic for H.225 traffic because it makes the search for a specific packet much easier. If we do so we will find packets sent between 172.16.161.12 and 172.16.161.14 as well as packets between 172.16.161.12 and 172.16.161.15.

      Let's focus on the traffic between 172.16.161.12. and 172.16.161.14 first. 172.16.161.14 is an IP111 using the H.323/UDP method to register. As we learned in theVOIP protocols book when an endpoint is powered on, it will try to find its responsible PBX. It uses the Gatekeeper Discovery mechanism for it, if no Gatekeeper was configured in advance. The trace will show us that Gatekeeper discovery was used indeed, consequently we know that no primary Gatekeeper was configured. As an answer to our Gatekeeper Request, the Gatekeeper will then respond with a gatekeeper confirm. The endpoint will then send a Registration Request to the answering Gatekeeper on his side, which is hopefully answered by a Registration Confirm. If everything goes as planned the phone is registered and will be able to make calls. This video will show you how you can find those packets in the trace. H.323/UDP registration

      Up next let's talk about the H.323/TCP method. It's important to know that H.323/TCP doesn't support gatekeeper discovery so you have to configure a Primary Gatekeeper address at the phone or distribute it via DHCP. By filtering the trace you will find out that the IP232 sends its RAS Registration request as part of a H.225 message. The registration request is tunnelled through TCP. This video will assist you if you have trouble finding the packets in the traceH.323/TCP registration


      Debugging a VOIP call

      Debugging a VOIP call


      After we registered both our phones we will now discuss a basic call between UserA and UserB, but to make things more interesting we are analysing a basic call using ICE. To get the idea how ICE works, you should read the ICE chapter in our Reverse Proxy book first.

      Once again there is a Wireshark trace provided by us you can download https://class.innovaphone.com/moodle2/pix/f/web.gif here. This time it's not created from the PBX instead it's captured from the phone because the signalling is simple and we are more interested in the exchange of ICE candidates and taking a look at the connectivity checks. We used the screenshot.png All IPv4 TCP/UDP and H.323 ICE trace flag to capture the appropriate data. This trace was created in screenshot.png a labwhich has no direct access to the Internet, consequently we had to put the STUN/TURN server in a different Class C network which was accessible via a NAT Router.

      To debug the trace we should first look at the VoIP Calls feature of Wireshark to find the call in the trace. If you can't find the call in there, signalling traffic will be missing.

      The call flow shows us that the signalling of the call was done between the PBX and the phone, but the RTP channel was established between the two phones directly. As a result we know that Media Relay (RTP Proxy) wasn't active. Let's focus on the exchange of ICE candidates. The exchange of ICE candidates is done via H.323. The SETUP message contains the candidates of the initiating phone and the CONNECT contains the candidates of the answering phone. The video shows an easy way to find the exchanged candidates. screenshot.png As you see both phones tell each other 3 different addresses. The HOST address is the local IP address of the phone, while the SRFLX (Server reflex address) is the public address of the NAT router returned by the STUN server. The RELAY address is the address of the TURN server. In this basic scenario a STUN or a TURN server is not necessary because both phones are in the same network, consequently they have no problem in routing traffic to each other.

      As soon as ICE candidates are exchanged both phones are trying to find the best way to reach the opposite side. ICE is sending a STUN packet the same way a RTP packet would take to check if a connection can be established. This video illustrates how you are able to find the information in a Wireshark trace. In the end ICE should find a way by checking every possible connection between the candidates.




      Debugging a call to the SIP Provider

      Debugging a call to the SIP Provider



      To make things more interesting we extended our screenshot.png setup with a SIP provider simulation at 172.16.84.10 and redesigned the internal network to a 192.168.1.0/24 network.https://class.innovaphone.com/moodle2/pix/f/web.gif This trace was created from the PBX and All IPv4 TCP/UDP traffic and the screenshot.png SIP/UDP Trace flag at x.x.x.x/debug.xml was active during the capture process. The SIP trace flag at the debug page is a very good tool because it gathers all SIP as well as the ICE related debug messages. If you don't know how SIP signalling works, I recommend reading the VoIP Protocols book before we go through the trace together.

      Once again the starting point to analyse the trace is checking the VoIP calls feature of Wireshark to see if the call is actually in the trace. The call has been made from UserA and extension number 100 to the destination number 000123456. Since we talked about the internal communication and the exchange of candidates in H.323 before, we should focus at the SIP signalling now. By clicking at the INVITE in the call flow graph, we will automatically jump to the INVITE in the list of traced packets. Since a packet is encapsulated and SIP is a application Layer protocol, you’ll find the screenshot.png SDP information if you expand the SIP tree and Message Body. The codec information is exchanged in the Media Description (m-line), while the Media Attributes (a-line) just describe the codec information in more detail. In the past the RTP endpoint address was exchanged in the connection information as part of the media description or the session description. Nowadays the connection information has to be part of the Media Description because Audio and Video demand two different RTP channels, but it's not really used because ICE provides a much better service in negotiating the best way to send and receive RTP packets. The ICE candidates are transmitted as media attributes.

      You'll find the answer from the provider in the screenshot.png OK message. You'll see several numbers in the m-line. Those numbers define the Payload of the RTP traffic. Since 101 is used to allow the transmission of DTMF tones and 13 is used for comfort noise, we know that g711a was used as speech codec because 8 was defined by IETF as Profile for g711a.You'll find a detailed description of the Audio/Video Profiles in www.png RFC-3551. In the end candidates are exchanged the same way as we saw in the INVITE.

      For debugging ICE we can filter the trace for innovaphone contains ICE again and search for the screenshot.png Checklist, Candidates and Conclusion innovaphone debug message. This will tell us if the connection was successfully and which candidate was nominated for the RTP traffic. In our trace the connection was established directly via the NAT Router. The SIP provider simulation will send its RTP traffic to the received Server Reflex address.