Course13:IT Plus - Tools for Troubleshooting

The book gives a deeper understanding how to get information to troubleshoot an issue

Scope of the book

In this book, we will take a look at the various troubleshooting tools available to us. We will talk about how to use them to analyze a problem and to find a working solution. We will not only look at the theory behind the protocols and the Innovaphone solutions, but we will also test our skills with practical exercises. Please feel encouraged to stop reading if you encounter a problem and try to solve it yourself.

Please upload the start configuration this lesson to your devices, afterwards click  the button on your device page to upload the start configuration to your app services as well.

It is necessary to factory reset all phones after uploading the start configuration to your App Service.

As a next step please log-in to your myApps client with user john.doe and password ip411

Wireshark

What is Wireshark

Wireshark is an open source tool that allows us to capture and analyze network traffic. This means that Wireshark enables us to view the traffic inside our network interface. As a result Wireshark is a great tool for understanding the source of a protocol error.

How to get Wireshark

Of course, you can download the latest version of Wireshark from their website, but since all screenshots, videos and explanations were created with the Wireshark version found in the  Recommended Tools for this Course section, we recommend using this version.

As a first task, please go ahead and install Wireshark.

Although Wireshark  detects that a new version is available when you start it, please click Skip this version because the book was created with version 3.4.6.

How to create a capture

How to create a local capture

When you start Wireshark for the first time , the menu, the main toolbar, the filter toolbar and the capture area is displayed on the  start page of the application. Throughout this book, we will look at the different sections of this start page. First, let us take a look at the Capture area. The Capture area lists the local network interfaces on your PC.

An easy way to create a capture of one of your local network interfaces is to double click on the interface name.  Wireshark will immediately start capturing packets on this interface.

To stop the capture click on the red square.

Packet list overview

As soon as you start the trace,  3 new sections will be displayed in Wireshark (note that these sections can be resized, so some may be small or even not visible).

• Packet list: The upper section is a list of all packets. Each line in this list represents an IP packet and contains important information like source and destination address, protocol and time.

• Packet detail:The middle section shows a more detailed view of the current packet. The packet is divided into layers of the OSI model and displayed in a tree that can be expanded and collapsed.

• Packet bytes: The third section shows the bytes actually transmitted. This information is rarely useful, so this section will not be discussed further in this book.

How to create a remote capture

In many cases, you are not interested in the traffic of your PC, but you want to see the traffic of another component, such as the PBX or a phone. For this situation, you can create a remote capture.

To do this, we need to set up a Remote PCAP server (RPCAP) on our innovaphone device  as shown in the video. We will use our IP411LEFT for this.

So

• go to Maintenance / Diagnostics / Tracing and  activate the Enable RPCAP checkbox. This option allows Wireshark to connect
• to collect packets, we need to enable trace options. In this case we will enable the
  
• All IPv4 TCP/UDP Traffic and
• All IPv4 TLS Traffic options

The All IPv4 TLS traffic is important to be able to see encrypted packets, otherwise we cannot analyze HTTPS or H.323/TLS

It is highly recommended to disable the RPCAP checkbox after creating the trace, because a malicious attacker could use this open port to capture packets. In addition to the security issue, any enabled trace flag can also cause unnecessary CPU load. If you want a healthy and secure system, we recommend to disable all options after creating the trace.

Decoding innovaphone debug messages

How to get innovaphone's Wireshark dll's

If you look at  the trace sample, you will notice a lot of UDP packets sent from port 4 to port 4. These are actually no real network packets but innovaphone debug messages. However, Wireshark cannot decode them at the moment. To do so, Wireshark needs a plugin (known as Wireshark DLL).

Unfortunately, different versions of Wireshark require different versions of the Wireshark dll. A list of Wireshark versions and their corresponding Wireshark dll versions  can be found in our Wiki. As we are using Wireshark version 3.4.6 in this book, the Wireshark dll you need is version 1066.

To enable your Wireshark to decode these messages, you need to

• download the innovaphone Wireshark dll version 1066. You can find the appropriate dll for your system in our store at  store.innovaphone.com: release/download.htm. In the Software tab, scroll down to Wireshark DLLs.  As shown in the video, there are two different dll files. One for a 32-bit system and one for a 64-bit system
•  copy either the 32bit or 64bit innovaphone Wireshark dll (depending on your system) to the C:\Program Files\Wireshark\plugins\3.4\epan folder on your PC
• restart your Wireshark

Wireshark is now able to decode innovaphone debug messages and as a result we can  filter the created trace for innovaphone.

Getting rid of a persistently stored RPCAP interface

If you close and reopen Wireshark and the remote capture options, you will see that Wireshark has saved your remote RPCAP interface. While this is a nice feature when the device is always online with the same IP address, it becomes frustrating when the device is offline or has been assigned a different IP address via DHCP. As soon as you open the capture options, Wireshark will attempt to connect to the device. If the device is offline, Wireshark will fail to connect.

These interfaces are stored in a file called recent_common in the Wireshark's personal configuration folder (to find the personal configuration folder  just have a look at this video).

To inhibit wireshark from opening previous capture sources

• open the recent_common file with a text editor
• delete all the configured  recent.remote_host entries
•  set the file to Read-only
• and restart Wireshark

Now you can open the capture options without any error message.

How Wireshark works

Before we continue with the topic, we need to get a little more technical and talk about how Wireshark works.

 Figure Remote PCAP interface

When you establish a connection from your Wireshark application to an innovaphone device, Wireshark uses TCP port 2002 as the default port. Please note that an innovaphone device only listens to the default port, so do not change the port configuration in your Wireshark.

If you set up a remote PCAP interface in your innovaphone device, the most recent packets are buffered in the local memory. As soon as a connection is established on TCP port 2002, two randomly selected ports are used to transfer the captured data from the buffer of your innovaphone device to your Wireshark application. As a result, you will see the captured traffic in a list of packets in Wireshark.

Please note that the buffer on the RPCAP server (i.e. the innovaphone device) has a limited amount of space and it is therefore possible that Wireshark cannot read the captured packets as quickly as they arrive. In this case, a buffer overflow occurs and the oldest packets in the buffer are discarded. This situation must be avoided, otherwise useful information is lost and the trace becomes unusable.

After creating the trace, you should check the trace for TRACE-LOST innovaphone debug messages, as these indicate a buffer overflow. This problem usually occurs if the connection between the RPCAP server and the Wireshark application is not fast enough or if too many packets are captured due to too extensive trace flags. For this reason, it is recommended to trace from a PC in the customer's network. Also, only the trace flags that are really needed should be enabled.

Troubleshooting the registration process

In this chapter we will look at a failed registration. At first glance we can see that  the IP112 is not registered.

Since you are logged into the account of John Doe, you can go through the current configuration and make sure everything is set up correctly. So let's find out why:

• We would recommend starting the investigation in the Devices App to see if everything is set up as expected.

• Are you able to see the phone in your device app?

You can search for a device in the devices app either by (part of)  a name (e.g. 112),  a MAC address (e.g. 00903351) or  an IP address (IPv4 or IPv6) (e.g. 172.31.31.14). Just type the filter in the search bar. You can also enter only part of the search string and still find the device.

• Can you access the phone via the websocket connection?

In order for the phone to receive the  phone device configuration the phone has to  establish a websocket connection to the devices app. So if you cannot access the phone's advanced UI through your devices app, make sure  a sysclient URL is configured on the phone and the device is assigned to your dvl-ckl2.net domain.

• Did you assign a category to the phone?

Device configurations are assigned to devices based on their category. Therefore, it is important  to assign a category to the phone, otherwise the phone may not receive the correct or no configuration at all.

• Did you create a phone device configuration for this category?

You need to create  a phone device configuration for the phone's category.

• Are you using H323/TLS as protocol for the registration of your phone?

Because the Jane Doe user object  only allows TLS-based registrations, it is important  to select H323/TLS as the protocol for registration.

• Have you added a primary gatekeeper and, in case the registration is forwarded through a reverse proxy, a gatekeeper identifier?

It is important that the registration request is sent to the correct PBX. Therefore,  a primary gatekeeper and gatekeeper identifier must be configured.

You will most likely conclude that the configuration should work. Therefore, we need to dig a little deeper.

Logging information

A good starting point for troubleshooting is to check the logging information on Maintenance / Diagnostics / Logging.

Simply

• set the H.323 Registrations option on Maintenance / Diagnostics / Logging and click Ok
• then click on  the syslog link below

Alternatively, you could also create a Wireshark trace and if you have installed the innovaphone Wireshark dll, you can see  the logging information in the trace. This is useful whenever you are analyzing a trace and want to see the syslog context in the trace. This improves the readability of the trace because you can see which signaling packet belongs to which log message.

To get a useful error message about the registration process, we recommend that you create a log on the PBX rather than on the phone. When a log is created on the phone, it usually doesn't provide a useful error message; it only shows that  the registration was rejected. Phone logs may not have a helpful error message because the H.323 protocol often does not send meaningful reasons to the phone.

To try it out please

• open the syslog on your IP411LEFT as shown above
  
• and then reboot your IP112

Once the phone has rebooted, it will try to register with the PBX. It will send a registration request every 45 seconds. Unfortunately, you will not see any registration attempts from this phone on the PBX. This gives us an important clue. The PBX is not receiving the registration request, so we need to investigate the problem on the phone.

Filtering in Wireshark

To see what is happening on the phone (and why nothing is happening on the PBX) proceed as follows:

• go to the phone's advanced UI on Maintenance / Diagnostics / Tracing
•  tick only the
• All IPv4 TCP/UDP Traffic,
• All IPv4 TLS Traffic and
• Enable RPCAP trace options
• start a PCAP trace with Wireshark on your IP112 (remember to use a remote interface to Host 172.31.31.14)
  

Although we have limited the trace to only a few options, you can already see that the All IPv4 TCP/UDP Traffic option generates a lot of data. Since many of the packets are not useful for our analysis, we can filter the capture to show only interesting packets. This is why  wireshark has the display filter feature.

Let's go through some filter options that will help us analyze the trace in this situation. Please feel free to try them as you go.

• Let's start examining the trace by filtering for the protocol used to register with the PBX. As you recall, we configured the phone to use H323/TLS. As a result, we will filter the trace for h225. H.225 is part of the H.323 family and is used for call signaling and the registration process. After applying the filter, we should be able to see the registration requests in our trace. However, the truth is that the trace is empty. Obviously, there are no registration requests at all.

• The next step we should investigate is a TCP error. Perhaps a firewall is blocking port 1300, the default port for H323/TLS. We can define a specific TCP port in Wireshark to filter the traffic. So please use  the filter tcp.port==1300. Fortunately, this time we can see packets in our trace. We see  TCP SYN attempts with no response, which means that the TCP session could not be established.
  
  To complete the 3-way handshake, a SYN must be answered with an ACK SYN, which in turn is answered with an ACK. This explains why we did not see an h225 packet. In order to make a registration request, a TCP session must first be established.

• You may wonder why the phone is trying to establish a TCP connection to 172.31.31.199, isn't the IP of the PBX 172.31.31.2? You are absolutely right, this is not the correct destination IP.
  
  The phone will send its registration request to the configured primary gatekeeper hq.dvl-ckl2.net (which you can see at Phone / User-1).
  
  If hq.dvl-ckl2.net is associated with the wrong IP, there has to be a DNS error.
  
  You can now  filter the RPCAP trace for dns. You will see a  DNS query response for hq.dvl-ckl2.net (Queries) from 172.31.31.3 (Src:) with IP address 172.31.31.199 (Answers)
  
  You can also initiate a DNS request from your phone on Services / DNS / Query: type hq.dvl-ckl2.net and lick on the search button if you see no dns packet in your trace.
  

You have most likely found the answer to the problem. The DNS server on your IP811 has the wrong DNS entry for hq.dvl-ckl2.net.

The simplest solution to this problem is to set the IP112 to DHCP Mode client on IP4 / ETH0 / DHCP. As a result, the phone will receive the DHCP lease from the IP411RIGHT, which will distribute the correct DNS server and eh voilà: the IP112 will register as Jane Doe.

Additionally fix the DNS Entry hq.dvl-ckl2.net on your IP811 to point to 172.31.31.2

Advanced filters

You can combine display filters with comparison and logical operators such as or or and in order to better filter the traffic of interest. This way you can combine all the filter options you use into one expression.

This table lists all operators available in Wireshark. You can enter either the symbol or the text of an operator to create a filter.

Description	Symbol	Text
equal to	==	eq
or	||	or
and	&&	and
greater than	>	gt
less than	<	lt
greater than or equal to	>=	ge
less than or equal to	<=	le
not	!	not
not equal to	!=	ne
contains		contains

One of the more interesting operators is the contains operator. It allows you to filter for a specific word or number that has to be part of the searched packet. A common use case for the contains operator is innovaphone contains "LOG ". This filter lists all logging messages in a trace.

To ensure that your expression is evaluated in the intended order, you should use parentheses to group the filter.

An example would be this filter: (ip.src==172.31.31.14 && tcp.port==1300) || tcp.port==443

This filter will only show you HTTPS packets and H323/TLS packets sent from the IP 172.31.31.14

Troubleshooting a VoIP call

As a next step we can test an internal call.

Call Richard Roe (13) from Jane Doe (IP112)

The signaling for this call works, but there is no sound and the call is automatically disconnected after 10 seconds.

So let's find out what might be going wrong here and create a new trace. This time we will set the All IP4 TCP/UDP Traffic, All IP4 TLS Traffic and H.323/ICE trace option on Maintenance / Diagnostics / Tracing and recreate the problem by placing the call again and wait until the call is disconnected.

Enabled Protocols

Even if you have enabled ALL IP4 TLS traffic, you will not see any decrypted H.323/TLS packets. This is because Wireshark cannot decrypt TLS packets if the trace was created on a phone and if TLS is enabled as a protocol.

To solve this issue, go to Analyze->Enabled Protocols and  uncheck TLS as protocol. (You can search for TLS in the search bar on the top)

Analyzing a Call Flow

To see if the call is actually included in the capture, we recommend  to openTelephony->VoIP Calls. The VoIP Calls window shows you a list of all VoIP calls detected in this capture.

This overview gives us a good amount of information about all the VoIP calls in this list.

• The start and end time of each call
• The IP address of the initial caller
• The From and To address/number of each call
• The protocol used for the call (H.323 or SIP in most cases)
• How many signaling packets have been exchanged during the call
• Whether the call establishment was successful or not

You may be surprised to see more than one call in this overview, even though we only made one call. The reason is that VoIP protocols can be used to signal more than just audio calls. H.323 or SIP can also be used to transmit a presence subscription or a chat message.These calls do not require a media channel (RTP channel).

As a pro tip: innovaphone devices always use Extended Fast Connect (EFC) to establish an audio channel. This means that  you only need to search for calls with Tunneling and Fast Start on in the comments section of this overview.

Try to find the call we just made in this list.  If you click on Flow Sequence, after selecting the correct call, a new window will appear showing us the call flow.

This call flow image shows us all the IP addresses involved in setting up the call. Each vertical line represents one IP endpoint. Each horizontal line represents an IP packet, labeled with the source and destination port of that packet. The arrow indicates which destination a packet is being sent to.

Let's take a look at the important transmitted packets:

• Setup: This packet initiates a call. In our case, the IP112 sends a SETUP message to the PBX (gatekeeper), which is then forwarded to the IP222 by the PBX. The phone sends its codec offer and RTP endpoint address to the called party.

• Alerting: The IP222 responds with an alert message to indicate that it is ringing.

• Connect: When you pick up the call on the IP222, the phone sends a Connect message. The Connect message is used to respond to the codec offer and to inform the caller of his codec selection and RTP endpoint address.

• Facility: Facility messages can be ignored for the time being. These messages contain information about additional services that are not relevant for call setup.

• ReleaseComplete: This packet initiates the termination of the call.

After reading the trace, we know that all expected signaling messages have been exchanged. Both parties have received an endpoint address from the other phone, but no RTP traffic is exchanged. So we need to dig deeper to find the cause of our problem.

ICE

Since we concluded that signaling is not the problem, the next step is to look at the ICE mechanism to see if we have found a way to exchange RTP packets. We have already discussed the ICE mechanism in our advanced training. If you need a refresher  please read the book. Now we will look at the trace to see if the packets were transmitted.

An ICE trace must be created on the IP endpoint of a call. This is because the endpoint itself is responsible for finding the best path to exchange RTP packets.

 Filter the trace for innovaphone contains "ICE"

In this rare case we start troubleshooting at the end of the call and not at the beginning.

• Conclusion: Since no working network path was found,  ICE mechanism failed is reported. Other results are Success or Aborted. Aborted means that the ICE mechanism was aborted before a conclusion could be reached.

• Candidates: In the Candidates packet we see that both phones  only exchanged their HOST address. An SRFLX(Server reflex) or RELAY (TURN Server) address is missing. As a result we can conclude that a STUN/TURN server was not configured or the configured STUN/TURN server could not be reached.

• Checklist: The checklist  shows us all connection attempts to establish an RTP stream. Since only HOST addresses were exchanged, only these are listed. Usually, connection pairs from all exchanged candidates can be found in this packet.

To demonstrate a non-functioning call, we have configured a static route to a non-existent address in the IP configuration of your IP222. This way, no IP packets can be exchanged between the phones.

To see if our assumption is correct, we will check if both phones received a Media Device Configuration from the Devices App:

• Please go to your Devices app
• Select your domain (dvl-ckl2.net) in the Domains section of the app
• Select Device configuration
• click on the little down-arrow of the [Media] Global entry

You will notice that the configuration has  only been applied to the IP811 and an IP411 but not to the phones. A look at the categories listed reveals the reason for it: there are no categories for IP phones.

To fix that proceed a follows:

• Open the Media Configuration
  
  The hq IP Phones category is indeed missing which means that all phones of this category won't receive any STUN or TURN server
  
  
• To fix our problem we have to make sure that the media configuration is used for all devices and add the hq IP Phone category to the Categories list of the Media device configuration

Now the configuration is  applied to the IP phones too.

We can capture a new trace from another call from Jane to Richard which will work fine now.

The trace of this working call shows us that the connection was established through the TURN server. The TURN server is located on our IP411LEFT (172.31.31.2).

•  The conclusion is a success.
•  The candidate list shows a RELAY and an SRFLX address in addition to the HOST address from both, the local and remote sides.
•  The checklist shows us which path was tested and which path was nominated in the end.

Note that usually there are more than one working RTP paths available. In many situations even all of the paths tried by ICE do work. ICE will choose the path that works first. This is not necessarily the best path. For instance, ICE may choose a path that goes through a TURN (RELAY) server, even though another path through STUN (SRFLX) could potentially be used.

Troubleshooting a call to the provider

In this chapter, we face the challenge of troubleshooting a failed call to a SIP provider. To find the source of the problem, we will look at the SIP signaling between the SIP interface and the provider.

This time we have created a trace beforehand. Please download  the trace and open it with Wireshark

In this trace, there is a problem where a call made from 850279598 to 850279514 could only be heard in one direction.

The first task is to identify the problematic call. Since we wanted to create a more realistic problem, there is more than one SIP call in the trace and we need to select the call in question.

For this reason, it is always a good idea to let our support know who called whom, because we can use this information to filter the trace accordingly, and we can be sure that we are looking at the right call!

You most likely remember from the last chapter the easiest way to see all calls of a trace.

Please

• open the  Telephony->VoIP Calls feature of Wireshark
• highlight the  the correct SIP call
• and click Flow Sequence

The  call flow overview shows us a SIP call between two endpoints. On the one hand 172.31.31.2 (the customer PBX) and on the other side 46.232.228.36, which is our SIP provider simulation.

So let's look at the exchanged SIP signaling packets as we did for the H.323 call signaling.

• INVITE: The first packet in the call flow is an Invite packet. This signaling message is sent by the provider to initiate a call. It's important to note that this packet contains SDP information, which we'll discuss in the next chapter.

• Status 180 Ringing: The PBX responds with 180 Ringing, indicating that the call request has been accepted and the phone is ringing.

• Status 200 OK: When the call is answered, an OK message is sent. This confirms that everything went well. This message includes SDP information, which specifies where the audio exchange will occur.

• ACK: To complete the call setup, the SIP interface sends an ACK message to indicate that the OK has been received. At this point RTP packets can be sent from both sides.

• BYE: The call ends with a BYE message from the provider simulation followed by an 200 OK. The simple reason for this BYE is that the remote party has hung up. The BYE message terminates the call, so it is always a good idea to start the investigation at the source of the BYE message.

SIP is very similar to HTTP. Therefore, SIP, like HTTP, has two types of packets: Requests and Responses. For example, Invite is a request message. 180 Ringing and 200 OK is the direct response to the Invite request and is therefore called a Response.

Here are some of the more common responses you may encounter when troubleshooting.

• 407 Proxy Authentication Required: This response is sent if the Invite request did not include an Authorization header. Therefore, the client must send another Invite, but this time with an Authorization header. If a second invite is not sent, check if credentials are configured for this SIP account.

• 401 Unauthorized: The authorization was rejected. Check if the invite contains an authorization header. If yes, check if the correct credentials are configured.

• 488 Not acceptable here: The server did not accept a parameter of the SDP request. The most likely problem is that SRTP or T38 is not supported. Please disable these options in the SIP interface.

Going back to our trace, the call flow is not unusual. In fact, it is the expected behavior, which means that the signaling is fine and not the cause of the problem.

SDP

As mentioned earlier, SIP is very similar to HTTP. Like HTTP, SIP also consists of a header and a body. If the SIP message has a body, it contains the SDP protocol. SDP is important because both endpoints exchange information about the media stream with the help of this protocol. As a result both endpoints know where to send the RTP traffic and which codec they have to use.

SDP can also be seen in our Wireshark trace. By  clicking on a packet in the call flow, we can jump directly to a packet.

So let's take a closer look at the INVITE and OK messages.

Although there is much to discuss about it, we will focus on three basic parameters.

IP-Address

The IP address of the RTP endpoint  is part of the Connection Information (c-line) found in the Session Description Protocol part of the Message Body. The other endpoint has to send its RTP traffic to this IP address.

You might be surprise to see a private IP address (172.31.3.2) in the second Connection Information of the first 200 OK message of our trace:

c=IN IP4 172.31.3.2

The external endpoint has no possibility to send IP packets to this (private) address. To fix the issue, in such a case the provider has to send its RTP traffic to the same address from which it receives the RTP packets in the other direction. The private IP is the trigger for enabling this behavior.

When you configure STUN on the SIP interface, the STUN reflexive address (external IP) is used as connection information as long as no symmetric NAT is involved. In the case of symmetric NAT, the NAT workaround described above must be used. The private IP address is used as connection information even though you configured STUN.

If both SIP endpoints exchange ICE candidates, ICE is used to find the best path between the endpoints. ICE candidates are exchanged in a-lines, as you can see in the INVITE of our trace

e.g. a=candidate:1 1 UDP 2130569471 46.232.228.36 16518 typ host

However, SIP providers rarely speak ICE.

You might have seen a c-line like "c=IN IP4 0.0.0.0" in the OK message. Looks a bit surprising as it is an invalid IP address. Also, the port in the m-line above ("m=audio 0 RTP/SAVP 8") is 0. This has to say that the proposed encrypted audio RTP (SAVP) is rejected and the unencrypted audio (AVP) is chosen.

Port

 The port on which the endpoint expects RTP traffic from the remote peer is part of the m-line.

If a STUN Server is configured, the external NAT port is written in the m-line but if the device detects a symmetric NAT the internal port of the device is used.

Codec

The payload types or codes are also part of  the m-line. Instead of the name of the codec, only a number corresponding to the payload type is transmitted. A list of all payload types can be found in this  RFC-3551.

In our scenario trace the codec offer is part of the INVITE request. The offer contains all supported codecs. All codecs of this list may be used for the communication. The answer to this offer contains the selected codecs as you see in our OK message. This codec is then used for the RTP traffic.

You may wonder why each codec is also listed in an a-line. An a-line contains additional information about the codecs, like the sampling rate.

RTP

For the next step, we look at the actual RTP packets between the two peers. After examining the SDP of the exchanged SIP messages, we know which ports are involved in the communication.

Now we can filter our trace for the RTP ports. Please enter udp.port==16522 or udp.port==16518 as filter.

You can  see RTP traffic from the phone (172.31.3.4) to the PBX (172.31.3.2) and from the PBX to the provider simulation (46.232.228.36). Unfortunately, RTP packets from 46.232.228.36 are not received for this connection.

If you encounter such a problem at a customer site, you should first check the customer's firewall or NAT router. It is possible that RTP packets are being discarded. If you still can't see incoming packets from the provider you should open a ticket at the provider.

Audio Quality

Wireshark gives us the possibility to analyze RTP traffic in more detail. Since RTP uses UDP to transmit voice data, packets can be dropped and thus not retransmitted, resulting in poor audio quality.

To identify audio quality problems we  go to Telephony->RTP->RTP streams. All RTP streams for each direction are listed in this overview. You can sort this list by source address, source port, destination address, destination port or payload to find the RTP stream you are looking for.  Open any RTP stream you want to take closer look at and afterwards press Analyze.

RTP traffic is only shown  if the filter of the packet list shows you the RTP packets.

• The status column on the right indicates problems in the RTP stream. To find a packet loss in this list, you can look for the error message Wrong Sequence number. Since each RTP packet has a sequence number, Wireshark detects if one is missing and displays this error message.

• The delta column might be useful for you as well, because the time shown in this column is the time since the last RTP packet was received. In an ideal world, this value should be constant 20ms.

• The jitter column shows us a deviation in the packet rate. Packets are sent in a continuous stream by the sender. Due to a network congestion or slow packet queues, the packet stream may vary and a delay may occur between packets. This delay is displayed as jitter.

To listen to the audio stream you can  click the button Play Streams which will open the RTP Player. Keep in mind when using SRTP, only noise is played because Wireshark cannot decrypt the traffic.

Developers Console

As you probably remember we already talked about the developers console in the  IT Connect training and  the IT Advanced training. In those trainings, we learned about the developer console by investing the login procedure. This time we will focus on more advanced topics.

As a reminder, there are several ways to get the information from the developer console.

• If you opened myApps in the browser, you can simply press F12.

• If you opened myApps in the launcher, you can right-click and select Inspect.

• When you  create a myApps log, you can enable the browser flag to see all messages on the developer console as well.

Of course we could also create a wireshark trace on the PBX to analyze the websocket communication between PBX and myApps client. This can be easily achieved by going to Maintenance / Diagnostics / Command and  enter debug.xml in the command input. A list of advanced trace options will appear. By  enabling the servlets option we can capture the exchanged JSON information of the websocket connection in  readable form.

If you want to read the JSON information in a wireshark trace more conveniently, you can additionally enable the trace options ALL IP4 TCP/UDP traffic and All IP4 TLS Traffic and filter the trace for innovaphone or websocket.  This will reveal websocket packets that contain the JSON information as a string. You can  copy this string as printable text and paste it into a tool like  jsonformatter.org, which will  display the string in a nice and orderly form.

Console Trace - softphone

In this chapter we will discuss how to use the console output to troubleshoot a problem. At first we will take a closer look at the softphone app.

• Make sure you logged into the myApps account of John Doe using https (innovaphone myApps (https://172.31.31.2/PBX0/APPCLIENT/137840/appclient.htm?lang=en))
• Press F12 or choose inspect to open the developers console
•  select the Console tab
•  clear the console so we have a fresh log

Registration

Please open the softphone app of John Doe

At first glance, you might be overwhelmed by the log, but luckily the console has a filter option.

We are going to investigate the PbxSignal Api which is used to transmit call signaling information via the websocket connection of myApps.

Therefore enter PbxSignal in  the filter of the console.

Our log will now contain only a few JSON messages.

 The prefix send in the first JSON messages tells us that the softphone has sent this message to the PBX.  The JSON content indicates that the message type ("mt") is a registration message ("Register") sent via the PbxSignal api ("api") for the hardware ID ("hw") SwPh_john-doe_618a4d00.

In return the softphone  receives a message from the PBX. The message type ("mt") is a RegisterResult received through the PbxSignal Api ("api"). The message contains information like the dialing location, the STUN/TURN server information, the capabilities (video and application sharing) and the endpoint address.

As you can see, the TURN password ("turnPwd") is  transmitted in plain text, so do not use a password that you would use anywhere else for TURN.

The softphone app acts like a normal RTP endpoint, resulting in a  registration in the PBX.

Call Setup

We go back to the myApps client of John Doe and select our video and audio device in the softphone app to be able to make a call. Next, call Jane (12) from John Doe's softphone app.

The call establishment works pretty similar to H.323 signaling:

• A  setup message initiates the call setup
• This message is answered by a setup acknowledge, which indicates the setup has been received
• The alert message tells us that the remote endpoint is ringing. As a consequence a ringbacktone can now be heard by the caller
• As soon as the call is picked up a connect message is been transmitted

The offer in the setup message contains codecs and ICE candidates for both audio and video. The answer from the other endpoint just contains audio parameter, therefore no video is available. This makes sense since the remote endpoint is just a IP112, without a phone app attached to it.

But there is a lot more information you can read from this log. Based on the  call ID, you know whether it is an incoming or outgoing call. An outgoing call has a positive value, while an incoming call has a negative value. Additionally you can see which number is called and the name and number of the connected party.

Console trace - phone app

The phone app is a little bit different than the softphone app because it is no endpoint of an audio call. As a result the phone app uses a different api to communicate with the PBX. It uses the EpSignal Api.

Registration

Please open the phone app of John Doe.

Once the phone app is loaded, you will receive  an error message that no phone device is available, although John Doe's IP111 is registered and you can use it to place calls. In this chapter we have to find out why the connection between phone and phone app fails.

This is the perfect opportunity to

• open the developer console,
  
• clear to console trace and
• start the phone app of John Doe once again
• then  filter the console trace for EpSignal

Instead of sending a Register like the softphone did, we can see that an Attach function is used. This makes sense since the Phone App is no endpoint of a call, it rather attaches it to a phone. To be more precise the phone app tries to attach itself to the Hardware ID 0090334f22f9, which is the MAC address of your IP111.

At the first glance this looks good but the devil is in the detail. The phone app is associated to a specific Hardware ID. The problem is that  John Doe has more than one hardware ID. If we look at the  PBX registrations (PBX / Registrations) we can see that the phone is registered to the wrong hardware ID. This is due to  entering a registration name (alias) on the phone.

To fix the issue we need to make sure the phone registers to the correct hardware ID.

Therefore please delete the registration name and password from Phone / User-1 / General.

As a result the device will use the MAC address as identification.

Call Setup

As soon as the phone is up and running, please

• clear the console again
  
• call John Doe(14) from Jane Doe and
• pick up the call at John Doe's IP111

Interestingly, the log shows us the call more then once. The reason is that the phone app is now  part of the signaling chain. The phone app receives the call from the PBX, reads the necessary information from the setup and forwards it back to the PBX, which then forwards it to the phone. Actually, the phone app adds information to the signaling messages as well. The phone itself cannot know the video codes, the address of the video endpoint or the video resolution. This information is added by the phone app. As a result, the phone app must be part of the signaling chain and cannot be contacted in parallel. Please keep this in mind when troubleshooting a failed call setup.

As an alternative to entering a text, you can also enter a regular expression to filter the console trace specifically to your investigation. For example, if we want to see all the information received from the device 0090334f22f9.  We can filter by typing /0090334f22f9.*recv/ in the filter bar. Be sure to enclose the regular expression in slashes.

Network Tab - chat app

Now let's take a closer look at the network tab of the developers console. The network tab helps us to find and solve network issues since it records all network traffic from and to the your browser.

---

To understand our showcase of the network tab we have to talk about how the chat app works first.

Non-persistent chat

As we learned in the IT Connect training, a peer-to-peer chat is available for free. In this case, chat messages are not stored in the messages database. Chat messages are lost as soon as you close your chat app.

Technically, the myApps client uses the PbxSignal Api  to transmit chat messages. The chat app establishes a connection the same way the softphone app establishes a call, although in this case no RTP channel is opened. The chat app uses a facility message to transmit a chat message to its counterpart.

Persistent chat

In case a premium chat license is used,  chat messages are stored inside the messages database. In order for this to work, the PBX has to know the name of of the Messages-api. Therefore each user object  has a configuration option called Messages. This option is already set by the install, so in most cases you don't need to worry about it, but if chat messages are not stored, this should be the first place to look to see if the user has the correct configuration.

As soon as the PBX receives a chat message through the PbxSignal Api, the PBX forwards the message to the Messages App service through the messages-api. Since both chat partner need to be able to review the stored message, the message has to be stored twice, one time for the sender and one time for the recipient.

If you open your chat app, it will connect to the messages app service through the PbxMessages Api to collect the exchanged messages.

File transfer

If you use  the file transfer function to send someone a file through the chat app, your chat app requests a temporary buffer space in the Messages app service through the PbxMessages Api. The Messages app service allocates this storage space and sends the chat app a link to that exact location within the Messages app service.

The chat app uses this link and uploads the file to this location via an HTTP POST. The recipient receives this link through his chat app, which allows them to download the file from the messages app service.

The file is then placed in permanent storage so that the file can be downloaded once the temporary buffer is freed.

---

Let's go back to practice:

• open the chat app of John Doe
• open the developers console
• clear the console messages
• use /^chat:/ as filter
  
• send a chat message with a PDF file attached to your own user (ckl). If you need a PDF file, you can use  this one

Now let's find out if the above theory is correct.

As we see in the log the chat app is allocating space on the messages app service (using a CreateFileBuffer message). The messages app service responds with  a success message (CreateFileBufferResult) and a link to that buffer space.

Let's now switch to the Network tab.

We can see that  two requests have been sent.

We can ignore the first packet which is an OPTIONS message which is used to find out the capabilities of the server. The second packet is the  HTTPS POST message to transmit the PDF to  the destination received through the websocket connection.

Furthermore we can see the  destination address and port of this packet. Additionally  the response code received from the server can be seen. Fortunately for us, 200 OK means that the data transfer was successful.

If the transfer would have been not successful, you can see an  error message. In this particular case shown in the screenshot, we can conclude the problem from the error message. The certificate was not trusted and therefore the HTTPS request from the browser was declined by the AP. Do not forget that the app service of the chat is part of the firmware of the PBX, but the exchanged files are buffered on the AP, which means your browser has to trust both certificates, the certificate of the PBX and the certificate of the AP.

App Service Logs

In the previous chapters, we talked about analyzing myApps problems with the developer's console. As you learned in the IT Advanced training, a lot of the apps work in conjunction with their corresponding App Service instances on the Application Platform. As a result it could be necessary to troubleshoot the App Service instance as well. Luckily we can do that in our  AP Manager app.

Before we start please make sure you are logged into myApps with john.doe and password ip411

Let's first recall the AP Manager app. When you  click on one of the app services, you will see all instances of the app service, the name of the app service, the version number, whether it is started or not, and the uptime.

Additionally you can see  a line concerning the log file.

• Display: Allows you to see the log of the app service in a new tab of the browser.
• Save: You can save the log file on your hard disk.
• Delete: Deletes the log file, backtraces and core dump of the app service.
• Core dump: The core dump gives us information of an unexpected termination of the app service.
• Backtraces: A backtrace contains a list of functions leading to the crash of an app service. In addition to that, backtraces include the logs of the app service and the webserver. Up to four backtrace files are saved and can be downloaded through the links (0) (1) (2) (3).

Before we can start analyzing a problem in the log file, we have to activate trace options, otherwise the log will contain very little information. It's basically useless. Trace options can be enabled per App service or by App service instance, so you need to make sure you click on the instance you want to log before you  select Diagnostics in the top menu.

As a practical exercise we want to the test if the SMTP connection to the training mail server works properly. We will test this by sending a fax from John Doe to your own account (ckl) and the fax to mail feature will deliver the received fax as mail. Luckily the  SMTP configuration and  the Fax App configuration in our Fax App PBX Manager Plugin is already in place.

You just need to save the Fax App configuration once so that the connection between app service and app object is established. This can be verified by the green check mark in the Fax PBX Manager plugin.

Afterwards, please  select DNS, TLS, TCP and SMTP as trace options for the Fax app service (AP Manager) so we can see something useful in the log.

To receive the fax as email, a user has to enable email notification in the burger menu of his or her personal fax app.

Please check the  Forwarding as PDF checkbox in ckl's fax app (you can either log in as ckl in your current myApps tab and then back to john.doe or use another browser). There are various options available. All of them will eventually send you an email, but the fax attachment will only be sent if you enable the option Forwarding as PDF.

Now open the fax app of John Doe and and send a fax to your own user by entering 810 as destination. 8 is the fax prefix and 10 is your extension number. If you need a PDF file you can use  this one.

To see if you received the email, please log in to your Outlook Web Access account.

•  training-mail.innovaphone.com/owa
• Username: dvl-ckl2@class.local
• Password: ckl-pw

Unfortunately, you did not receive an email. However, this is a perfect opportunity to investigate the problem in the Fax App service log.

• First of all, we can see that the DNS resolution for  training-mail.innovaphone.com worked and the result is 46.232.228.40.
• After that we can see that  a connection to this IP is established on port 587.
• Since SMTP is a very polite protocol, the server and the client introduce themselves to each other and the server transmits its capabilities.
• Next, we can see that the SMTP client  tries to transmits the FROM and TO address of the email.
• At this point we can see the cause of our problem. The mail server responds with 501 - Invalid address.
• The FROM address does not exist. If you inspect closer, this is no email address at all as it has no @ symbol.
• The FROM address should be dvl-ckl2@class.local not dvl-ckl2.class.local

Please change the Sender address in the Fax PBX Manager configuration to dvl-ckl2@class.local and send the fax again. This time you will receive an E-mail and the log will show you that DATA is being transmitted.

There is also the possibility to set the logging options for the app service itself. Simply select the app service on the left panel, and then click the Diagnostics button again. Since you didn't select an instance this time, you get a different view with only three options.

• Enable logging: This is a rarely used option for tracking traffic between the AP manager and the app service.
• Disable hexdumps: This option is also rarely used. To reduce the amount of data written to the log file, hex dumps are excluded from the file.
• Logfile size: Here you can set the size of the log file. To capture a sporadic problem, it may be useful to increase the size of the log file.

Catch Trace on Event

Sometimes you can see a recurring event entry but the given information is not enough to resolve the issue. A prominent example is when  the reverse proxy has blacklisted an IP. To see the packet that led to this blacklisting you can use the Catch Trace on Event feature.

Just go to Services / Logging and  paste the Code of the event in the Catch trace on Event configuration option. A hexadecimal code will be converted and stored as a decimal value.

As soon as the event is generated again, the trace buffer stops writing. This means you can  empty the trace buffer on Maintenance / Diagnostics / Tracing and look at the packet that caused the error.