Course11:Advanced - Operations

Explains how to setup an innovaphone PBX system so that it can be maintained smoothly.

Monitoring and Debugging

Sometimes some things do not work or not just as expected. To make sure your installations run smoothly, you need to

• make sure you notice problems before your end users do
• gather as much information to understand the nature and cause of any problems

The innovaphone devices provides several integrated tools to help solving these tasks.

Events

An Event as shown on the  Maintenance/Diagnostics/Events page of any innovaphone devices is a notification of an incidence that should receive the administrators attention.

There are a  number of events defined.
Some of them indicate bad user behaviour such as  0x00020003 which indicates an external call transfer while the system configuration does not allow it.
Others may indicate a misconfiguration of the system such as  0x00010002 which brings a failing registration to your attention.
Yet others will likely report hardware problems such as  0x00090001 which notifies a damaged compact flash card.
Also, temporary performance problems in your network could create an event, such as  0x00050002, indicating loss of data while transmitting RTP media data.

In any case, events shown in the event log must be examined carefully and it needs to be determined if they indicate a problem that need to be fixed.
Clicking on the event-code listed in the Code column will open a popup window with some useful context information. For example, for the aforementioned type 0x00010002,

Protocol	SIP
AOR	<sip:a@x>
Comment	Timeout

will be shown.

Alarms

An Alarm as shown on the  Maintenance/Diagnostics/Alarms page of any innovaphone device is a notification of a persisting problem that needs to be corrected by the administrator. Alarms are always triggered by a corresponding event. Also, when the error condition goes away, the alarm is removed, but the event list will include a new event reporting the clearance of the alarm.

In a managed installation, there should be no entries in the alarm list at any time!

The event and alarm list is always kept in local flash memory. The number of entries to keep can be configured in the  Services/Logging tab. You should rather not configure values that deviate from the defaults. The reason for this is that flash memory available to store events is rather limited. If you need to configure larger values, make sure you check flash memory availability in the VARS flash segment (use mod cmd FLASHMAN0 info and look at the VARS line).

All alarms are sent via SNMP traps. The SNMP trap delivers: code,severity,txt (e.g interface down) and alarm–source (e.g IP0/ETH1). This can be useful if the customers has a SNMP network management tool in use.

Syslog

The syslog (found on the  Maintenance/Diagnostics/Logging tab) is a first point you should look at if you have a problem. Syslog provides you real-time information about things like:

• PBX/Gateway Calls - calls and their flow in the PBX or Gateway
  
• Gateway Routing - number mapping and call routing in Relay
  
• H.323/SIP-Registrations - endpoint/interface registration successes or fails
  
• Administration - configuration changes performed on the device
• and more

So when you are able to reproduce the problem, you should turn on the relevant (and only those) check marks on the  logging tab, repro the problem and examine the syslog.

Support will always ask you for a log demonstrating the problem. To reduce the number of turnarounds required for them to understand the problem, make sure your log includes the maximum of relevant information and the minimum of irrelevant information. To do so

• tick on all logging check marks that relate to information relevant to the problem
• tick off any logging check mark that creates irrelevant information
• try to trim down the log so that it fully captures the reproduced problem but nothing else
• clearly indicate the situation captured in the syslog (not: there is a problem, see syslog. Rather: in the syslog, there is a call coming from TEL1, calling party is xyz, called party is abc. It is sent to extension -xx and terminates with cause code 21. Expected behaviour though is for the call to alert at the phone registered with user efg)
  

Capture the sylog as .txt file. Do not put it into a MS Word or MS Excel file. Do not send a screen shot of the syslog screen. You can easily save the syslog as a text file by right-clicking on the frame and saving the source.

Traces

A Trace (found in the  Maintenance/Diagnostics/Tracing) tab is a kind of advanced syslog, that provides significantly more information for debugging purposes. It is usually needed if the information in syslog is not enough.

As opposed to syslog, traces are permanently saved into an in-core ring buffer. This implies that

• you can obtain information about events that happened before you obtain the trace (provided you obtain it before they get overridden in the ring buffer)
• when you reproduce a problem and then obtain the trace, the trace will include information prior to your reproduction of the problem (which is probably not useful)
• tracing always consumes CPU resources, not only while obtaining a trace

The trace buffer is cleared whenever it has been retrieved. In order to capture a problem with a trace, best practice is

• obtain the trace, thereby clearing the trace buffer
• reproduce the problem
• obtain the trace again

The trace ring buffer is of limited size. So you should make sure that you uncheck all irrelevant check marks in the  tracing tab. Also, when you are done with trouble shooting, it is a good idea to turn all tracing off.

Writing to the trace ring buffer is disabled when the device traps. Also, the trace ring buffer is not cleared on a re-start. Thus, after a re-start, you can obtain the trace from before the re-start, showing the trap situation!

Trace is also a simple text based tool and the notes made about obtaining and providing it to support made for the syslog do apply here too.

Wireshark

One of the disadvantages of the trace mechanism is that it can be difficult or impossible to capture a reproed problem in a life installation.

From V6 SR2, the trace mechanism was greatly enhanced by the  PCAP trace based on  Wireshark. This allows any innovaphone device to work as remote capture driver for Wireshark. This works much like the rpcapd on a windows or linux system.

When the Enable RPCAP check mark is ticked in the  tracing tab, the device will allow wireshark to connect and obtain packet captures. Wireshark will do so if you specify something like rpcap://172.16.15.4/trace as Capture Interface.

Usually, you will also tick All TCP/UDP Traffic in the tracing tab, so that Wireshark will capture all network traffic from and to the device. However, wireshark will also capture all other trace information too. Wireshark's advanced filtering and analysis capabilities allow to capture even huge traces in a life system and drill down the information so that the problem reproed can be isolated from all the irrelevant information. To begin with, try to capture a normal call, then look at Telephony / Voip Calls.

Tracing encrypted Signalling

Nowadays, most installations run encrypted signalling and media (SRTP).

In such installations, you can also get traces for trouble shooting (either with text trace or wireshark) and send them to us for analysis. However, as they are encrypted, we cannot read them! (no, there is no manufacturer back-door)

While we usually have no intention to listen to your talks (so SRTP is not a problem), we usually do want to analyze you call signalling. As a result, traces from encrypted communication are useless to us. To circumvent this problem, you must tick the All IPv4 TLS Traffic (or All IPv6 TLS Traffic) check-mark in the tracing page when obtaining traces. This will enclose an uncrypted copy of the trace signalling data.

 Encrypted Tracing



If you want to see the encrypted signalling messages in Wireshark, you can mark a specific packet and apply the  Wireshark-Option: Analyze ? Decode As….-> Do not decode on this packet/ports. To find the encrypted message, you should filter for the  used port - e.g. for H323/TLS it would be TCP port 1300(or as wireshark filter tcp.port==1300)

Storing and Consolidating Syslog and Events

While alarms, events, syslog and traces are great tools to trouble shoot an installation, administrators often have a big problem: when they are made aware of a problem (either by users or events), they are often unable to reproduce the problem. It is thus clearly desirable, to capture and save syslogs permanently for later perusal.

Also, in a distributed system such as a VoIP PBX, it is often difficult to determine where the problem actually would be logged. For this reason, it is clearly desirable to consolidate all events, alarms and sylogs to a single place. You can do this by relaying this type of information to a single web server for storage. This will usually be one of your innovaphone devices that has a CF card installed.

Relaying is configured in the  Services/Logging tab. To setup a central storage, you would configure Log Server Type LOCAL on the central device. On all other devices, you would configure Log Server Type HTTP with Method Standard and specify the central device's IP address. On the central device, you would leave Alarm and Event Forward Server empty. On all other devices you would configure the Method Standard, again using the central device's IP address.

This way, all events, alarms and syslog entries are permanently sent to the central device. All events and alarms will show up in the  Maintenance/Diagnostics/Events tab of this device and all syslog information will be stored on the central devices CF card in the /log directory (e.g. \\x.x.x.x\drive\CF0\log) in files LOG0.0 to LOG0.3.

Remote events and alarms are not stored in the Flash memory of the receiving device (i.e. your central event/alarm server). If the central event server is restarted, all remote events are lost. Alarms are retransmitted by the generating device(2 minutes after reboot and every 30 minutes), updating the alarms list on the central device. If you have to reboot the central event/alarm server, make sure to use the Save option to store the Event-list.

Using DHCP for basic Configuration

Some configuration options may be required to set up a working VoIP PBX system. Some of them have to be configured on the devices itself (as opposed to the PBX for example). Taking into account that such a system can easily include hundreds of devices (e.g. phones), configuring such options manually (either by phone menu or web interface) is often not an option.

innovaphone devices thus feature a number of configuration options that can be  deployed via DHCP. This includes, amongst others, things like the user interface language for phones, the local time zone and clock type, the dial tone type and much more. This mechanism allows to set many options without touching the devices. This not only saves a lot of deployment time, but makes later replacements easier.

Generally, when a devices receives an option via DHCP, it overwrites the configuration found in the device itself. This makes it possible to overwrite default options. The options received by DHCP are shown in each devices Current Lease area in the  IP4/ETH/DHCP tab.

Which DHCP Server to use?

Most configuration options are sent to the DHCP client as DHCP vendor options. These are vendor specific options that need to be made known to the DHCP server before it can handle them. All relevant details on  how to configure 3rd party DHCP servers such as on Windows or Linux can be found in the wiki.

Much easier however is to use innovaphone's  built-in DHCP server! This is a fully fledged DHCP server that can replace the standard server. It already knows about all innovaphone vendor options and is thus easily configured. Also, as opposed to most other DHCP server implementations, it supports the dynamic update of configuration options to clients that have already obtained a lease. This is done by clicking the Renew button in the  IP4/ETH/DHCP-Server tab.

Running a "private" DHCP Server

In some scenarios, it is not an option to use the innovaphone DHCP server as a replacement for an existing DHCP server. Also, sometimes there is no DHCP server available at all and the local network policy does not allow a DHCP server to offer IP leases in the network.

In such cases, the innovaphone DHCP server can be used to serve innovaphone clients only. This is enforced by setting the Reserved and same Vendor Clients only check mark on the  IP4/ETH/DHCP tab of the device running the DHCP server. As innovaphone clients (e.g. phone) will by default wait up to 8 seconds for an innovaphone DHCP server lease offer (even if they meanwhile receive another one), they usually do not need to be configured specifically for this to work.

Please note that it is usually not required to set the Server Identifier or Selected Server only field.

Starting with version 10, it is possible to use the option  DHCP Custom in order make the DHCP-server distribute vendor specific information.
The vendor class identifier and the vendor options can be entered as character strings or in hex representation.

How to find correct option values

Most innovaphone vendor specific DHCP options are actually used by the phones only. To deploy these option you need to find out how to translate settings you would normally do using the web user interface into DHCP option values.

To help with this, you can have a look at the  Phone/State/DHCP-Options page in the phone. it will show the relevant DHCP option values you would need to supply if the phone would be configured via DHCP.

To find out the right settings, you would thus

• turn off DHCP client mode in the phone (to make sure there are no settings received with DHCP)
• configure the phone according to your needs using the phone menus or the web user interface
• copy the DHCP option values from  Phone/State/DHCP-Optionsto your DHCP option settings

To verify your settings, reset your phone to factory defaults and it should behave correctly.