Howto:How to implement large PBXs

From innovaphone-wiki

Jump to: navigation, search

innovaphone PBX installations scale from very few users (such as with an IP202) to several thousand users. Here is how to design and configure innovaphone devices to support many users.


Contents

Applies To

This information applies to

  • All innovaphone PBX platforms

build 05-5842 and later.


More Information

Technical issues to be considered

There are a few issues to be considered when configuring a large PBX:

  • CPU power

The platform must be able to cope with all processing requirements to serve the user base. The basic requirement is to support the number of calls created. The ability of a given platform can be measured as its “business hour call attempts” capability, a figure which essentially indicates the number of calls per hour the platform can handle.

Naturally, not only calls take CPU cycles. Major scenarios to consider here include user registration, replication and SOAP/CTI applications (such as TAPI)

  • Main Memory

The platform must feature enough main memory to serve the user and call scenarios. Rough “memory/feature” figures will be given.

  • Flash Memory

There must be enough persistent memory to store the system configuration. Although the amount of memory used largely depends on the details of the configuration, a rule-of-thumb can be given for the “memory/user” ratio.

  • Configuration

The configuration must be appropriate to the intended usage scenario. Guidelines are given for various scenarios.

  • Number of channels

Naturally, the number of gateway channels (e.g. to the ISDN or to the DECT network) must be appropriate. The number is a function of the so-called “erlang” value. This issue will not be discussed here.


The information presented in this article is based on practical verification by means of a sample configuration. The setup is made up of 6000 user entries, 10 location entries, 500 speed dial entries and a number of other entries such as trunks etc.

The configuration is attached for your reference.


The standard Testsetup User
The standard Testsetup User
The standard Testsetup User (Details)
The standard Testsetup User (Details)

All measurements given in the remainder of this article apply to the IP800. Since the IP800 both in terms of processor architecture and processing power is comparable to the IP3000, the results apply to the IP3000 too.


Theoretical up to 10000 User PBX can be implemented with multiple IP6000:

Image:How to implement large PBXs03.jpg

Likewise, up to 15.000 users can be set up with multiple IPxx10.

Technical data and recommended number of users supported

The following table shows technical data and differences between various hardware devices. It also includes figures for the maximum number of “users” supported by the various innovaphone PBX platforms under certain conditions (i.e. when running the box solely as an LDAP master). These figures are intended as a rough estimate “rule of thumb” guideline.

Please note that these are respective maximum figures. That is, more than this will not work. However, each role used in a box (e.g. LDAP db, Registration PBX, Media gateway, ...) consumes resources and the figures below are estimates of the expected possible maximum. If you run multiple roles on a single device, you may run out of resources before.

Product CPU (Mhz) RAM(GB) FLASH (MB) NAND (GB) LAP RAM for LAP (GB) out of RAM DSP CONF Fax Ch Audio Fax Ch out of Fax Ch G711 G723 G729 G722 OPUS ETH loop in PRI BRI BRI inline power FXO FXS SSD CF myPBX clients[1] PBX User (all-in-one)[2] Voice Media-Relay[3]
IP3114001320,128x0,7686-10xxx-x2*1000----42x-755075
IP4114001320,128x0,7686-11xxx-x2*1000--2--2x-755075
IP8118002321x1,536101022xxx-x2*1000x-5---x-750200120
IP30118002321x1,536303015[4]15xxx-x2*1000-1----x-750500120
IP00118002321x1,5360-15[4]0-----2*1000------x-750500120
IP11308000,2532---303015[4]15xxx-x2*1000-1--------120
IP294000,2532---8---xxx-x1*100-----8----n.a.
IP281250,03216---8--xxx--1*100-----8----n.a.
IP381250,03216---8-10x-x--1*100----8---1010n.a.
IP3021250,0328---4-10xxx--2*100--1--2-x1010n.a.
IP3051250,0328---4410xxx--2*100--2----x1010n.a.
IP8001250,0328---101020xxx--2*100x-5x---x4040n.a.
IP8108000,532-x0,38420201010xxx--2*100x-5----x200200105
IP00108000,532-x0,25660601010xxx--2*100-------x650500105
IP30108000,532-x0,25642421010xxx--2*100-11----x650500105
IP60108000,532-x0,25660601010xxx--2*100x41----x650500105
IP10608000,516-x-60606030x----2*100-21----x--105
IP221250,03216---2---xxx--1*100-----2----n.a.
IP241250,03216---4---xxx--1*100-----4----n.a.
IP60005330,12816---60601010xxx--2*100x41----x500n.a.n.a.
IP3000133n.a.n.a.---30---xxx--2*100-21----xn.a.n.a.n.a.
IP20005330,12816---20000xxx--2*100-11----500n.a.n.a.
IPVA0-x[5]0-----x[6]------virtualized-x[5]x[5]x[7]

NB: All scenarios involving encryption assume that a standard 1024-bit certificate is used (see Reference7:Certificate_management#Certificate_Key_Length_and_CPU_Usage).

  1. This (Recommended maximum number of myPBX clients when using full encryption) refers to a setup, where all communication (including signalling) is encrypted.
  2. The PBX User (all-in-one) refers to a setup including all functionality such as UC.
  3. Maximum number of parallel G.711 20ms calls with Media-Relay (e.g. when run as SBC) with one end encrypted and one unencrypted, at 100% CPU load.
  4. 4.0 4.1 4.2 from 12r1 Service Release 3, was 7 before
  5. 5.0 5.1 5.2 Depends on the used PC hardware in terms of available CPU power and memory
  6. Depends on the amount of NICs used PC hardware offers
  7. Media-Relay performance of IPVA depends on the platform used quite a bit. Generally, encryption/decryption is better compared to an IPxx11, network performance is not as good. We have measured a performance of 120 concurrent calls on a single IPVA.

For a calculation of IPVA resources, see Howto:A rough estimate of IPVA Performance


The recommended figures are pretty conservative, so as to make sure that a configuration obeying the figures will work well under normal circumstances. However, in certain usage situations, such as use of broadcast groups or waiting queues, missing CPU cycles could result in slow service.

All figures are related to a single platform (that is, innovaphone box). Larger installations can be designed using several boxes running related PBXs. See below for details.

From version 8 upwards, the maximum amount of flash memory allocatable for the directory (FLASHDIR0) is limited to a platform specific reasonable value by default. It can be raised using the

mod cmd FLASHDIR0 /seg n

command, where n is the number of allowed 64kB segments.

Figures for the IPVA platform obviously depend on the machine used. See Howto:A rough estimate of IPVA Performance for an estimate.

Please note that these figures do not apply to applications on the Linux Application Platform. More specifically, the figures in the Recommended maximum number of myPBX clients column do not suggest that a LAP running on such a device can serve the same number of users with all applications such as fax, calendar replication, call lists, reporting. It is extremely difficult to estimate the number of users that can be served on a LAP of a given platform, as it again heavily depends on the applications and usage patterns. However, as a rule of thumb, a LAP running on an xx10 platform should be able to support 150 users with average usage patterns. When more users have to be supported, we strongly recommend to consider a LAP running on VMware. For comparison to a VMware platform, keep in mind that the xx10's ARM processor is a RISC platform, whereas VMware runs on a CISC platform. You should plan for more RAM on a VMWare platform thus. This might lead to the following calculation: if you look at supporting 300 users on a LAP, use VMWare and dedicate at least 1.6GHz CPU and 1 GB RAM (please note that the RAM column in the table above shows the total RAM available in the system which is shared by the PBX and the Linux-OS). The total RAM available for the LAP is shown on the LAP's Diagnostics/Status page).

Load Calculator

We have a nice online tool available to partners which allows you to calculate the expected load given a specific usage scenario.

Flash Memory

Flash memory is mainly used to store object definitions in the LDAP database. So the number of defined objects is calculated as a function of the flash memory used to define the objects.

The actual space used by an object definition heavily depends on the number and size of attributes set for the object since the objects are stored “packed” in flash memory. For example, users with long names will consume more space than those with short names. Also, users with a defined call forwarding will consume more space. Because of this, in a given scenario, the number of objects that can be stored in the platforms available flash memory can vary considerably. However, since in normal operation, object definitions are changed frequently, you need to calculate a rather large amount of memory per entry. We recommend to plan for 512 bytes per object entry. This should be sufficient even for unusual complicated configurations. In a given installation, it is well possible that a platform supports significantly more defined objects than indicated in the table above.

Please note that the number of defined objects is not identical to the number of users supported. This is because there are usually a lot of non-user objects defined. As an example, speed dial entries will count into the total number of defined objects. The actual ratio of supported users to defined objects obviously is depending on the installation at hand. Still, as a rule of thumb, you should calculate at least 33% more defined objects than supported users. That is, if you intend to support 1000 users, you may want to calculate for 1400 defined objects.

Also, you need to be aware that the LDAP database needs to keep track of deleted items for replication purposes. Such items are kept for a while in flash memory (although they are not seen in the user interface any more). The amount of memory required to store such items obviously depends on the deletion rate of objects. Especially when using automated tools to delete and create a large number of objects, you may create a lot of “deleted” items which consume flash space. For this reason, when you create a new object database in a PBX which already had a significant number of objects before, make sure first to do a “clear PBX config”. This will physically clear the LDAP database whereas any other method (including ad-hoc replication from a master PBX database) will create a deleted object for each previously existing object.

You can query the current flash memory usage with

mod cmd FLASHMAN0 info

For more information about flash directory sizing, see Concept Flash Directory.

Main Memory

Main memory is obviously required for any kind of operation in the PBX. However, there are some prominent situations which require sufficiant memory. These include amongst others

  • System boot

Depending on the firmware size (the firmware is store compressed in flash and thus will be uncompressed into main memory at boot time) and the basic configuration, there is an initial main memory requirement of slightly less than 3MB currently.

During boot, all objects are retrieved from the LDAP database and shadowed in main memory. This will take significantly more space than it consumes in flash memory. The exact amount again depends on the specific object configuration. As a rule of thumb, you can expect 2.7kB per object (based on the same configuration complexity assumed above for flash memory usage). However, during boot time, there is an additional need for main memory of slightly less than twice the flash memory used. We recommend to calculate for approximately 3.5kB per user main memory thus.

  • Endpoint Registration

When an endpoint is registered with the Gatekeeper (PBX), it will consume an additional 1.6kB main memory.

  • CDR delivery

CDRs to be delivered via HTTP are buffered if delivery cannot take place immediately. The buffer space reserved is 300kB. Since there are 2 CDR interfaces and the LOG interface is treated identically, 900kB must be calculated for buffering.

  • Call signaling

Concurrent call signaling will consume further main memory. The exact amount of memory obviously vastly depends on the exact nature of the calls (e.g. internal calls, calls to local interfaces, broadcast calls, etc.). We have measured a test setup where 100 registered clients constantly setup and tear down calls so that there is an average of 15 calls at a time. All calls were monitored by a SOAP application (such as TAPI).

This creates a memory load of an additional 700kB.

Please note that this is a heavy duty load scenario since long lasting calls will consume much less memory load.

You can query the current memory usage with

mem

CPU

The overall performance of a PBX can be measured and given as busy hour call attempts (BHCA) figure. We have measured 2 scenarios:

  • Pure gatekeeper mode

This is the same scenario as in the “Main memory / Call Signaling” section above. 100 clients are registered with the PBX and constantly setup and tear down calls so that there is an average of 15 calls at a time. Both ends of all calls terminate in an IP endpoint, so external call signaling issues such as those induced by an ISDN interface are not considered. No SOAP monitoring.

The IP800 performed with 4.57 calls per second with a CPU load of 30%. This amounts to 15 calls per second with a 100% CPU load and thus approximately 50.000 BHCA.

  • Combined gatekeeper and gateway mode

Same scenario as above except that additionally all ISDN interfaces are busy with calls from IP to ISDN with a 60ms packet size. This base load creates 6% CPU load.

The IP800 performed with 45.000 BHCA in this scenario.

The IP3000, having approximately the same CPU power, has more ISDN channels to drive when in gateway mode (30 instead of 8). The respective figures for the IP3000 do differ thus. The base load (all 30 channels active with 60ms packets) is 21%. The IP3000 performs with little better than 40.000 BHCA in this scenario.

For both platforms, the base load increases significantly if the packet size is reduced. For example, the IP3000’s base load increases to 42% when 30 20ms channels are active. It is thus recommended to avoid small packet sizes.

An IP800 with 6500 defined objects will take about 20 seconds to boot.

You can query the current CPU load with

mod

To reset the CPU tick counter, use

mod clr

LDAP performance

LDAP is used both for replication of PBX and DECT gateway configurations and for name and number resolution in the IP200.

Using current firmware, there are no performance issues with the LDAP replication.

LDAP name and number resolution performance largely depends on the number of entries defined in the PBX/LDAP database. Based on a setup with 1000 user and 500 speed dial entries configured, the IP800 performs with a maximum of 16 LDAP connect/query/result/disconnect cycles per second.

Assumed that there is one such cycle per call, this will effectively cut down the BHCA to 50%.

It is recommended to use an external LDAP server such as MetaDirectory for large (> 1000 users) PBXs thus.

There is no "hard coded" limitation of simultaneous LDAP connections for the LDAP Server running on Innovaphone devices, so the limitation will be the CPU handling of the ammount of simultaneous connections.

Small PBX

For smaller PBXs (1000 users and less), a single IP800 or IP3000 can serve as a platform. If redundancy is desired, one platform can be used as master PBX, the other can be used as gateway and standby PBX (so as to distribute gateway and gatekeeper load, especially for the IP3000).

Medium PBX

If the PBX has to support more users, then we recommend to use one IP800/IP3000 per 1000 PBX users. If a uniform numbering scheme is desired and all users shall be maintained in a single PBX, we recommend to add an IP800 which will then hold the complete PBX/LDAP database. This will be the “master PBX” and all other platforms will be defined as “slave location” and register with the master PBX. Their configuration will be replicated from the master PBX, restricted to their specific location data. This is because both hosting the complete configuration and serving a large number of registrations at the same time will decrease performance. The actual endpoint registrations will be only on the slave locations so that each platform needs to handle no more than 1000 registrations at a time.

Each endpoint will register with its defined location PBX. You can freely assign endpoints to locations at will, provided the number of endpoints per location does not exceed 1000.

The master PBX should be used as standby PBX for all locations, so that if a single location PBX fails, the master PBX will take over the 1000 registrations.

While the endpoints register with their location PBX, the IP200s LDAP connection should be configured such that the central master location PBX is used as LDAP server. This is to make sure all IP200 will see the complete phone book. As discussed above, you may want to consider an external LDAP server in which case this should be configured as LDAP connection.

Up to 6000 users can be supported this way.

Large PBX

If a uniform numbering plan and centralized administration is not an issue, large PBXs can be design virtually without limits.

A tree of PBX locations would be configured, each of them storing its own endpoints and/or sub-locations. Each single location can support up to 1000 endpoints.

Since there is no single instance that knows all of the endpoints, the locations must be defined using the “prefix” mode so that calls can be routed according to the called number by traversing the location tree towards the leafs.

All endpoints must register with the correct PBX as defined by the extension prefix-chain down from the root/master location PBX.

Replication

As noted above, when replicating a larger location PBX from a master PBX, the location PBX’s replication settings must restrict the data to be replicated to their own location.

Especially when a location I using a smaller platform such as an IP202 or IP400, this configuration is critical in order not to overflow the locations LDAP capacity.

The same applies to the DECT gateways IP600 and IP1500. They will normally replicate all users, not just the DECT users. However, usually not all the location users are DECT users too and the number of location users may not fit into the number LDAP database of the DECT gateway. You can restrict the objects replicated to the real DECT users using a special configuration of the DECT gateways LDAP replication client. Assumed the DECT users are configured such that the “DECT-GW” is set to “dect” in the PBXs user entries. You would then configure the “Location” in the “LDAP Replication” settings of the DECT gateway to

(pbx=<gw *name=”dect”*>)

This configuration is taking advantage of the fact that a location search attribute starting with a ‘(‘ is considered a complete LDAP search expression. The “DECT Gateway” section of a PBXs user configuration is stored in an LDAP attribute such as

pbx=<gw name="dect" dsp="user" ipei="12345"/>


Known Problems

PBX Groups are PBX-local objects. Users registered on different PBXs/locations belonging to groups with the same name are this effectively members of different groups. The other way round, in a single group there cannot be members registered with different PBXs/locations. If you need to add a PBX user to a remote PBX, you will need to create a co-registration on that PBX and add this registration user to the group.

Since V11r2 see also Reference11r2:Concept_Group_Pickup_across_PBXs


The TAPI service provider currently maintains a single connection to a single PBX. It will thus not be able to monitor users on more than one PBX/location. (ckl 12:22, 11 January 2011 (CET) this changed with TAPI V7)

Image:Tools clipart.png FIXME: "This article urgently needs a V6,7,8,9,10 rewrite"

Relevance for V6 Firmware and later

Most of the conclusions presented in this article still hold true for V6 and V7 firmware. You should just calculate slightly less available memory for the devices as the firmware grows and thus leaves less memory for operation. Also, when looking at the IP2000/6000 you need to know that it has 4 times more memory than the IP800 and is 4 times faster than the IP800.

Relevance for V7 and later

A IP2000/IP6000 can handle up to 10.000 Objects.

Group limitation is up to 2048 Objects in a group.

IP30x is slightly faster than an IP800, but has considerably less memory. Watch out, the relevant memory (DRAM) space is after system startup, not what is physically available. Especially for the IP30x, a lot of memory is consumed for the DMA interface to the DSP processor. Here is a sample out !mem output directly after a IP302 system startup:

physcial memory:	  16777216
allocated:      	    561916
total malloc space:	   4590600
free for malloc:	   3019008
Total memory usage:	   1504584
Memory Load:    	        32%

Allocatable memory is only 2948kB (3019008)!

config change LDAPSRV0 /max_session n is not supported any more (nor is it necessary).

Relevance for V8 and later

mod cmd FLASHDIR0 /max nitems is not supported anymore (nor is it needed).

Related Articles

Personal tools