Reference13r3:Concept App Platform

From innovaphone wiki
Jump to navigation Jump to search
There are also other versions of this article available: Reference13r1 | Reference13r2 | Reference13r3 (this version) | Reference14r1 | Reference14r2

General

Requirements

  • V13 or up
  • Gateway (arm): IPx10 (with CF card) or IPx11 (with mSATA SSD)
  • Gateway (arm64): IPx13 (with m2 SSD)
  • Virtual (x86_64)

Default credentials

During INSTALL, the default passwords are replaced with the global Admin PW!

  • SSH-Login with admin and ipapps
  • root login with root and iplinux (the root login is not directly possible, you have to login as admin first and use the command su root)
  • manager App (web login) pwd

App Platform - arm/arm64 (Gateway)

  • The installation image has a size of ~50MB. During installation, the following partitions are created[1]:
    • /dev/sda1 fat32: 200MB (contains ramdisk, rootfs and kernel)
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB

When comparing potential performance of the IPxx11 platform compared to the IPxx10 platform, there are some major differences:

  • SSDs found in the xx11 are faster and more reliable than the CF found in the xx10
  • the available RAM for the App Platform (as specified in column RAM for LAP (GB) out of RAM in chapter Technical data and recommended number of users supported of How to implement large PBXs) is factor 6 larger on the xx11 (1,536 GB) than on the xx10 (0,256 GB))
  • the xx11 has gigabit Ethernet while the xx10 has 100Mbps Ethernet. The xx10 is therefore not well suited for Apps with larger network traffic, such as Recordings
  • the CPU of the xx11 (although it runs on the same frequency) is roughly 20% faster than the xx10

While it is hard to predict the performance of the App Platform in a specific scenario, we see that in a real life environment an App Platform running on an xx11 platform can well support 150 users. The xx10 platform is estimated to support 120 users. Because CPU performance is the limiting factor, larger setups can be built based on the virtual machine platform (see App services and multi-threading below).

  1. Maximum partition sizes are given. The real sizes are returned by the linux 'df -h'-command. These real values will be lower, but can grow in the future with new releases.

App Platform - x86-64 (Virtual Machine 64bit)

  • The default disk size is 16GB. It should be increased before the first start if needed!
  • Multiple CPUs are supported, default is one CPU
  • default RAM: 512MB
  • static IP address, DNS, Gateway can be configured with the command setip on the console. Run setip --help to get a list of parameters. (Example: setip --addr=x.x.x.x --mask=x.x.x.x --gateway=x.x.x.x --dns1=x.x.x.x)
  • If you have permission problems change to su user (Password is iplinux or your new admin password)
  • To figure out your ip address you can use the command: ip address on the console.
  • loadkeys de can be used to change to german keyboard layout (etc.)
  • partitions[1]:
    • /dev/sda1 ext2: 350MB (contains ramdisk, rootfs and kernel)
      • the real partition format size is smaller but there is no need to monitor sda1 as sda1 won't grow during usage of the App Platform
      • sda1 is completely exchanged during an image update and its size might have changed but will never exceed these 350MB
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB
  • VMWare Tools: Open VM Tools
  • Qemu Guest Agent: installed since 110032
    • The qemu guest agent can be configured in Proxmox with Virtio or ISA. You must use ISA, as Virtio is currently not supported.
  1. Maximum partition sizes are given. The real sizes are returned by the linux 'df -h'-command. These real values will be lower, but can grow in the future with new releases.

Installation

ARM Gateway

If you setup a Gateway with the install procedure, the App Platform is installed automatically (Https Download has to be allowed and shouldn't be blocked by any firewall.)

You can also install it manually:

  • Open App Platform -> General and Enable Linux Support. Restart the gateway.
  • You need to enable Proxy-ARP on ETH0 or ETH1, so your Gateway and the Linux Appliance will share the same physical interface.
  • Open App Platform -> IP and configure the IP settings of the App Platform. Restart the Gateway.
  • Open App Platform -> Installation and select the given version or enter an own path to the app-platform-armel.img image file.
  • The installation runs without any further required step.

ARM64 Gateway

If you setup a Gateway with the install procedure, the App Platform is installed automatically (Https Download has to be allowed and shouldn't be blocked by any firewall.)

You can also install it manually:

  • Open App Platform -> General and Enable Linux Support. Restart the gateway.
  • You need to enable Proxy-ARP on ETH0 or ETH1, so your Gateway and the Linux Appliance will share the same physical interface.
  • Open App Platform -> IP and configure the IP settings of the App Platform. Restart the Gateway.
  • Open App Platform -> Installation and select the given version or enter an own path to the app-platform-arm64.img image file.
  • The installation runs without any further required step.

Virtual machine

  • Import the image into your server environment.
  • Edit the disk size, if needed.
  • Start the machine and wait until it reboots and starts again.
  • Note: If you need to access an IP addresses available through a VPN connection from from inside the virtual machine, it could be that you need to set the network of your VM to NAT (and also add the URL for an IP to /etc/hosts)

More Configuration Hints regarding VM Ware

Backup of the Apps

Each App Service can have multiple instances and each instance has its own database. The manager app itself also has its own database.
There are no other files which need to be backuped.
The standard way to backup the databases is through the Devices App Reference13r3:Concept_App_Service_Devices#Backups.

An alternate way is to use a command file which is similar to the command files from the firmware.

Commands

Hash parameters

  • #L App Platform label (neu), e.g. 10024
  • #A App label (neu), e.g. 130004
  • #I instance name (neu), e.g. reporting1
  • #D instance domain (neu), e.g. innovaphone.com
  • #m MAC address of the LAP, e.g. 00ab11eeff
  • #d Current date and time (plain UTC without daylight saving and timezone adjustments) 20051010-170130
  • #bn rolling backup index
  • ## escapes a hash mark

App Platform Infrastructure and Concept

Webserver

The app platform includes a webserver that is highly optimized for handling many Websocket connections at a low memory footprint. All apps use that webserver by registering for specific HTTP subpath. So they can all use the same HTTP/HTTPS ports - typically the standard ports.

Import Custom SSL Certificate

You have to upload a PEM Certificate with the following chain structure and without password encoding.

-----BEGIN CERTIFICATE-----
(certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate: DigiCertCA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
(certificate Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----

If you have problems generating the complete and correct certificate chain, you can use tools such as https://whatsmychaincert.com/ as an aid. Please keep in mind that you never give away the private part of your key.

Known issues

  • The app platform webserver can use only the default http/https ports 80/443.
  • By design, there is no possibility to restore the default webserver certificate on the App Platform, if another certificate was once uploaded.
    • If nevertheless needed, you must login with Putty (see Retrieve files) and execute these commands as root:
    • psql -d manager -c "DELETE FROM config WHERE name='webserverCertificate'"
    • /etc/init.d/S92manager restart
  • The app platform webserver interprets URLs case-sensitive. In other words, http://<addr>/file.txt is not the same as http://<addr>/FILE.TXT

Database

The app platform creates a database for each app instance with a given password. In the installer there will be used randomly generated passwords. You can set a new database password for every instance in the Application Platform.

The apps should store all data in that database. That makes sure that a consistent backup and restore of app instances can be done by the app platform manager. In hosted scenarios, having separate databases for each instance also makes sure that the data of different customers are clearly separated and can easily be moved from one physical platform to another.

The default configuration decline database request from extern. If you need external access you can change the PGSQL configuration (as root) in the file /mnt/sda2/pgsql/pg_hba.conf. After editing pg_hba.conf, the database-service has to be restarted with the command /etc/init.d/S50postgresql restart (Please think about it before you do it, because a better way is to create your own local app with local database access.)

You can find the official documentation for the pg_hba.conf here: https://www.postgresql.org/docs/14/auth-pg-hba-conf.html

Please note that you do this on your own responsibility, that it makes sense to create a backup of the original pg_hba.conf 
and that you may break your database server if you do not know exactly what you do!

Sometimes the Apps are not proper working after this. You can wait a while until everything is working again or you go the hard way:

  • stop the manager
    • /etc/init.d/S92manager stop
  • restart the database-service (maybe not really necessary, but makes a good feeling)
    • /etc/init.d/S50postgresql restart
  • start the manager
    • /etc/init.d/S92manager start

App Platform Manager

The App Platform Manager is the central component of the App Platform. It does the following:

  • Installing app services by downloading the binaries from an app store.
  • Running and monitoring app services. If an app service crashes it is restarted, automatically.
  • Management of app instances and providing them with the environment they need:
    • A database
    • A webserver path
    • A password for authentication
  • Backup and restore of app instances.
  • Collecting debug information like tracing and crash dumps.
  • System monitoring (CPU usage, memory usage, etc).

App Services

App services are runned by the App Platform Manager. They implement an interface that is used by the manager to start, stop and configure app instances. Each service runs in a separate child process of the manager.

App Instances

The actual functionality of an app service is provided by app instances. They run in the same process as the app service but have a distinct webserver path and their own database. There can be 0..n instances of an app service. Instances can optionally host (web) apps that can be opened in the myApps client.

Each instance of an App service has two passwords. The instance password itself and a database password. If you want to change it you must update the password on both sides.
The instance password must match to the password in the corresponding PBX App objects, while one instance can have different App objects.
The database password must be just known to the manager and not outside of the App Platform.

The Database name and Database user are both limited to a length of 63 characters. By default, the App Manager would create <domain>_<instance-name> as value for both. This is why it is recommended to use instance names so that length(name) + length(domain) is less than 63 characters. However, if this is not possible, you can manually shorten the suggested values (both must still be unique on your App Platform).

Cleanup database

You can cleanup the instance database inside the instance settings. This starts the vacuumdb process for the instance database which frees disk space of deleted rows.

Note that this process locks the database and that sufficient disk space is needed to complete it!

External database host

You can optionally configure an external database host, if the database shall be hosted on an external host.
Note that the App Platform Manager still creates a local database, which is then not used as long as the external host is configured.

What are external databases useful for?:

  • User data is too large (files are stored as BLOB in the database)
  • Other (non-innovaphone) database servers are to be used due to other regulations
  • There is already an external database cluster with its own backup/restore processes which should be used
  • More performance is required (please note that a local UNIX socket provides much faster results than a remote database server). Depending on the app and the task of the app, it can still be useful to use a separate server if it is faster.

Supported database types

In principle, PostgreSQL and MySQL are supported. However, please note that the respective app must be designed for the target database server type. If an app has been developed for PostgreSQL, it cannot be operated on a MySQL server.

Please note: All innovaphone apps are developed exclusively as PostgreSQL apps.

MySQL therefore only makes sense if own apps are developed and MySQL or certain MySQL features are to be used. For such apps, it is very important to deactivate the automatic Application Platform APP-Backup. The database backup process of the App Platform uses PostgreSQL backup commands, which is why the backup would fail, so the app must be excluded from the backup, and you must take care of the backup yourself.

Database creation

If you use an external database, you have to create the database itself. The App Platform Manager itself creates a PostgreSQL database like this (you may respect this on your database host to be compatible with the database implementation inside the Apps):

  • CREATE DATABASE "dbname" ENCODING 'UTF8';
  • CREATE USER "dbuser";
  • ALTER USER "dbuser" WITH PASSWORD 'dbpassword';
  • GRANT ALL PRIVILEGES ON DATABASE "dbname" TO "dbuser";
  • ALTER DATABASE "dbname" OWNER TO "dbuser";
  • REVOKE CONNECT ON DATABASE "dbname" FROM public;

Database connection

There are different possibilities to specify the host. You can use a DNS-Name, IPv4 or IPv6. A port can be optionally added with a colon.

Our PostgreSQL implementation sets sslmode=prefer, so SSL is used by default if enabled on the server.

Backup/Restore

External databases are still backuped as normal, but you can't restore such a database with the App Platform Manager on the external host! You must restore such a dump manually on your database host.

Database redundancy

In principle, it is possible to create redundancies for failure scenarios through this function.

However, there are different scenarios that need to be evaluated on a case-by-case basis:

  • One database, multiple App Platforms (primary online / secondary offline) where multiple apps access the same database.
    • This mode of operation is legal as long as you can really ensure that only one App Platform is active at a time.
  • One database, multiple App Platforms (primary online / secondary online) with multiple apps accessing the same database.
    • This operating mode cannot be used for innovaphone apps, as the app itself would have to be developed for such a mode, which it is not.

However, we advise great caution here! The use of the same databases from different nodes may only be used if you can ensure that the databases or runtime environments of apps cannot get into a split-brain mode.

Relationship between app instances and app objects in the PBX

Typically an app instance is connected to one or more app objects in a customer PBX. This is done by configuring the same parameters on both sides:

  • URL
  • Password

The password is used by the PBX for authenticating itself, users and services against the app instance.

Some apps need a websocket connection with the PBX. When "websocket" is activated at the app object, the PBX establishes a websocket connection to the app instance and provides the APIs that are configured at the app object.

Supported scenarios

It is important to understand that the concept does not do any assumptions on how PBXes and APs are correlated. So you can have

  • One App Platform for one customer
  • One App Platform for many customers
  • Many App Platforms for one customer
  • Many App Platforms for many customers

Attention: The V13 installer can only configure the scenario "One App Platform for one customer". If you want to have a different scenario, you have to configure it manually.

For hosting or cloud scenarios you need special scenarios. Please refer our V13 Hosting instructions.

Restrictions

Currently we don't have redundancy for app instances or App Platforms.

Update of the App Platform itself

The App Platform is build on top of buildroot and will receive updates and fixes from time to time.
You can update the used build inside the Manager App by using the Update button at the top.

It is strongly advised to make a full backup (VM) or at least backup all apps (Gateway) before you run such an update!

App services and multi-threading

Each App Service runs in a single process with no multi-threading, independent of the instances of the App Service. However, each instance maintains its own database connection and the database responds to this connect with the creation of a new process. Each app service therefore uses 1+n threads (where n is the number of instances). All communication between app service instances and their clients must pass the single-threaded web server. On platforms with multi-threading support (i.e. VMware or Hyper-V), up to 1 + m + m * n (with m being the number of Apps and n the number of instances per App) threads can be utilized.

AP Manager settings

General

  • Enable Developer mode: in developer mode, apps can be manually uploaded without an App Store
  • Disable App security: each App has an own unix user and if this flag is set, the user can login with SSH for debugging
  • App Store URL: the URL to the App Store where Apps are searched and also an update of the AP image itself
  • Devices app URL: the URL to the Devices App to manage the AP through Devices
  • Devices app URL 2: don't use!
  • App Platform DNS name: currently not used
  • NTP server 1/2: NTP servers for this AP (in addition to NTP servers retrieved by DHCP)
  • Timezone string:
  • DNS server 1/2: DNS servers for this AP (in addition to DNS servers retrieved by DHCP)
  • Database optimization time: The database optimization process will be started at this hour (local time), with a random offset of up to 7 hours
  • Command file: obsolete, use the Devices App for backups!

Security

  • Webserver certificate: upload a webserver certificate in PEM format
  • AP Manager password: the password to the web interface of the AP Manager (normally set through Devices)
  • Linux root user: password of the root user (normally set through Devices)
  • Linux admin user: password of the admin user (normally set through Devices)

Alarms and events

  • URL: URL to the Events app
  • Username/Password: HTTP credentials of the Events app
  • Email address: an email address which will get emails on full disk alarms/warnings
  • Threshold: All Apps will be stopped and an hourly email sent on reaching this threshold. An alarm is generated 10% before reaching this threshold and an email is sent every 24 hours.

SMTP

SMTP server settings for sending emails from within the AP Manager.

Registered Access Domains

App Platform replication

Requirements

  • 13r3 on both standby servers and the primary server
  • at least an App Platform image starting with version 110002 (otherwise postgresql is too old)
  • standby and primary servers must have the same system architecture
    • arm gateways (IPx11) just with arm gateways (IPx11)
    • arm64 gateways (IPx13) just with arm64 gateways (IPx13)
    • x86_64 virtual machines just with x86_64 virtual machines
  • the disk size on standby servers must be at least equal as the disk size on the primary server

General

The configuration is done within the App Platform Manager next to the settings button under "Replication".

Off

No replication is done at all.

Primary

The App Platform acts as primary server for one or multiple standby servers.
Max 8 standby servers can be configured.

Every standby server must get a unique name on the primary server which is then also configured on the standby server itself. This is due to the fact, that the primary server preserves data for every standby server while it's not reachable.
A standby server name must be a valid domain name, although this name is currently not used as DNS name (but might be used in the future).

The password is randomly pregenerated and must be used as handed by the App Platform Manager.

PostgreSQL port

The default PostgreSQL port is 5432 and can't be changed on the primary. It must be reachable through your firewall or at least forwarded on a different port towards this App Platform on port 5432.

Standby

Configure the password from the primary and one of the not yet used standby server names.
The host and port must be reachable through the network. You can specify a non default port which then must be forwarded somewhere else to port 5432 on your primary.

  • Note: Do not configure the same standby name on different standby servers, as this will break the replication on at least one standby!
  • Note: On a standby, all locally installed apps are removed during installation

Failure detection

The replication state on both primary and standby servers are continuously monitored.
In case of a broken connection, an alarm is triggered at once and just cleared if the failure goes away.

If such an alarm lasts for 5 minutes an email is send. The recipient address must be configured in the App Platform Manager settings under "Alarms and Events" and you must also provide a valid SMTP configuration.

Failover

There is no automatic failover procedure. If you detect the failure state, you must get active yourself.

Standby down

Fix the standby server and see if the replication comes up again. If not, setup a new standby server which will replicate the whole database cluster from scratch.

Primary down

If the primary is down and cannot be brought into life again, you must take the following steps, depending on your configuration:

One standby server

  • Disable the replication in the replication configuration. You will then have a standalone App Platform afterwards.

Multiple standby servers

  • Change the replication type on one of the standby servers from standby to primary. Please note that the other standby entries must still exist here (they are automatically offered)!
  • Change the host entry on the other standby servers to the new standby server. The standby server will sync the whole database cluster again, which will take time.

Configuration changes

You must also tell the PBXes and other tools to use the new primary server.

  • You may simply change the IP address behind the DNS of your primary server to point to the new primary server.
  • Without DNS, you must currently modify the configuration by replacing the IP address of the old primary with the new address.
    • This must be done in every master PBX configuration.
    • This must be done in every App configuration which may use this IP address, e.g. Devices which rolls out an alarm server configuration

App Platfom update

If you're running an active standby server and the major version of the PostgreSQL database didn't change, you can simply update primary and standby servers independently.

If it contains a new major version of the PostgreSQL database, you must follow these steps:

  • deactivate replication on all standby servers (which promotes these standby servers to active servers)
  • ensure that the same App Platform Manager version is running on primary and standby servers!
  • perform the App Platform update on the primary server
  • perform the App Platform update on the standby servers
  • reactivate replication on the standby servers (which will do a full replication again, so it might take some time)

Alternativly you may setup the standby server from scratch directly with the new App Platform image version.

Sadly there is currently no better method offered by PostgreSQL to perform such an upgrade.

Technical backgrounds

Streaming replication

We use the asynchronous streaming replication of PostgreSQL. A transaction thus just waits for a commit on the primary server. Standby servers will receive the transaction asynchronously afterwards to avoid a reduced writing performance.
A primary server keeps the neccessary WAL files for every standby server until the standby fetched the neccessary information.
This fact can lead to higher disk space usage on the primary if one or multiple standby servers are offline for a while.

Take care to fix broken replications as soon as possible to avoid a full disk on the primary server!

What is replicated

The whole database cluster of an app platform is replicated. This includes every database of every instance and the database of the App Platform Manager itself.

App Services on standbys

The database of the App Platform Manager itself also contains the App service binaries to be able to restore applications in a failover case.
You do not have to install App services on the standby if you install them on the primary.
App Platform Manager and Webserver can be updated as usual through the App Store or the Devices App if the standby is connected to Devices.

App Platform Manager configuration of the standby server

The App Platform Manager configuration of the standby server cannot be stored inside the database, as the database is readonly. So it is stored inside a configuration JSON file under /home/root/standby.conf.
This configuration is backed up as usual by a Devices backup job and can be also manually restored on a standby (not on a primary though!).

Configuration options from the primary are not synced to the standby server, as the standby server has its own configuration (as it could be in a different network etc.)!

No automatic failover

There is currently no automatic failover mechanism. PostgreSQL doesn't offer multi master replication, so writing is just possible on the primary itself. Standby servers have a readonly database.

Let's consider the following use case if we would implement an automatic failover with the current technical possibilities:

  • The primary server is connected to a master PBX.
  • A slave PBX is also connected to the primary server in another location.
  • A standby server is in this location and replicates from the primary.
  • The internet connection between primary and standby fails.
  • The standby promotes itself to primary, so that the master PBX now writes to the old primary and the slave PBX now writes to the "new" primary.
  • The internet connection comes up again and both primary and standby servers now have databases which are out of sync and which cannot be merged.

Security

  • port 5432 must be reachable on the App Platform
  • For every standby a PostgreSQL user is created with the REPLICATION role on the primary. Just this user has access to the database server from non localhost connections.
  • You cannot establish a standard connection with such a user and modify/read from tables.
  • The replication connection is established over TLS by default.

Known Issues

Reboot after an image update hangs (ARM gateway)

If it happens, that the App Platform doesn't recover after the reboot, please open the Admin UI of the corresponding gateway and take a look at App Platform -> General.
If Kernel command line is set to /dev/ram0, the App Platform booted the ramdisk.

Try to fetch the upgrade log file: https://[APP-PLATFROM-IP/DNS]/manager/ramdisk.log
If this doesn't work, try to login with putty in this case (default credentials admin/ipapps and root/iplinux) and issue this command:
cat /apps/install_step1.log

If this file contains finished at the end, you can reconfigure the settings under App Platform -> General:

  • press Stop
  • Initrd file: empty
  • Kernel command line: root=/dev/sda3
  • Ramdisk size: empty
  • press Start

The App Platform should boot and run the already updated image.

---WARNING---
 If the file doesn't contain "finished" at the end, you may still need to wait, as an upgrade may take some time!
 Do not reconfigure without being sure that the upgrade has finished, otherwise your system may won't run!

Reboot after an image update boots always into rescue mode (virtual machine)

This most likely means that the image installation didn't finish the first time and was interrupted before the bootloader settings could be changed back to the default partition.

If your App Platform doesn't boot or isn't normally accessible after manual selection of the standard entry in the boot menu, you need to revert to a snapshot/backup prior to the update.
If your App Platform runs normally though, you can follow this instruction to change the default boot entry:

  • boot into the rescue partition
  • login with root/iplinux
  • edit the file /boot/grub/grub.cfg
    • search the line with set default=0 and change it to set default=1
    • reboot

Reboot after an image update doesn't start as update is already running

If it happens, that the App Platform doesn't want to start an update because one is already running, please open the Admin UI of the corresponding gateway and take a look at App Platform -> General.

  • Shutdown the App Platform and Stop it.
  • Set Kernel command line to /dev/ram0.
  • Set Initrd file to ramdisk.ext2.xz.
  • Set Ramdisk size to 100000.
  • Start the App Platform now.

The App Platform now either applies the image update or it reboots into the old image. Try the image update afterwards again.

Webserver doesn't respond after an image update

If you can't reach the web UI after an image update, please try to connect with SSH as admin user and delete old coredumps, which may prevent the App Platform Manager from starting correctly:

  • su root (become root)
  • rm -f /mnt/sda2/log/core_dumps/*/*
  • /etc/init.d/S92manager restart

Check if you can now reach the App Platform again.

Tracing

Each App Service has its own log file, which can be accessed through the Manager App. You can configure a log file size for each App Service.
Each App Intance has its own trace flags. The following trace flags can be set:

  • Alarm client: used by the manager to send alarms to an alarm server
  • App: logs from the App Service itself
  • App WebSocket: logs app websocket connections (e.g. from PBX objects to an App Service or from the UI to the App Service)
  • AppSharing: just native clients
  • AppProxy: just native clients, logs requests which are proxied between the local webserver and the remote server
  • Audio: just native clients
  • Browser: just native clients
  • Command: the command interface is used to execute shell commands, e.g. used by the manager App
  • Config: logs config changes of an App
  • Database: database logs
  • DB files: database file logs
  • DNS: DNS request logging
  • DTLS: just native clients, DTLS request logging
  • Ethernet: interface to get ethernet adapater infos, just manager App
  • File: logs for file system access (synchronous), e.g. manager App
  • Files: logs for file system access (asynchronous)
  • HTTP client: http client logs
  • HTTP file: logs for static HTTP files
  • ICE: just native clients
  • LDS: local domain sockets
  • Media: just native clients
  • Media channel: just native clients
  • Process: IProcess interface logs which is used for spawning, killing processes etc.
  • SMTP: SMTP client logs
  • TCP: TCP logs
  • Time: ITime interface logs
  • TLS: TLS logs
  • TURN: just native clients
  • UDP: UDP logs
  • Video: just native clients
  • WebSocket client: logs outgoing websocket connections
  • Webserver traffic: logs incoming HTTP traffic, which is forwarded from the webserver to the App
  • WebDAV service: logs WebDAV requests to the App
  • Webserver: enables webserver specific logs

RPCAP

If you open the Manager App, click on the Manager in the left list and then on the Diagnostics button, you can enable RPCAP.
You can add the interface in wireshark with the string:

rpcap://<APP-Platform-IP>/eth0

Please don't forget to disable RPCAP after your testing!

How-Tos

How to retrieve files from the App Platform

To retrieve files from the App Platform which can not be retrieved via the App Platform manager UI, you can connect to the App Platform using the SCP protocol on port 22.

First copy files to /home/admin as root with Putty or another SSH client:

  • use the DNS name or IP address of your App Platform (not the PBX)
  • use user admin and the appropriate password (ipapps by default)
  • use su root to be root (iplinux as default password)


You can now copy files to /home/admin (e.g. from /var/log/apps/manager/...).

You can create a tar archive with all logs like this:

  • cd /home/admin
  • tar -cf - /var/log/* | xz -z - > log.tar.xz
    • you may want to exclude coredumps (due to their size):
    • tar -cf - --exclude=/var/log/core_dumps/* /var/log/* | xz -z - > log.tar.xz
  • chown admin:admin /home/admin/log.tar.xz


Then copy the files to your local system, e.g. with WinSCP

  • use SCP as protocol (NB: WebDAV is not supported on most of the directories on the App Platform )
  • use the DNS name or IP address of your App Platform (not the PBX)
  • use user admin and the appropriate password (ipapps by default)
  • use /home/admin as start directory
  • copy the needed files

App Platform disk space warning

If the configured threshold is reached, all Apps are stopped inside the App Platform . You must then free disk space somehow.

Find large instances

Click through your apps in the tree on the left side and take a look at the database size of each instance.

Delete data inside an instance

It depends on the type of app if you can delete data or not. E.g. you can start the Files app and delete files inside this app.
This won't reclaim disk space though due to the way how PostgreSQL databases work, so you need to follow this guide to reclaim the disk space:

  • start the corresponding App
  • delete data inside the App
  • optimize disk usage of the database by cliking on the button "Clean up database" under Edit instance. If this does not help, continue with the following steps.
  • stop the corresponding App again
  • download a backup of the instance (backup button at the top), this backup contains the whole instance data, also the password of the instance
  • delete this specific instance
  • restore the downloaded backup (restore button at the top)
Depending on the hardware and the size of the instance, this process may take hours to complete!

Resize the disk

The resizing of a disk is just possible for virtual machines, see Resizing the disk of a Virtual machine.

Delete the whole instance

If there is no other possibility, you can delete the whole instance. Afterwards you recreate the instance with the same values and a new random password. Don't forget to set this password in the corresponding PBX App objects though!

Restart the Apps or the App Platform

If you have enough disk space again, you must either restart the whole App Platform or you manually start all Apps again.

App Platform/Apps app not online anymore due to full disk

If the apps app is not online anymore and you can't access any apps anymore, try to login with an SSH client to see if your disk is full. Login as admin and afterwards as root (su root).

  • issue df -h and see the disk usage of /dev/sda2, if this is 100%, your disk is too full
  • stop the manager
    • /etc/init.d/S92manager stop
  • empty the 500 MB file, which is exactly for this case here (manager must be 13r1 SR9 or higher to have this file)
    • echo "" > /mnt/sda2/empty_if_no_space
  • delete log files to recover some space
    • rm /var/log/apps/*/*
    • rm /var/log/core_dumps/*/*
  • restart the postgresql server (see the output if this worked or not)
    • /etc/init.d/S50postgresql restart
  • restart the manager
    • /etc/init.d/S92manager restart
  • wait until everything is online again
  • open the Apps app and try to find the instance which uses the most disk space and try to delete files/content from it
    • you may want to stop all app services first to prevent more writes to the database
    • if not possible, you can delete this instance, but you'll loose all data from this instance then!
  • after that you can try Reference13r3:Concept_App_Platform#Shrink_the_physically_size_of_PostgreSQL_database_files to free up space
    • If this does not work you can create a backup from the database, delete the database and import the database again.
  • free up at least 500 MB so that the manager can create the file again
  • delete /mnt/sda2/empty_if_no_space if you are done and restart the manager:
    • rm /mnt/sda2/empty_if_no_space
    • /etc/init.d/S92manager restart
    • the manager restart automatically recreates the empty_if_no_space file if this file doesn't exist
Make sure, that you do not have backups configured to a local files instance while this files instance is not excluded from backups.
An instance can be excluded from backups in the instance settings in the App Platform Manager.

If all of this doesn't help, you can resize the file system on a VM:

Resizing the disk of a Virtual machine

Do NOT resize if you're running an App Platform version higher than 110000 and lower than 110027 or your data will be lost!
Please update your App Platform to version 110027 or higher before you start resizing your disk.
  • stop the VM
  • expand the disk using your VM utilities
  • start the VM and boot from the first boot entry rescue/setup
  • login with root and iplinux
  • execute this command: /home/root/install_step1.sh log.txt resize
  • the VM reboots automatically after a successful resize

Shrink the physically size of PostgreSQL database files

Tuples that are deleted in your database are not physically removed from the database-file. So the claimed space on the harddisk is still in use after the delete operation. If you need to free up some disk space you can force to reorganize the physically database-file on your harddisk.

You can also start this process through the instance settings as long as the App Platform Manager is still running.

Important: You are operating on the Database, you have to make a Backup of your Database before you do this!

Login via SSH to the APPlatform with the admin User

su root
/etc/init.d/S92manager stop  # not always needed, but in case of database errors recommended, of course no app is online then
sudo -u postgres reindexdb -a
sudo -u postgres vacuumdb -a -f
/etc/init.d/S92manager restart # just execute if the manager has been stopped above
sudo -u postgres reindexdb -d dbname # for a single database with dbname
sudo -u postgres vacuumdb -d dbname -f # for a single database with dbname

Note: This may take some time and CDRs (or other data written to a DB) won't be received during this time.

After the process you can check the free dispace via df -h. You can check the claimed space from the database file with the command du -sh /mnt/sda2/pgsql/.

Change IP Addresses / DNS Names / System Name

If you want to change the System Name or the DNS Name of the PBX and/or App Platform Platform you must change records manually in the described order! You have to know the Admin password to directly Login to the App Platform .

App Platform

Settings - General
  • Devices app URL
  • App platform DNS name
Settings - Alarms and Events
  • URL
All Instances
  • Domain
  • Webserver path

PBX

Download the configuration and search/replace in the config file. Upload the config file and reboot.

Manual:

Depending on your change you have to activate/deactivate the Setting Operation without DNS

Additional Devices / Steps

  • Devices Registration URL
    • There is no automatism to change the URL on all devices in your setup.
    • If you have the option to use DHCP, you can temporarily overwrite the Update URL and execute a custom update script to change the Device Registration URL
  • Alarm server
  • Reverse Proxy configuration
  • Change Domain Name in Devices if you have also changed the system name

How to recover from a broken File System

Sometimes you may find messages in the messages log file (in var/log) like

initial error at 1500329378: ext4_journal_start_sb:328
last error at 1500329378: ext4_journal_start_sb:328

Or you get events like "Broken file system" from your App Platform .

This indicates a file system failure on the Linux Installation.

When the Linux file system is broken, you can try to repair it using some command line Linux tools.

If this doesn't fix your issue, you need to replace the SSD with a new one, re-install the App Platform and any applications and restore your backups.

Gateway

  • Open the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/sda3 to root=/dev/ram0
    • modify the Initrd file to ramdisk.ext2.xz
    • modify the ramdisk size to 100000
    • start Linux again
This will run Linux on another (hopefully sane) partition.
  • use putty to log in to the LAP's command line ( try either user: root, pw: iplinux OR user: admin, pw: ipapps)
    • use this to be root su - root (password iplinux)
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • go back to the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/ram0 to root=/dev/sda3
    • clear the Initrd file field
    • clear the ramdisk size field
    • start Linux again
This will run Linux on the original partition.

VM

Restart the VM and select the first entry in the boot menu from grub.

  • use putty to log in to the LAP's command line ( try either user: root, pw: iplinux OR user: admin, pw: ipapps)
    • use this to be root su - root (password iplinux)
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • Reboot your VM

How to create a memory dump of a running process

Sometimes it's usefull to have a memory dump of a running process to investigate certain issues.

  • login with SSH and become root
  • find out the PID of the relevant process, e.g. with ps aux | grep /apps/manager/manager | grep -v grep
  • gdb --pid PID -ex gcore --batch
  • this creates a core file core.PID
  • download this core file with WinSCP or similar tools