Reference13r1:Concept App Platform

From innovaphone wiki
Jump to navigation Jump to search
There are also other versions of this article available: Reference13r1 (this version) | Reference13r2 | Reference13r3 | Reference14r1 | Reference14r2

General

Requirements

  • V13 or up
  • Gateway (arm): IPx10 (with CF card) or IPx11 (with mSATA SSD)
  • Virtual (x86_64)

Default credentials

During INSTALL, the default passwords are replaced with the global Admin PW!

  • SSH-Login with admin and ipapps
  • root login with root and iplinux (the root login is not directly possible, you have to login as admin first and use the command su root)
  • manager App (web login) pwd

App Platform - arm (Gateway)

  • The installation image has a size of ~50MB. During installation, the following partitions are created:
    • /dev/sda1 fat32: 200MB (contains ramdisk, rootfs and kernel)
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB

When comparing potential performance of the IPxx11 platform compared to the IPxx10 platform, there are some major differences:

  • SSDs found in the xx11 are faster and more reliable than the CF found in the xx10
  • the available RAM for the AP (as specified in column RAM for LAP (GB) out of RAM in chapter Technical data and recommended number of users supported of How to implement large PBXs) is factor 6 larger on the xx11 (1,536 GB) than on the xx10 (0,256 GB))
  • the xx11 has gigabit Ethernet while the xx10 has 100Mbps Ethernet. The xx10 is therefore not well suited for Apps with larger network traffic, such as Recordings
  • the CPU of the xx11 (although it runs on the same frequency) is roughly 20% faster than the xx10

While it is hard to predict the performance of the AP in a specific scenario, we see that in a real life environment an AP running on an xx11 platform can well support 150 users. The xx10 platform is estimated to support 120 users. Because CPU performance is the limiting factor, larger setups can be built based on the virtual machine platform (see App services and multi-threading below).

App Platform - x86-64 (Virtual Machine 64bit)

  • The default disk size is 16GB. It should be increased before the first start if needed!
  • Multiple CPUs are supported, default is one CPU
  • default RAM: 512MB
  • static IP address, DNS, Gateway can be configured with the command setip on the console. Run setip --help to get a list of parameters. (Example: setip --addr=x.x.x.x --mask=x.x.x.x --gateway=x.x.x.x --dns1=x.x.x.x)
  • If you have permission problems change to su user (Password is iplinux or your new admin password)
  • To figure out your ip address you can use the command: ip address on the console.
  • loadkeys de can be used to change to german keyboard layout (etc.)
  • partitions:
    • /dev/sda1 ext2: 350MB (contains ramdisk, rootfs and kernel)
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB

Installation

ARM Gateway

If you setup a Gateway with the install procedure, the App Platform is installed automatically (Https Download has to be allowed and shouldn't be blocked by any firewall.)

You can also install it manually:

  • Open App Platform -> General and Enable Linux Support. Restart the gateway.
  • You need to enable Proxy-ARP on ETH0 or ETH1, so your Gateway and the Linux Appliance will share the same physical interface.
  • Open App Platform -> IP and configure the IP settings of the App Platform. Restart the Gateway.
  • Open App Platform -> Installation and select the given version or enter an own path to the app-platform-armel.img image file.
  • The installation runs without any further required step.

Virtual machine

  • Import the image into your server environment.
  • Edit the disk size, if needed.
  • Start the machine and wait until it reboots and starts again.
  • Note: If you need to access IP addresses available through a VPN connection from inside the virtual machine, it could be that you need to set the network of your VM to NAT (and also add the URL for an IP to /etc/hosts)

More Configuration Hints regarding VM Ware

Backup of the Apps

Each App Service can have multiple instances and each instance has its own database. The manager app itself also has its own database.
There are no other files which need to be backuped.
The standard way to backup the databases is through the Devices App Reference13r1:Concept_App_Service_Devices#Backups.

An alternate way is to use a command file which is similar to the command files from the firmware.

Commands

Hash parameters

  • #L App Platform label (neu), e.g. 10024
  • #A App label (neu), e.g. 130004
  • #I instance name (neu), e.g. reporting1
  • #D instance domain (neu), e.g. innovaphone.com
  • #m MAC address of the LAP, e.g. 00ab11eeff
  • #d Current date and time (plain UTC without daylight saving and timezone adjustments) 20051010-170130
  • #bn rolling backup index
  • ## escapes a hash mark

App Platform Infrastructure and Concept

Webserver

The app platform includes a webserver that is highly optimized for handling many Websocket connections at a low memory footprint. All apps use that webserver by registering for specific HTTP subpath. So they can all use the same HTTP/HTTPS ports - typically the standard ports.

Import Custom SSL Certificate

You have to upload a PEM Certificate with the following chain structure and without password encoding.

-----BEGIN CERTIFICATE-----
(certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate: DigiCertCA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
(certificate Key in PKCS#1 format: your_domain_name.key)
-----END RSA PRIVATE KEY-----

or

-----BEGIN PRIVATE KEY-----
(certificate Key in PKCS#8 format: your_domain_name.key)
-----END PRIVATE KEY-----

Known issues

  • The app platform webserver can use only the default http/https ports 80/443.
  • By design, there is no possibility to restore the default webserver certificate on the App Platform, if another certificate was once uploaded.
    • If nevertheless needed, you must login with Putty (see Retrieve files) and execute these commands as root:
    • psql -d manager -c "DELETE FROM config WHERE name='webserverCertificate'"
    • /etc/init.d/S92manager restart
  • The app platform webserver interprets URLs case-sensitive. In other words, http://<addr>/file.txt is not the same as http://<addr>/FILE.TXT

Database

The app platform creates a database for each app instance with a given password. In the installer there will be used randomly generated passwords. You can set a new database password for every instance in the Application Platform.

The apps should store all data in that database. That makes sure that a consistent backup and restore of app instances can be done by the app platform manager. In hosted scenarios, having separate databases for each instance also makes sure that the data of different customers are clearly separated and can easily be moved from one physical platform to another.

The default configuration decline database request from extern. If you need external access you can change the PGSQL configuration (as root) in the file /mnt/sda2/pgsql/pg_hba.conf. After editing pg_hba.conf, the database-service has to be restarted with the command /etc/init.d/S50postgresql restart (Please think about it before you do it, because a better way is to create your own local app with local database access.)

You can find the official documentation for the pg_hba.conf here: https://www.postgresql.org/docs/11/auth-pg-hba-conf.html

Please note that you do this on your own responsibility, that it makes sense to create a backup of the original pg_hba.conf 
and that you may break your database server if you do not know exactly what you do!

Sometimes the Apps are not proper working after this. You can wait a while until everything is working again or you go the hard way:

  • stop the manager
    • /etc/init.d/S92manager stop
  • restart the database-service (maybe not really necessary, but makes a good feeling)
    • /etc/init.d/S50postgresql restart
  • start the manager
    • /etc/init.d/S92manager start

App Platform Manager

The App Platform Manager is the central component of the App Platform. It does the following:

  • Installing app services by downloading the binaries from an app store.
  • Running and monitoring app services. If an app service crashes it is restarted, automatically.
  • Management of app instances and providing them with the environment they need:
    • A database
    • A webserver path
    • A password for authentication
  • Backup and restore of app instances.
  • Collecting debug information like tracing and crash dumps.
  • System monitoring (CPU usage, memory usage, etc).

App Services

App services are runned by the App Platform Manager. They implement an interface that is used by the manager to start, stop and configure app instances. Each service runs in a separate child process of the manager.

App Instances

The actual functionality of an app service is provided by app instances. They run in the same process as the app service but have a distinct webserver path and their own database. There can be 0..n instances of an app service. Instances can optionally host (web) apps that can be opened in the myApps client.

Each instance of an App service has two passwords. The instance password itself and a database password. If you want to change it you must update the password on both sides.
The instance password must match to the password in the corresponding PBX App objects, while one instance can have different App objects.
The database password must be just known to the manager and not outside of the App Platform.

Relationship between app instances and app objects in the PBX

Typically an app instance is connected to one or more app objects in a customer PBX. This is done by configuring the same parameters on both sides:

  • URL
  • Password

The password is used by the PBX for authenticating itself, users and services against the app instance.

Some apps need a websocket connection with the PBX. When "websocket" is activated at the app object, the PBX establishes a websocket connection to the app instance and provides the APIs that are configured at the app object.

Supported scenarios

It is important to understand that the concept does not do any assumptions on how PBXes and APs are correlated. So you can have

  • One AP for one customer
  • One AP for many customers
  • Many APs for one customer
  • Many APs for many customers

Attention: The V13 installer can only configure the scenario "One AP for one customer". If you want to have a different scenario, you have to configure it manually.

For hosting or cloud scenarios you need special scenarios. Please refer our V13 Hosting instructions.

Restrictions

Currently we don't have redundancy for app instances or APs.

Update of the App Platform itself

The App Platform is build on top of buildroot and will receive updates and fixes from time to time.
You can update the used build inside the Manager App by using the Update button at the top.

It is strongly advised to make a full backup (VM) or at least backup all apps (Gateway) before you run such an update!

App services and multi-threading

Each App Service runs in a single process with no multi-threading, independent of the instances of the App Service. However, each instance maintains its own database connection and the database responds to this connect with the creation of a new process. Each app service therefore uses 1+n threads (where n is the number of instances). All communication between app service instances and their clients must pass the single-threaded web server. On platforms with multi-threading support (i.e. VMware or Hyper-V), up to 1 + m + m * n (with m being the number of Apps and n the number of instances per App) threads can be utilized.

Known Issues

Reboot after an image update hangs (ARM gateway)

If it happens, that the App Platform doesn't recover after the reboot, please open the Admin UI of the corresponding gateway and take a look at App Platform -> General.
If Kernel command line is set to /dev/ram0, the App Platform booted the ramdisk.

Try to fetch the upgrade log file: https://[APP-PLATFROM-IP/DNS]/manager/ramdisk.log
If this doesn't work, try to login with putty in this case (default credentials admin/ipapps and root/iplinux) and issue this command:
cat /apps/install_step1.log

If this file contains finished at the end, you can reconfigure the settings under App Platform -> General:

  • press Stop
  • Initrd file: empty
  • Kernel command line: root=/dev/sda3
  • Ramdisk size: empty
  • press Start

The App Platform should boot and run the already updated image.

---WARNING---
 If the file doesn't contain "finished" at the end, you may still need to wait, as an upgrade may take some time!
 Do not reconfigure without being sure that the upgrade has finished, otherwise your system may won't run!

Reboot after an image update doesn't start as update is already running

If it happens, that the AP doesn't want to start an update because one is already running, please open the Admin UI of the corresponding gateway and take a look at App Platform -> General.

  • Shutdown the AP and Stop it.
  • Set Kernel command line to /dev/ram0.
  • Set Initrd file to ramdisk.ext2.xz.
  • Set Ramdisk size to 100000.
  • Start the AP now.

The AP now either applies the image update or it reboots into the old image. Try the image update afterwards again.

Webserver doesn't respond after an image update

If you can't reach the web UI after an image update, please try to connect with SSH as admin user and delete old coredumps, which may prevent the AP Manager from starting correctly:

  • su root (become root)
  • rm -f /mnt/sda2/log/core_dumps/*/*
  • /etc/init.d/S92manager restart

Check if you can now reach the AP again.

Tracing

Each App Service has its own log file, which can be accessed through the Manager App. You can configure a log file size for each App Service.
Each App Intance has its own trace flags. The following trace flags can be set:

  • Alarm client: used by the manager to send alarms to an alarm server
  • App: logs from the App Service itself
  • App WebSocket: logs app websocket connections (e.g. from PBX objects to an App Service or from the UI to the App Service)
  • AppSharing: just native clients
  • AppProxy: just native clients, logs requests which are proxied between the local webserver and the remote server
  • Audio: just native clients
  • Browser: just native clients
  • Command: the command interface is used to execute shell commands, e.g. used by the manager App
  • Config: logs config changes of an App
  • Database: database logs
  • DB files: database file logs
  • DNS: DNS request logging
  • DTLS: just native clients, DTLS request logging
  • Ethernet: interface to get ethernet adapater infos, just manager App
  • File: logs for file system access (synchronous), e.g. manager App
  • Files: logs for file system access (asynchronous)
  • HTTP client: http client logs
  • HTTP file: logs for static HTTP files
  • ICE: just native clients
  • LDS: local domain sockets
  • Media: just native clients
  • Media channel: just native clients
  • Process: IProcess interface logs which is used for spawning, killing processes etc.
  • SMTP: SMTP client logs
  • TCP: TCP logs
  • Time: ITime interface logs
  • TLS: TLS logs
  • TURN: just native clients
  • UDP: UDP logs
  • Video: just native clients
  • WebSocket client: logs outgoing websocket connections
  • Webserver traffic: logs incoming HTTP traffic, which is forwarded from the webserver to the App
  • WebDAV service: logs WebDAV requests to the App
  • Webserver: enables webserver specific logs

RPCAP

If you open the Manager App, click on the Manager in the left list and then on the Diagnostics button, you can enable RPCAP.
You can add the interface in wireshark with the string:

rpcap://<APP-Platform-IP>/eth0

Please don't forget to disable RPCAP after your testing!

How-Tos

How to retrieve files from the AP

To retrieve files from the AP which can not be retrieved via the AP manager UI, you can connect to the AP using the SCP protocol on port 22.

First copy files to /home/admin as root with Putty or another SSH client:

  • use the DNS name or IP address of your AP (not the PBX)
  • use user admin and the appropriate password (ipapps by default)
  • use su root to be root (iplinux as default password)


You can now copy files to /home/admin (e.g. from /var/log/apps/manager/...).

You can create a tar archive with all logs like this:

  • cd /home/admin
  • tar -cf - /var/log/* | xz -z - > log.tar.xz
    • you may want to exclude coredumps (due to their size):
    • tar -cf - --exclude=/var/log/core_dumps/* /var/log/* | xz -z - > log.tar.xz
  • chown admin:admin /home/admin/log.tar.xz


Then copy the files to your local system, e.g. with WinSCP

  • use SCP as protocol (NB: WebDAV is not supported on most of the directories on the AP)
  • use the DNS name or IP address of your AP (not the PBX)
  • use user admin and the appropriate password (ipapps by default)
  • use /home/admin as start directory
  • copy the needed files

App Platform/Apps app not online anymore due to full disk

If the apps app is not online anymore and you can't access any apps anymore, try to login with an SSH client to see if your disk is full. Login as admin and afterwards as root (su root).

  • issue df -h and see the disk usage of /dev/sda2, if this is 100%, your disk is too full
  • stop the manager
    • /etc/init.d/S92manager stop
  • empty the 500 MB file, which is exactly for this case here (manager must be 13r1 SR9 or higher to have this file)
    • echo "" > /mnt/sda2/empty_if_no_space
  • delete log files to recover some space
    • rm /var/log/apps/*/*
    • rm /var/log/core_dumps/*/*
  • restart the postgresql server (see the output if this worked or not)
    • /etc/init.d/S50postgresql restart
  • restart the manager
    • /etc/init.d/S92manager restart
  • wait until everything is online again
  • open the Apps app and try to find the instance which uses the most disk space and try to delete files/content from it
    • you may want to stop all app services first to prevent more writes to the database
    • if not possible, you can delete this instance, but you'll loose all data from this instance then!
  • after that you can try Reference13r1:Concept_App_Platform#Shrink_the_physically_size_of_PostgreSQL_database_files to free up space
    • If this does not work you can create a backup from the database, delete the database and import the database again.
  • free up at least 500 MB so that the manager can create the file again
  • delete /mnt/sda2/empty_if_no_space if you are done and restart the manager:
    • rm /mnt/sda2/empty_if_no_space
    • /etc/init.d/S92manager restart
    • the manager restart automatically recreates the empty_if_no_space file if this file doesn't exist
Make sure, that you do not have backups configured to a local files instance while this files instance is not excluded from backups.
An instance can be excluded from backups in the instance settings in the AP Manager.

If all of this doesn't help, you can resize the file system on a VM:

Resizing the disk of a Virtual machine

Do NOT resize if you're running an App Platform version higher than 110000 and lower than 110027 or your data will be lost!
Please update your App Platform to version 110027 or higher before you start resizing your disk.
  • stop the VM
  • expand the disk using your VM utilities
  • start the VM and boot from the first boot entry rescue/setup
  • login with root and iplinux
  • execute this command: /home/root/install_step1.sh log.txt resize
  • the VM reboots automatically after a successful resize

Shrink the physically size of PostgreSQL database files

Tuples that are deleted in your database are not physically removed from the database-file. So the claimed space on the harddisk is still in use after the delete operation. If you need to free up some disk space you can force to reorganize the physically database-file on your harddisk.

Important: You are operating on the Database, you have to make a Backup of your Database before you do this!

Login via SSH to the APPlatform with the admin User

su root
/etc/init.d/S92manager stop  # not always needed, but in case of database errors recommended, of course no app is online then
sudo -u postgres reindexdb -a
sudo -u postgres vacuumdb -a -f
/etc/init.d/S92manager restart # just execute if the manager has been stopped above
sudo -u postgres reindexdb -d dbname # for a single database with dbname
sudo -u postgres vacuumdb -d dbname -f # for a single database with dbname

Note: This may take some time and CDRs (or other data written to a DB) won't be received during this time.

After the process you can check the free dispace via df -h. You can check the claimed space from the database file with the command du -sh /mnt/sda2/pgsql/.

Change IP Addresses / DNS Names / System Name

If you want to change the System Name or the DNS Name of the PBX and/or AP Platform you must change records manually in the described order! You have to know the Admin password to directly Login to the AP-Plattform.

App Platform

Settings - General
  • Devices app URL
  • App platform DNS name
Settings - Alarms and Events
  • URL
All Instances
  • Domain
  • Webserver path

PBX

Download the configuration and search/replace in the config file. Upload the config file and reboot.

Manual:

Depending on your change you have to activate/deactivate the Setting Operation without DNS

Additional Devices / Steps

  • Devices Registration URL
  • Alarm server
  • Reverse Proxy configuration
  • Change Domain Name in Devices if you have also changed the system name

How to recover from a broken File System

Sometimes you may find messages in the messages log file (in var/log) like

initial error at 1500329378: ext4_journal_start_sb:328
last error at 1500329378: ext4_journal_start_sb:328

Or you get events like "Broken file system" from your AP.

This indicates a file system failure on the Linux Installation.

When the Linux file system is broken, you can try to repair it using some command line Linux tools.

If this doesn't fix your issue, you need to replace the SSD with a new one, re-install the App Platform and any applications and restore your backups.

Gateway

  • Open the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/sda3 to root=/dev/ram0
    • modify the Initrd file to ramdisk.ext2.xz
    • modify the ramdisk size to 100000
    • start Linux again
This will run Linux on another (hopefully sane) partition.
  • use putty to log in to the LAP's command line ( try either user: root, pw: iplinux OR user: admin, pw: ipapps)
    • use this to be root su - root (password iplinux)
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • go back to the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/ram0 to root=/dev/sda3
    • clear the Initrd file field
    • clear the ramdisk size field
    • start Linux again
This will run Linux on the original partition.

VM

Restart the VM and select the first entry in the boot menu from grub.

  • use putty to log in to the LAP's command line ( try either user: root, pw: iplinux OR user: admin, pw: ipapps)
    • use this to be root su - root (password iplinux)
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • Reboot your VM