Reference13r1:Concept App Platform: Difference between revisions

From innovaphone wiki
Jump to navigation Jump to search
Line 229: Line 229:
* use ''/home/admin'' as start directory
* use ''/home/admin'' as start directory


You can now log-in to the AP using putty and move the files to retrieve to ''/home/admin''.  
You can now log-in to the AP using putty and copy files to retrieve to ''/home/admin'' (e.g. from /var/log/apps/manager/...).


== App Platform/Apps app not online anymore due to full disk ==
== App Platform/Apps app not online anymore due to full disk ==

Revision as of 14:17, 4 February 2020

General

Requirements

  • V13 or up
  • Gateway (arm): IPx10 (with CF card) or IPx11 (with mSATA SSD)
  • Virtual (x86_64)

Default credentials

During INSTALL, the default passwords are replaced with the global Admin PW!

  • SSH-Login with admin and ipapps
  • root login with root and iplinux (the root login is not directly possible, you have to login as admin first and use the command su root)
  • manager App (web login) pwd

App Platform - arm (Gateway)

  • The installation image has a size of ~50MB. During installation, the following partitions are created:
    • /dev/sda1 fat32: 200MB (contains ramdisk, rootfs and kernel)
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB

App Platform - x86-64 (Virtual Machine)

  • The default disk size is 16GB. It should be increased before the first start if needed!
  • Multiple CPUs are supported, default is one CPU
  • default RAM: 512MB
  • static IP address, DNS, Gateway can be configured with the command setip on the console. Run setip --help to get a list of parameters. (Example: setip --addr=x.x.x.x --mask=x.x.x.x --gateway=x.x.x.x --dns1=x.x.x.x)
  • If you have permission problems change to su user (Password is iplinux or your new admin password)
  • To figure out your ip address you can use the command: ip address on the console.
  • loadkeys de can be used to change to german keyboard layout (etc.)
  • partitions:
    • /dev/sda1 ext2: 350MB (contains ramdisk, rootfs and kernel)
    • /dev/sda2 ext4: depends on disk size (contains databases, log files and apps)
    • /dev/sda3 ext4: 500MB (contains the rootfs)
    • /dev/sda4 swap: 512MB

Installation

ARM Gateway

If you setup a Gateway with the install procedure, the App Platform is installed automatically (Https Download has to be allowed and shouldn't be blocked by any firewall.)

You can also install it manually:

  • Open App Platform -> General and Enable Linux Support. Restart the gateway.
  • You need to enable Proxy-ARP on ETH0 or ETH1, so your Gateway and the Linux Appliance will share the same physical interface.
  • Open App Platform -> IP and configure the IP settings of the App Platform. Restart the Gateway.
  • Open App Platform -> Installation and select the given version or enter an own path to the app-platform-armel.img image file.
  • The installation runs without any further required step.

Virtual machine

  • Import the image into your server environment.
  • Edit the disk size, if needed.
  • Start the machine and wait until it reboots and starts again.
  • Note: If you need to access an IP addresses available through a VPN connection from from inside the virtual machine, it could be that you need to set the network of your VM to NAT (and also add the URL for an IP to /etc/hosts)

More Configuration Hints regarding VM Ware

Backup of the Apps

Each App Service can have multiple instances and each instance has its own database. The manager app itself also has its own database.
There are no other files which need to be backuped.
The standard way to backup the databases is through the Devices App Reference13r1:Concept_App_Service_Devices#Backups.

An alternate way is to use a command file which is similar to the command files from the firmware.

Commands

Hash parameters

  • #L App Platform label (neu), e.g. 10024
  • #A App label (neu), e.g. 130004
  • #I instance name (neu), e.g. reporting1
  • #D instance domain (neu), e.g. innovaphone.com
  • #m MAC address of the LAP, e.g. 00ab11eeff
  • #d Current date and time (plain UTC without daylight saving and timezone adjustments) 20051010-170130
  • #bn rolling backup index
  • ## escapes a hash mark

App Platform Infrastructure and Concept

Webserver

The app platform includes a webserver that is highly optimized for handling many Websocket connections at a low memory footprint. All apps use that webserver by registering for specific HTTP subpath. So they can all use the same HTTP/HTTPS ports - typically the standard ports.

Import Custom SSL Certificate

You have to upload a PEM Certificate with the following chain structure and without password encoding.

-----BEGIN CERTIFICATE-----
(certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate: DigiCertCA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
(certificate Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----

Known issues

The app platform webserver can use only the default http/https ports 80/443.

Database

The app platform creates a database for each app instance with a given password. In the installer there will be used randomly generated passwords. You can set a new database password for every instance in the Application Platform.

The apps should store all data in that database. That makes sure that a consistent backup and restore of app instances can be done by the app platform manager. In hosted scenarios, having separate databases for each instance also makes sure that the data of different customers are clearly separated and can easily be moved from one physical platform to another.

The default configuration decline database request from extern. If you need external access you can change the PGSQL configuration in the file /mnt/sda2/pgsql/pg_hba.conf. (Please think about it before you do it, because a better way is to create your own local app with local database access.)

App Platform Manager

The App Platform Manager is the central component of the App Platform. It does the following:

  • Installing app services by downloading the binaries from an app store.
  • Running and monitoring app services. If an app service crashes it is restarted, automatically.
  • Management of app instances and providing them with the environment they need:
    • A database
    • A webserver path
    • A password for authentication
  • Backup and restore of app instances.
  • Collecting debug information like tracing and crash dumps.
  • System monitoring (CPU usage, memory usage, etc).

App Services

App services are runned by the App Platform Manager. They implement an interface that is used by the manager to start, stop and configure app instances. Each service runs in a separate child process of the manager.

App Instances

The actual functionality of an app service is provided by app instances. They run in the same process as the app service but have a distinct webserver path and their own database. There can be 0..n instances of an app service. Instances can optionally host (web) apps that can be opened in the myApps client.

Each instance of an App service has two passwords. The instance password itself and a database password. If you want to change it you must update the password on both sides.
The instance password must match to the password in the corresponding PBX App objects, while one instance can have different App objects.
The database password must be just known to the manager and not outside of the App Platform.

Relationship between app instances and app objects in the PBX

Typically an app instance is connected to one or more app objects in a customer PBX. This is done by configuring the same parameters on both sides:

  • URL
  • Password

The password is used by the PBX for authenticating itself, users and services against the app instance.

Some apps need a websocket connection with the PBX. When "websocket" is activated at the app object, the PBX establishes a websocket connection to the app instance and provides the APIs that are configured at the app object.

Supported scenarios

It is important to understand that the concept does not do any assumptions on how PBXes and APs are correlated. So you can have

  • One AP for one customer
  • One AP for many customers
  • Many APs for one customer
  • Many APs for many customers

Attention: The V13 installer can only configure the scenario "One AP for one customer". If you want to have a different scenario, you have to configure it manually.

For hosting or cloud scenarios you need special scenarios. Please refer our V13 Hosting instructions.

Restrictions

Currently we don't have redundancy for app instances or APs.

Update of the App Platform itself

The App Platform is build on top of buildroot and will receive updates and fixes from time to time.
You can update the used build inside the Manager App by using the Update button at the top.

It is strongly advised to make a full backup (VM) or at least backup all apps (Gateway) before you run such an update!

Tracing

Each App Service has its own log file, which can be accessed through the Manager App. You can configure a log file size for each App Service.
Each App Intance has its own trace flags. The following trace flags can be set:

  • Alarm client: used by the manager to send alarms to an alarm server
  • App: logs from the App Service itself
  • App WebSocket: logs app websocket connections (e.g. from PBX objects to an App Service or from the UI to the App Service)
  • AppSharing: just native clients
  • AppProxy: just native clients, logs requests which are proxied between the local webserver and the remote server
  • Audio: just native clients
  • Browser: just native clients
  • Command: the command interface is used to execute shell commands, e.g. used by the manager App
  • Config: logs config changes of an App
  • Database: database logs
  • DB files: database file logs
  • DNS: DNS request logging
  • DTLS: just native clients, DTLS request logging
  • Ethernet: interface to get ethernet adapater infos, just manager App
  • File: logs for file system access (synchronous), e.g. manager App
  • Files: logs for file system access (asynchronous)
  • HTTP client: http client logs
  • HTTP file: logs for static HTTP files
  • ICE: just native clients
  • LDS: local domain sockets
  • Media: just native clients
  • Media channel: just native clients
  • Process: IProcess interface logs which is used for spawning, killing processes etc.
  • SMTP: SMTP client logs
  • TCP: TCP logs
  • Time: ITime interface logs
  • TLS: TLS logs
  • TURN: just native clients
  • UDP: UDP logs
  • Video: just native clients
  • WebSocket client: logs outgoing websocket connections
  • Webserver traffic: logs incoming HTTP traffic, which is forwarded from the webserver to the App
  • WebDAV service: logs WebDAV requests to the App
  • Webserver: enables webserver specific logs

RPCAP

If you open the Manager App, click on the Manager in the left list and then on the Diagnostics button, you can enable RPCAP.

Please don't forget to disable RPCAP after your testing!

How-Tos

How to retrieve files from the AP

To retrieve files from the AP which can not be retrieved via the AP manager UI, you can connect to the AP using the SCP protocol on port 22.

For example, in WinSCP

  • use SCP as protocol (NB: WebDAV is not supported on most of the directories on the AP)
  • use the DNS name or IP address of your AP (not the PBX)
  • use user admin and the appropriate password (ipapps by default)
  • use /home/admin as start directory

You can now log-in to the AP using putty and copy files to retrieve to /home/admin (e.g. from /var/log/apps/manager/...).

App Platform/Apps app not online anymore due to full disk

If the apps app is not online anymore and you can't access any apps anymore, try to login with an SSH client to see if your disk is full. Login as admin and afterwards as root (su root).

  • issue df -h and see the disk usage of /dev/sda2, if this is 100%, your disk is too full
  • stop the manager
    • /etc/init.d/S92manager stop
  • empty the 500 MB file, which is exactly for this case here (manager must be 13r1 SR9 or higher to have this file)
    • echo "" > /mnt/sda2/empty_if_no_space
  • delete log files to recover some space
    • rm /var/log/apps/*/*
    • rm /var/log/core_dumps/*/*
  • restart the postgresql server (see the output if this worked or not)
    • /etc/init.d/S50postgresql restart
  • restart the manager
    • /etc/init.d/S92manager restart
  • wait until everything is online again
  • open the Apps app and try to find the instance which uses the most disk space and try to delete files/content from it
    • you may want to stop all app services first to prevent more writes to the database
    • if not possible, you can delete this instance, but you'll loose all data from this instance then!
  • after that you can try Reference13r1:Concept_App_Platform#Shrink_the_physically_size_of_PostgreSQL_database_files to free up space
    • If this does not work you can create a backup from the database, delete the database and import the database again.
  • free up at least 500 MB so that the manager can create the file again
  • delete /mnt/sda2/empty_if_no_space if you are done and restart the manager:
    • rm /mnt/sda2/empty_if_no_space
    • /etc/init.d/S50postgresql restart

If all of this doesn't help, you can resize the file system on a VM:

Resizing the disk of a Virtual machine

  • stop the VM
  • expand the disk using your VM utilities
  • start the VM and boot from the first boot entry rescue/setup
  • login with root and iplinux
  • call /home/root/install_step1.sh log.txt resize
  • the VM reboots automatically after a successful resize

Shrink the physically size of PostgreSQL database files

Tuples that are deleted in your databse are not physically removed from the database-file. So the claimed space on the harddisk is still in use after the delete operation. If you need to free up some disk space you can force to reorganize the physically database-file on your harddisk.

Important: You are operating on the Database, you have to make a Backup of your Database before you do this!

Login via SSH to the APPlatform with the admin User

su root
sudo -u postgres reindexdb -a
sudo -u postgres vacuumdb -a -f

Note: This may take some time and CDRs (or other data written to a DB) won't be received during this time.

After the process you can check the free dispace via df -h. You can check the claimed space from the database file with the command du -sh /mnt/sda2/pgsql/.

Change IP Addresses / DNS Names / System Name

If you want to change the System Name or the DNS Name of the PBX and/or AP Platform you must change records manually in the described order! You have to know the Admin password to directly Login to the AP-Plattform.

App Platform

Settings - General
  • Devices app URL
  • App platform DNS name
Settings - Alarms and Events
  • URL
All Instances
  • Domain
  • Webserver path

PBX

Download the configuration and search/replace in the config file. Upload the config file and reboot.

Manual:

Depending on your change you have to activate/deactivate the Setting Operation without DNS

Additional Devices / Steps

  • Devices Registration URL
  • Alarm server
  • Reverse Proxy configuration
  • Change Domain Name in Devices if you have also changed the system name

How to recover from a broken File System

Sometimes you may find messages in the messages log file (in var/log) like

initial error at 1500329378: ext4_journal_start_sb:328
last error at 1500329378: ext4_journal_start_sb:328

Or you get events like "Broken file system" from your AP.

This indicates a file system failure on the Linux Installation.

When the Linux file system is broken, you can try to repair it using some command line Linux tools.

If this doesn't fix your issue, you need to replace the SSD with a new one, re-install the App Platform and any applications and restore your backups.

Gateway

  • Open the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/sda3 to root=/dev/ram0
    • modify the Initrd file to ramdisk.ext2.xz
    • modify the ramdisk size to 100000
    • start Linux again
This will run Linux on another (hopefully sane) partition.
  • use putty to log in to the LAP's command line (user: root, pw: iplinux)
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • go back to the WebGUI of the gateway running your LAP and proceed to App Platform/General
    • terminate Linux (Status/Stop)
    • modify the Kernel command line from root=/dev/ram0 to root=/dev/sda3
    • clear the Initrd file field
    • clear the ramdisk size field
    • start Linux again
This will run Linux on the original partition.

VM

Restart the VM and select the first entry in the boot menu from grub.

  • use putty to log in to the LAP's command line
    • on the command prompt, use e2fsck -p -f /dev/sda2
    • on the command prompt, use e2fsck -p -f /dev/sda3
this should fix any issue on the file system
  • Reboot your VM