14. Software Upgrade

14.1. Release Notes
14.2. Upgrade from previous versions to mr5.5.3
14.2.1. Set the proper software repositories
14.2.2. Preparing for maintenance mode
14.2.3. Switch to new repositories
14.2.4. Upgrade the sip:provider CE
14.3. Post-upgrade tasks
14.3.1. Disabling maintenance mode
14.3.2. Post-upgrade checks

14.1. Release Notes

The sip:provider CE version mr5.5.3 has the following important changes:

  • Upgrade Debian from Debian 8 (jessie) to Debian 9 (stretch). SysV is still the init system in use. [TT#21905]
  • Migration from MariaDB 10.0 to MariaDB 10.1 [TT#21824]
  • [PRO/Carrier] Upgrade GlusterFS from 3.8.4 to 3.12.1 [TT#21896]
  • [PRO/Carrier] Remove now unused ngcp-glusterfs-config package. [TT#23433]
  • The monitoring backend has been migrated away completely from Redis to InfluxDB [TT#18250]
  • [PRO/Carrier] ngcp-collective-check will now check the free swap space. [TT#22752]
  • [PRO/Carrier] The ngcp-snmp-agent supports logging to syslog. [TT#21756]
  • Add ability to configure supported SSL protocols/ciphers in config.yml [TT#12195]
  • [PRO/Carrier] Discontinued and removed sems-app and sems-ha (sems-pbx takes over their tasks) [TT#22069] [TT#22070]
  • [PRO/Carrier] Presence event as-feature-event is now supported in kamailio with optional CloudPBX module [TT#18851]
  • [PRO/Carrier] Extension dialing within Auto-Attendant is now possible with CloudPBX module [TT#18736]
  • Implemented request and response processing runtime logging in proxy [TT#22659]
  • Updated asterisk to 13.14.1 [TT#20081]
  • Updated kamailio to 4.4.6 [TT#18950]
  • Implemented ability to check History-Info against allowed_clis or aliases per preference [TT#19559]
  • [PRO/Carrier] Add support for the new HTTP/2 and JWT based protocol for sending push messages to Apple/iOS devices (APNs) [TT#14952]
  • [PRO/Carrier] Different SIP domains can now be configured with distinct authentication data for sending push messages (Google GCM and Apple/iOS APNs) [TT#18841]
  • Add support for dynamically adding iptables firewall rules for RTP proxy media ports as they are opened and closed [TT#19350]
  • [PRO/Carrier] Add license management and enforcement system [TT#23200]
  • Optionally send anonymous usage statistics [TT#23200]
  • Adressed an issue in ngcp-credit-warning where it reported only 10 customers [TT#10964]
  • Adressed an issue with reseller termination via the API and his related contract remained active [TT#21106]
  • Adressed issues with customer and susbcriber lock level settings [TT#21271, TT#14285, TT#19656, TT#18765]
  • Addressed an issue with the fraud notification script not enlisting all the automatically locked subscribers [TT#18755]
  • Improved NGCP Panel UI and the API performance [TT#22827]
  • Added partitioning support for accounting.cdr [TT#3668]
  • Optimised performance, addressed several minor issues and introduced partitioning support in the acc-cleanup tool [TT#3668]
  • Added a "Manager Secretary" feature, currently only supports Polycom phones [TT#20302]
  • Addressed several minor issues in the rewrite rules data representation [TT#23426]
  • Added POST method support in the API to generate invoices [TT#17848]
  • libmyodbc is no longer supported in Debian and replaced by the MariaDB odbc connector library [TT#21307]
  • [PRO/Carrier] - Addressed an issue with reseller manipulation and the rtcengine provisioning when the rtcengine access was not configured [TT#20690]
  • [PRO/Carrier] Added Polycom phones provisioning support [TT#7515]
  • [PRO/Carrier] Improved Yealink phones provisioning support [TT#18335]
  • [PRO/Carrier] Improved error handling for the SMS delivery [TT#20095]
  • [PRO/Carrier] Improved Glusterfs server ngcp volume control [TT#22435]
  • [PRO/Carrier] Improved ngcp-sync-db to handle large blobs of data [TT#22806]
  • Performance tuning:

    • Disable Transparent Huge Pages (THP) support [TT#22160] (Redis performance tuning)
    • Linux sysctl changes:

      • net.core.somaxconn = 512 [TT#22160] (Redis performance tuning)
      • vm.overcommit_memory = 1 [TT#22160] (Redis performance tuning)
      • vm.swappiness = 10 [TT#22407] (prevent unnecessary swapping with available RAM)
      • sunrpc.min_resvport = 700 [TT#22658] (prevent heartbeat and NFS ports collisions)

Please find the complete changelog in our release notes on our WEB site.

14.2. Upgrade from previous versions to mr5.5.3

warning

Debian 9 (stretch) provides OpenSSH 7 with the ssh-dss (DSA) public key algorithm disabled (see more information here). Please ensure you are NOT using ssh-dss (DSA) based SSH keys, otherwise migrate to modern ssh-ed25519 (ED25519) algorithm BEFORE the upgrade.

The sip:provider CE system upgrade to mr5.5.3 will be performed in several steps:

  • Verify APT source lists
  • Upgrade the base system from Debian 8 (jessie) to Debian 9 (stretch)
  • Upgrade the NGCP software packages
  • Upgrade the NGCP configuration templates
  • Upgrade the NGCP DB schema
  • Upgrade the NGCP configuration schema
  • Upgrade the base system within Debian 9 (stretch) to the latest package versions

14.2.1. Set the proper software repositories

warning

Ensure you are using the Sipwise APT repositories. Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

These commands are for systems currently running previous LTS release mr4.5.*

Execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
# Debian repositories, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr5.5.3
deb https://debian.sipwise.com/debian/ jessie main contrib non-free
#deb-src https://debian.sipwise.com/debian/ jessie main contrib non-free
#
deb https://debian.sipwise.com/debian-security/ jessie-security main contrib non-free
#deb-src https://debian.sipwise.com/debian-security/ jessie-security main contrib non-free
#
deb https://debian.sipwise.com/debian/ jessie-updates main contrib non-free
#deb-src https://debian.sipwise.com/debian/ jessie-updates main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE
# Sipwise repository, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr5.5.3
deb https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ jessie main
#deb-src https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ jessie main
EOF

These commands are for systems currently running previous build mr5.5.*

Execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
# Debian repositories, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr5.5.3
deb https://debian.sipwise.com/debian/ stretch main contrib non-free
#deb-src https://debian.sipwise.com/debian/ stretch main contrib non-free
#
deb https://debian.sipwise.com/debian-security/ stretch-security main contrib non-free
#deb-src https://debian.sipwise.com/debian-security/ stretch-security main contrib non-free
#
deb https://debian.sipwise.com/debian/ stretch-updates main contrib non-free
#deb-src https://debian.sipwise.com/debian/ stretch-updates main contrib non-free

deb https://debian.sipwise.com/debian-debug/ stretch-debug main contrib non-free
#deb-src https://debian.sipwise.com/debian-debug/ stretch-debug main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE
# Sipwise repository, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr5.5.3
deb https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ stretch main
#deb-src https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ stretch main
EOF
warning

Do not use "ngcpcfg apply/build" after executing the steps from the above block, as otherwise the changes will be overwritten and you will have to redo these steps.

Run "apt-get update" and ensure you have no warnings/errors here.

14.2.2. Preparing for maintenance mode

Sipwise NGCP introduced Maintenance Mode with its mr5.4.1 release. The maintenance mode of NGCP will disable some background services (for instance, mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

important

For sip:provider CE systems running NGCP release older than mr5.4.1: As upgrading from an earlier release to mr5.4.1 or later will result in a system being aware and making use of the maintenance mode, it is necessary to prepare the NGCP for it before the software upgrade.

Enable maintenance mode:

  • Edit /etc/ngcp-config/config.yml file:

    • For systems running NGCP release older than mr5.4.1: insert a new line: "maintenance: yes" in the general section of the configuration file
    • For systems running NGCP release mr5.4.1 or later: set the general.maintenance parameter to yes
general:
  maintenance: yes

Disabling background services. These tasks are for systems currently running NGCP release older than mr5.4.1. The maintenance mode of NGCP is not available in your system prior to the software upgrade. It is therefore necessary to manually disable the following background services to avoid a potential system inconsistency:

  • Mediator
  • Rate-o-mat
  • CDR Exporter

Execute the following:

  1. Stop mediator and rate-o-mat services:

    service mediator stop
    service ngcp-rate-o-mat stop
  2. Prevent background services from (re)starting. You need to change configuration in the /etc/ngcp-config/config.yml file: set the enable parameter to "no":

    mediator:
      enabled: no
    ...
    rateomat:
      enable: no
  3. In order to disable the CDR exporter process, add a new custom template file because there is no such file by default:

    config="/etc/cron.d/ngcp-cdr-exporter"
    customtt="/etc/ngcp-config/templates/etc/cron.d/ngcp-cdr-exporter.customtt.tt2"
    [ -f "${customtt}" ] || cp "${config}" "${customtt}"
    vim "${customtt}"

    Comment out this single line in the file:

    # 25,55 * * * *   root    . /etc/default/ngcp-roles; if ...

Please don’t forget to reverse the manual changes after the software upgrade as described in Enabling background services section of the handbook!

For all NGCP systems, regardless of their version:

Apply configuration changes by executing :

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr5.5.3'

14.2.3. Switch to new repositories

To upgrade the sip:provider CE to release mr5.5.3, execute the following commands:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.5.3/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/jessie/stretch/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
grep -q "debian-debug" /etc/apt/sources.list.d/debian.list || \
  echo "deb https://debian.sipwise.com/debian-debug/ stretch-debug main contrib non-free" >> /etc/apt/sources.list.d/debian.list

apt-get update
apt-get install ngcp-upgrade-ce

14.2.4. Upgrade the sip:provider CE

Run the upgrade script as root like this:

ngcp-upgrade
info

sip:provider CE can be upgraded to mr5.5.3 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one.

info

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, just execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

14.3. Post-upgrade tasks

14.3.1. Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"

Enabling background services. If you upgraded from NGCP release earlier than mr5.4.1, you have to manually reverse the changes you applied before the upgrade in the (Disabling background services) section.

  1. Enable background services:

    ngcpcfg set /etc/ngcp-config/config.yml "mediator.enabled=yes"
    ngcpcfg set /etc/ngcp-config/config.yml "rateomat.enable=yes"
  2. In order to enable the CDR exporter process, remove the corresponding customtt file:

    rm /etc/ngcp-config/templates/etc/cron.d/ngcp-cdr-exporter.customtt.tt2

For all NGCP systems, regardless of their previous version:

  • Execute the command:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr5.5.3'

14.3.2. Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr5.5.3-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr5.5.3-*/upgrade.log.