15. Software Upgrade

15.1. Release Notes

The Sipwise C5 version mr9.3.1 has the following important changes:

  • Add support for AEAD-AES-GCM SRTP encryption (RFC 7714, for both SDES and DTLS). [TT#108551]
  • New emergency_location_format, emergency_location_object and emergency_provider_info domain’s preferences. The configuration of those params is described in the dedicated documentation at Emergency Geo-location Formats. [TT#54608]
  • New no_404_fallback subscriber’s preference to deactivate the fallback due to extended matching or e164_to_ruri modes. [TT#62051]
  • Add new method to access the admin voicebox: calling voiceboxpin it will asked you to enter the password/pin but not your extension number [TT#110943]
  • Add kamailio.proxy.rtpengine_bleg config.yml preference. The preference will activate RTPengine on the outgoing leg of the call (b-leg) even for cases where it is not strictly necessary like in no-pbx cases [TT#89560]
  • [Pro/Carrier] Add kamailio.proxy.skip_pbx_loop config.yml preference. It allows to skip the PBX loop for calls from/to subscribers of cloudpbx customers. The preference is at the moment kept disabled because still in experimental stage.
  • [Pro/Carrier] New send_sip_message TPCC command to send aribitrary SIP messages to devices. At the moment only NOTICE and INFO are supported. TPCC module is required. [TT#81003]

Some important technical points for those interested:

  • Change SRTP crypto suites order to prefer stronger crypto suites over weaker ones. [TT#108552]
  • Use SHA-256 DTLS fingerprint by default instead of SHA-1. SHA-1 is still available for use but must be configured explicitly. [TT#108552]
  • The monit service entries for MariaDB have been renamed from mysql and mysql_cluster to mariadb and mariadb-cluster. [TT#108502]
  • [PRO/Carrier] Stop installing PBX firmware by default. The CLI script ngcp-insert-pbx-devices downloads missing firmwares automatically. [TT#97751]
  • [Pro/Carrier] Call features like Parking, Pickup and Music on Hold have been moved from SEMS ngcp_pbx profile to ngcp base profile. [TT#89555]
  • [Pro/Carrier] In cases where PBX subscribers are involved, the second activation of RTPengine has been moved from before SEMS b2b to the outgoing leg of the call (b-leg). This to ensure a better control on the outgoing sdp. [TT#89560]

Please find the complete changelog in our release notes on our WEB site.

15.2. Overview

The Sipwise C5 software upgrade procedure to mr9.3.1 will perform several fundamental tasks:

  • upgrade the NGCP software packages
  • upgrade the NGCP configuration templates
  • upgrade the NGCP DB schema
  • upgrade the NGCP configuration schema
  • upgrade the base system within Debian 10 (buster) to the latest package versions

15.3. Pre-upgrade checks

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

15.3.1. Log into the C5 server

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

15.3.2. Check the overall system status

Check the overall system status:

ngcp-status

15.3.3. Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

warning

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

15.3.4. Check system integrity

Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt/patchtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated customtt.tt2 or patchtt.tt2) before the software upgrade.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.

Check and remove dpkg files left from previous software upgrades.

Make sure that the list is empty before you continue:

find /etc/ngcp-config -name \*.tt2.dpkg\*

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

15.3.5. Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Run "apt-get update" and ensure that you do not have any warnings and errors in the output.

warning

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

15.4. Pre-upgrade steps

info

Sipwise C5 can be upgraded to mr9.3.1 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and makes it possible to choose the proper one.

info

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

15.4.1. ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).
  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or continue without stop until the end
    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end
  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

15.4.2. Preparing for maintenance mode

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Enable maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
  • Apply configuration changes by executing:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr9.3.1'

15.4.3. Set the proper software repositories

warning

Ensure you are using the Sipwise APT repositories.

Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

15.5. Upgrading Sipwise C5 CE

Execute the following commands as root:

ngcp-prepare-upgrade mr9.3.1
ngcp-upgrade

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

15.6. Post-upgrade steps

15.6.1. Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr9.3.1'

15.6.2. Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr9.3.1-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/backup/ngcp-mr9.3.1-*/upgrade.log.

15.7. Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.

Execute all steps as described in Section 15.3, “Pre-upgrade checks”. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

15.7.1. Apply hotfixes

ngcp-update

15.7.2. Recheck or update the custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Execute the final checks as described in the Post-upgrade checks section.