13. Software Upgrade

13.1. Release Notes
13.2. Preparing to a Software Upgrade
13.3. Upgrading the sip:provider PRO
13.3.1. Preparing for maintenance mode
13.3.2. Switch to new repositories
13.3.3. Upgrade the first PRO node
13.3.4. The customtt files handling (if necessary)
13.3.5. Promote the upgraded standby node to active
13.3.6. Upgrade the second PRO node
13.4. Post-upgrade tasks
13.4.1. Disabling maintenance mode
13.4.2. Post-upgrade checks
13.5. Applying the Latest Hotfixes
13.5.1. Update the approx cache on the standby node
13.5.2. Apply hotfixes on the standby node
13.5.3. Promote the standby node to active
13.5.4. Apply hotfixes on the second node

13.1. Release Notes

The sip:provider PRO version mr6.1.2 has the following important changes:

  • Add timezone support to exported CDRs and time-based Call-Forwards [TT#27462, TT#24906]
  • [CloudPBX] Add support to skip hunt-group members being on a call during call hunting [TT#6272]
  • Improve Session Timer handling by adding a "transparent" mode and gracefully handle endpoints without SST support [TT#29931, TT#29926]
  • Improve ngcp-panel memory usage and performance [TT#30507]
  • Add mp3 and ogg formats support for fetching voicemail recordings via API

Please find the complete changelog in our release notes on our WEB site.

13.2. Preparing to a Software Upgrade

warning

Make sure that all the SIP domains and peering servers have the appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, incorrect network interface may be used for sending and receiving RTP traffic after the software upgrade.

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause service downtime, so it is safe to execute them in peak hours.

Check the overall system status:

ngcp-status --all

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary):

ngcp-status --integrity

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

Find local changes to the template files by executing:

ngcp-customtt-diff-helper

ngcp-customtt-diff-helper can help you to download the templates for you target release. To do so, you can run the following command:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script you can, now, merge the current customtt with the new tt2, creating the new customtt.tt2 files. Once did that, you can TAR the new customtts in a tar.gz file and you can use that after the upgrade to deploy the new customtt.

ngcp-customtt-diff-helper -t

You can check further details regarding the ngcp-customtt-diff-help with the option "-h".

Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated *.customtt.tt2) before the software upgrade.

Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Check/clean old dpkg backup files.

Make sure that the list is empty before you continue:

find /etc/ngcp-config -name \*.tt2.dpkg\*

Log in to both servers. Use their real IP addresses so you can switch over between the nodes later on.

Make sure the cluster status is OK: on both nodes execute:

  • monit summary - one should be running all services, the other all but rtpengine, lb, proxy, sbc, mediator and rate-o-mat
  • ngcp-check-active -v - one (with all services running) should print "active", the other one - "none"
  • mysql -e "show slave status\G" - make sure you see the following lines:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
  • ngcpcfg status - should print OK on both nodes
  • ngcp-collective-check - should not report any problems.

Test the cluster failover to see if everything works fine on the second node as well. On the standby node execute:

ngcp-make-active

Afterwards, check monit once again: ngcp-check-active and ngcp-collective-check.

Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.

Run "apt-get update" and ensure that you have no warnings/errors in the output.

warning

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

13.3. Upgrading the sip:provider PRO

Make sure you are prepared to spend about two hours upgrading the system. Note that a short service downtime is possible during the services switchover to the upgraded node.

The sip:provider PRO software upgrade to mr6.1.2 procedure will perform several fundamental tasks:

  • Upgrade the NGCP software packages
  • Upgrade the NGCP configuration templates
  • Upgrade the NGCP DB schema
  • Upgrade the NGCP configuration schema
  • Upgrade the base system within Debian 9 (stretch) to the latest package versions

Assuming that you have a running sip:provider PRO system and want to upgrade it, start with the software upgrade on the standby node. Then switch the services over to the upgraded node and upgrade the other (now standby) node, as described in the steps below.

13.3.1. Preparing for maintenance mode

Sipwise NGCP introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of NGCP will disable some background services (for instance: mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Pull pending configuration (if any):
ngcpcfg pull
  • Enable maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
  • Apply configuration changes by executing:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr6.1.2'
ngcpcfg push all

13.3.2. Switch to new repositories

To upgrade the sip:provider PRO to the latest mr6.1.2 release, execute the following commands on both nodes:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr6.1.2/" /etc/apt/sources.list.d/sipwise.list

ngcp-approx-cache-helper --auto --node localhost

apt-get update
apt-get install ngcp-upgrade-pro
warning

Do not use "ngcpcfg apply/build" after executing the steps from the above section, as otherwise the changes will be overwritten and you will have to redo these steps.

13.3.3. Upgrade the first PRO node

Execute ngcp-upgrade on the standby node as root:

ngcp-upgrade
info

sip:provider PRO can be upgraded to mr6.1.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one.

info

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, just execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

13.3.4. The customtt files handling (if necessary)

Merge/add the custom configuration templates if needed. Apply the changes to configuration templates and send them to the shared storage and the other node:

ngcpcfg apply 'The first node upgraded'
ngcpcfg push --nobuild --noapply

13.3.5. Promote the upgraded standby node to active

Execute on the current standby node as root:

ngcp-make-active

13.3.6. Upgrade the second PRO node

Go to the new standby node. Run ngcp-upgrade as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

13.4. Post-upgrade tasks

13.4.1. Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Pull outstanding ngcpcfg changes (if any):
ngcpcfg pull
  • Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr6.1.2'
ngcpcfg push all

13.4.2. Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr6.1.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr6.1.2-*/upgrade.log.

13.5. Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest appling the latest hotfixes and critical bug fixes.

Execute all steps as described in Section 13.2, “Preparing to a Software Upgrade”. They include the system checks, customtt handling and others. It is important to execute all the steps from the above chapter.

13.5.1. Update the approx cache on the standby node

The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.

ngcp-approx-cache-helper --auto --node localhost

13.5.2. Apply hotfixes on the standby node

ngcp-update

13.5.3. Promote the standby node to active

Execute on the standby node as root:

ngcp-make-active

Check in a minute that the node became active:

ngcp-check-active

13.5.4. Apply hotfixes on the second node

ngcp-update

Execute the final checks as described in the Post-upgrade checks section.