11. Software Upgrade

11.1. Release Notes
11.2. Preparing to a Software Upgrade
11.3. Upgrading the sip:provider PRO
11.3.1. Switch to new repositories
11.3.2. Upgrade the first PRO node
11.3.3. The customtt files handling (if necessary)
11.3.4. Promote inactive node to active.
11.3.5. Upgrade the second PRO node

11.1. Release Notes

The sip:provider PRO version mr5.2.2 has several important changes comparing to the previous release:

  • A preconfigured firewall subsystem was added to secure the NGCP. The firewall whitelists all services vital to NGCP’s operations while blocking all other traffic. After upgrade, the firewall subsystem will be disabled by default to avoid inadvertent self-lockout of the operator during upgrade. The firewall has to be enabled manually after successful upgrade in /etc/ngcp-config/config.yml setting security.firewall.enable: ‘yes’. During upgrade the NGCP configuration framework will prepare a standard rule set ready to be used after successful upgrade. If iptables rules already exist on the system, those will be save to a customtt.tt2 and will persist until custom.tt2 and tt2 are merged. If a third-party firewall system is detected, the upgrade procedure will stop. To resume the upgrade, the situation needs to be consolidated (e.g. by removing the unsupported firewall subsystem and merging existing rules into the NGCP firewall subsystem). Notice: Make sure SSH access is correctly configured in /etc/ngcp-config/config.yml to allow SSH access after activating the firewall. Please read the handbook carefully for further instructions before activating the firewall subsystem. [TT#9717]
  • [PRO/Carrier] The default rotate_days configuration for backuptools was decreased from 7 to 3 days to avoid disk space issues (if the configuration is already less than 7 days it will stay unmodified during upgrades) [TT#9816]
  • sshd: in preparation for the upcoming Debian Stretch release upgrade of the underlying operating system, the protocol version 1 specific settings KeyRegenerationInterval, RSAAuthentication, RhostsRSAAuthentication + ServerKeyBits have been removed from the sshd_config (using their defaults now)
  • Improved NGCP documentation style
  • [CPBX] Implement Yealink CP860 and Grandstream GXW-4008 auto provisioning
  • Migrate NGCP admin’s passwords to bcrypt and drop admin’s ssl client cert from DB, providing an API function to fetch PEM and P12 certificates.
important

Due to migrating to bcypt hashing of admin and reseller passwords both on the admin panel and the API, password authentication via the API will take ~500ms for each request. It is highly advised to use ssl client certificate based authentication instead of passwords on the API for performance reasons!

Please find the complete changelog in our release notes on our WEB site.

11.2. Preparing to a Software Upgrade

warning

Make sure that all the SIP domains and peering servers have appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, then incorrect network interface may be used for sending and receiving RTP media after the software upgrade.

Make sure you’re prepared to spend two hours or so upgrading the system. There can be service interruptions, so also notify the customer and get their approval.

Check the system overal status:

ngcp-status --all

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary):

ngcp-status --integrity

Try to find local changes to the template files by issuing:

find /etc/ngcp-config -name \*customtt.tt2

You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Also, please check/clean old dpkg backup files (just in case if previous person did the previous step not carefully enough). Normally the list should be empty:

find /etc/ngcp-config -name \*.tt2.dpkg\*

You will have to understand why the changes are there and if they are still needed after the upgrade. You should create a ticket in the bugtracker if there isn’t one yet.

Log into the two servers. Use their real IPs so you can switch the cluster forth and back later on. Make sure the cluster status is ok - on both nodes issue:

  • monit summary - one should be running all services, the other all but rtpengine, lb, proxy, sbc, mediator and rate-o-mat
  • cl_status rscstatus - one (with all services running) should print "all", the other "none"
  • mysql -e "show slave status\G" - look for the following:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
  • ngcpcfg status - should print OK all the times
  • ngcp-collective-check - should not report any problems.

A cluster fail-over could be a good idea to see if everything works on the second node too. On the standby node issue:

/usr/share/heartbeat/hb_takeover

Afterwards again check monit, cl_status and ngcp-collective-check.

Create two test subscribers, or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.

Run "apt-get update", ensure you have no warnings/errors here.

warning

Installation may use locally specified mirrors. Discuss with a customer possibility to switch on Sipwise APT repositories (at least for the time of upgrades), the public Debian mirrors may not provide packages for old Releases anymore or be at least outdated!

11.3. Upgrading the sip:provider PRO

The sip:provider PRO system upgrade to mr5.2.2 will perform a couple of fundamental tasks:

  • Upgrade NGCP software packages
  • Upgrade NGCP configuration templates
  • Upgrade NGCP DB schema
  • Upgrade the base system within Debian (v8) to the latest package versions

So assuming you have a running sip:provider PRO system and want to upgrade it, start on the inactive node by upgrading software, then take over from the other node and then upgrade the other (now inactive) node, as detailed in the steps below.

11.3.1. Switch to new repositories

For upgrading the sip:provider PRO to the latest mr5.2.2 release, execute the following commands on both nodes:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.2.2/" /etc/apt/sources.list.d/sipwise.list
ngcp-approx-cache-helper --auto --node localhost
apt-get update
apt-get install ngcp-upgrade-pro

11.3.2. Upgrade the first PRO node

Execute ngcp-upgrade in inactive node as root:

ngcp-upgrade
info

sip:provider PRO can be upgraded to mr5.2.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one.

info

If there is an error during upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem just re-execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

11.3.3. The customtt files handling (if necessary)

Merge/add the custom configuration templates if needed. Apply the changes to configuration templates if any and send them to the shared storage and the other node:

ngcpcfg apply 'upgrade node'
ngcpcfg push --nobuild --noapply

11.3.4. Promote inactive node to active.

Execute on inactive node as root:

/usr/share/heartbeat/hb_takeover

11.3.5. Upgrade the second PRO node

Go to the new inactive node. Run ngcp-upgrade, as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

When all finishes successfully check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr5.2.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr5.2.2-*/upgrade.log.