12. Software Upgrade

12.1. Release Notes

The Sipwise C5 version mr7.4.2 has the following important changes:

  • Add support for DTMF transcoding [TT#52477]
  • MariaDB database encryption support [TT#35652]
  • Add French announcements [TT#46162]
  • [PRO/Carrier] New customer Self-Care: assign SoundSet to PBXGroup or PBXSeat [TT#55911]
  • Update kamailio to version 5.1.8 [TT#57950]

Please find the complete changelog in our release notes on our WEB site.

12.2. Overview

The Sipwise C5 software upgrade procedure to mr7.4.2 will perform several fundamental tasks:

  • upgrade the NGCP software packages
  • upgrade the NGCP configuration templates
  • upgrade the NGCP DB schema
  • upgrade the NGCP configuration schema
  • upgrade the base system within Debian 9 (stretch) to the latest package versions

Sipwise C5 is a PRO-style system that has "A" and "B" sets of nodes with specific roles. The number of nodes can differ between installations and must be clarified before the upgrade at the planning stage.

The software upgrade is usually performed by Sipwise engineers according to the following steps:

  • create the software upgrade plan
  • execute pre-upgrade steps: patchtt, customtt, backups
  • make all "B" nodes active
  • ensure that all "A" nodes are standby
  • perform the software upgrade on all "A" nodes
  • schedule and make services switchover to all "A" nodes
  • ensure that "A" nodes performwell (otherwise, perform a switch back)
  • perform the software upgrade on all "B" nodes
  • perform the system post-upgrade testing and cleanup
warning

The only allowed software upgrade path is the one described above. All the other theoretically possible upgrade scenarios can lead to unpredictable results.

warning

Nodes "A" and "B" MUST be used as described in this document. It is NOT allowed to swap them unless proxy replication (of MySQL on port 3308) is configured on the db01b node.

12.3. Planning a software upgrade

Confirm the following information:

  • which system should be upgraded (LAB/LIVE, country, etc.)
  • the date and time schedule for each of the steps above (keeping the time zone in mind)
  • a confirmed timeframe for the upgrade operation (allowed switchover timeframe)
  • the basic functionality test (BFT) to be executed before the start of the software upgrade and after the switchovers to ensure that the new release does not show critical issues (the BFT scenario should be prepared by the customer engineers)
  • actions to be taken if the software upgrade operation cannot be completed within the defined maintenance window
  • contact persons and ways of communication in case of emergency
  • ensure that the customer and/or Sipwise engineers have access to the virtual consoles of the servers: KVM, iDRAC, AMM

12.4. Preparing the software upgrade

warning

Make sure that all the SIP domains and peering servers have the appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, the incorrect network interface may be used for sending and receiving RTP traffic after the software upgrade.

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

12.4.1. Log into the C5 standby management server (web01a/db01a)

tip

Use the static server IP address so you can switch between the nodes.

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

12.4.2. Check the overall system status

Check the overall system status:

ngcp-status --all

Make sure that the cluster health status is OK: Check the nodes in parallel, using the clish command:

  • ngcp-clish "ngcp version summary" - ensure that all cluster nodes have correct/expected from version
  • ngcp-clish "ngcp version package installed ngcp-ngcp-carrier" - ensure that the metapackages version is equal to the ngcp version above
  • ngcp-clish "ngcp version package check" - ensure that all nodes have the identical Debian package installed
info

Software must be identical on all nodes (before and after the upgrade!)

  • ngcp-clish "ngcp cluster ssh connectivity" - check SSH connectivity from the current node to all other nodes
  • ngcp-clish "ngcp cluster ssh crossconnectivity" - check SSH cross-connectivity from all nodes to all other nodes
  • ngcp-clish "ngcp monit summary" - all required services must be running on corresponding nodes
  • ngcp-clish "ngcp cluster status" - active node(s) (with all services running) must print "all", the other(s) must print "none"
  • ngcp-clish "ngcp status collective-check" - all checks must be OK
  • ngcp-clish "ngcp show date" - date and time must be in sync on all the servers
  • ngcp-clish "ngcp show dns-servers" - ensure that the DNS configuration is consistent among the nodes
info

to exit from ngcp-clish press Ctrl+Z (or type exit):

# ngcp-clish
Entering 'clish-enable' view (press Ctrl+Z to exit)...
# exit
#

12.4.3. Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

Create tickets to Sipwise developers to make relevant custom modifications part of the product in future releases. This allows you to get rid of the customtt files one day.

warning

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

warning

Starting from version mr7.0.1 a new kamailio module called "pv_headers" has been introduced. This new module enables storing all headers in XAVP to freely modify them in the kamailio logic and only apply them once when it’s time for the packet to be routed outside. The main goal of the module is to offload the intermediate header processing into the XAVP dynamic container as well as provide with high-level methods and pseudovariables to simplify SIP message header modifications. The module is enabled by default in kamailio proxy and all the templates have been updated to use this new logic. Before proceeding with the upgrade it is essential that the customtt/patchtt you have in place are updated to this new format. At appendix Appendix H, New kamailio pv_headers module you can find additional information on the module.

12.4.4. Check system integrity

Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt/patchtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated customtt.tt2 or patchtt.tt2) before the software upgrade.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.

Check and remove dpkg files left from previous software upgrades.

Make sure that the list is empty before you continue:

find /etc/ngcp-config -name \*.tt2.dpkg\*

Log into all the servers.

Open separate windows for all the servers inside your "screen" session. (Press Ctrl+a + c to open a new window, Ctrl+a + a or Ctrl+a + [0-9] to change the window. Ctrl+a + " shows the list of all your windows. Use Ctrl+a + A to change the window names to corresponding hosts).

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

12.4.5. Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Check the replication on both central DB servers and on ports 3306 and 3308 of all the proxy servers. Ensure that all the proxy nodes replicate the read-only DB (127.0.0.1:3308) from the db01a node. Otherwise, discuss a special plan to address your particular configuration. The result must always show:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0

Test the cluster failover to see if everything works fine on "B" nodes as well. On all the standby nodes execute:

ngcp-make-active

Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.

Run "apt-get update" on all nodes and ensure that you do not have any warnings and errors in the output.

warning

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

12.5. Upgrading Sipwise C5 CARRIER

Log in to all nodes and execute the checks from Section 12.4, “Preparing the software upgrade” again. This will ensure that nothing was broken since the preparation steps were finished. Also, execute ngcpcfg show and ngcpcfg status to check the latest configuration changes.

Perform the BFT test.

12.5.1. License check

The Sipwise C5 — starting from mr6.5.1 release — enforce software licensing restrictions in form of a regular comparison of the licensed services and capacities against the actual usage patterns of the platform. In case some functionalities are enabled but not licensed, an error in syslog will be reported and the impacted services will be automatically deactivated.

Before proceeding with the upgrade, please take some time to check that all the modules not licensed are actually disabled in config.yml file. To verify if they are enabled execute the following commands:

ngcpcfg values sems.prepaid.enable
ngcpcfg values sems.prepaid.inew.enable
ngcpcfg values pbx.enable
ngcpcfg values pushd.enable
ngcpcfg values intercept.enable
ngcpcfg values voisniff.admin_panel
ngcpcfg values voisniff.daemon.li_x1x2x3.enable
ngcpcfg values voisniff.daemon.start

If the output of one of the commands is yes but the module is not licensed, you have to deactivate it. For example, in case of pre-paid billing module execute:

ngcpcfg set /etc/ngcp-config/config.yml sems.prepaid.enable=no
ngcpcfg apply 'Disable prepaid module'
ngcpcfg push all
warning

Please, pay particular attention to pre-paid billing module because it is enabled by default.

12.5.2. Preparing for maintenance mode

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Pull pending configuration (if any):
ngcpcfg pull
  • Enable maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
  • Apply configuration changes by executing:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr7.4.2'
ngcpcfg push all

To upgrade Sipwise C5 CARRIER to mr7.4.2 release, execute the following commands on the standby management "A" node:

12.5.3. ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).
  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or just before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or just continue without stop until the end
    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end
  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

12.5.4. Upgrading ONLY the first standby management node "A" (web01a/db01a)

info

Sometimes the DB and MGMT roles are assigned to the same host. This is OK.

warning

Do NOT execute the software upgrade on web01a and db01a in parallel!

The main goal of the following commands is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.4.2/" /etc/apt/sources.list.d/sipwise.list

ngcp-approx-cache-helper --auto --node localhost

apt-get update
apt-get install ngcp-upgrade-pro
info

Don’t worry, ngcp-upgrade-carrier does not exist, use ngcp-upgrade-pro as outlined above.

warning

Do not use "ngcpcfg apply/build" after executing the steps from the above section, otherwise the changes will be overwritten and you will have to redo these steps. The same applies to similar sections below.

Run the upgrade script on the standby node as root:

ngcp-upgrade
info

Sipwise C5 can be upgraded to mr7.4.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and makes it possible to choose the proper one.

info

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, just execute ngcp-upgrade again and it will continue from the previous step.

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt for new the release mrX.X on xxx01a'

Send the new templates to the shared storage and the other nodes

ngcpcfg push --nobuild --noapply all
warning

Do NOT execute ngcpcfg push --shared-only at this stage, as it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glusterfs.

12.5.5. Upgrading the standby database node "A" (db*a)

info

If the DB and MGMT roles are assigned to the same host, then skip this step as you have already upgraded the standby MGMT node "A" above.

Run the following commands to upgrade the standby DB node "A" (select the same release version as above and follow the on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade
info

It is important to upgrade db01a node before upgrading any proxy nodes. Otherwise, the "local" MySQL (127.0.0.1:3308) on proxy nodes may become out of sync in case the new release has _not_replicated.up DB statements.

12.5.6. Upgrading other standby nodes "A" (lb*a/prx*a)

Run the below commands selecting the same release version and follow the on-screen recommendations:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade
12.5.6.1. Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades, because it is very likely that the backup was already performed:

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary.

See a more detailed description of the options in: ngcp-upgrade options

12.5.7. Promote ALL standby nodes "A" to active.

warning

Ensure that all standby nodes "A" are: * upgraded to the new release (check /etc/ngcp_version or use ngcp-clish) * have been rebooted (run ngcp-status on each standby node)

On all "A" nodes run:

ngcp-make-active

Ensure that the "A" nodes became active, by executing the 'ngcp-status' and 'ngcp-clish' commands described above.

Ensure that ALL "B" nodes are standby now!

12.5.8. Upgrading ALL standby nodes "B" (web*b/db*b/lb*b/prx*b)

Run the following commands selecting the same release version and following the on-screen recommendations:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade
info

You can upgrade all standby "B" nodes simultaneously (including the ones with the mgmt and db roles).

12.5.8.1. Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades:

  • --step-by-step: confirm before proceeding to next step.
  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db").

See a more detailed description of the options in: ngcp-upgrade options

12.6. Post-upgrade steps

12.6.1. Migrate location entries from Mysql to Redis DB

Starting from mr6.2.1, location, acc and dialogs data are stored in RedisDB allowing better system performaces. Before proceed with the final upgrade steps, check if location data are still stored on MySQL DB:

ngcpcfg values "kamailio.proxy.redis.usrloc"

If the answer is yes, then skip this sub-chapter and proceed with the next one. On the contrary, an answer equals to no means that the migration process has not been completed. This happens because, to be more flexible and to reduce the downtime of the system, only acc and dialogs data have been moved to RedisDB during the upgrade. To proceed with the migration and complete the process, execute the following commands:

  • On the standby management node (web01a/db01a on Carrier) pull outstanding ngcpcfg changes (if any):
ngcpcfg pull
  • Enable location data storage on RedisDB:
ngcpcfg set /etc/ngcp-config/config.yml "kamailio.proxy.redis.usrloc=yes"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Enable location data storage on RedisDB'
  • Migrate all location data from MySQL to Redis DB using an adhoc script:
ngcp-location-migrate -a
  • Push the changes to all the current passive nodes (e.g. "A" nodes):
ngcpcfg push db01a lb01a prx01a prx02a ...
  • On all "A" nodes run (proxy, ngcp-panel and mediator services will start with the new configuration):
ngcp-make-active
  • Push changes to the remaining nodes (e.g. "B" nodes):
ngcpcfg push db01b lb01b prx01b prx02b ...

12.6.2. Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Pull outstanding ngcpcfg changes (if any):
ngcpcfg pull
  • Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr7.4.2'
ngcpcfg push all

12.6.3. Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr7.4.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/backup/ngcp-mr7.4.2-*/upgrade.log.

12.7. Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest appling the latest hotfixes and critical bug fixes.

Execute all steps as described in Section 12.4, “Preparing the software upgrade”. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

It is suggested to promote B-nodes to active and start the update with A-nodes.

12.7.1. Update the approx cache on the standby management node

The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.

ngcp-approx-cache-helper --auto --node localhost

12.7.2. Apply hotfixes on the standby management node

ngcp-update

12.7.3. Recheck or update the custom configuration tempates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Send the new templates to the shared storage and the other nodes.

ngcpcfg push --nobuild --noapply all

12.7.4. Apply hotfixes on all other standby nodes

ngcp-update

12.7.5. Promote the standby nodes to active

Execute on the standby nodes as root:

ngcp-make-active

Check in a minute that the nodes became active:

ngcp-check-active

12.7.6. Apply hotfixes on new standby nodes

ngcp-update

Execute the final checks as described in the Post-upgrade checks section.