14. Software Upgrade

14.1. Release Notes

The Sipwise C5 version mr7.0.2 has the following important changes:

  • Preparation for migration to Debian/buster [TT#28953]
  • Migration of proxy config to new kamailio pv_headers module [TT#32007]. For additional details have a look at appendix Appendix D, New kamailio pv_headers module.
  • [PRO] Implement full node restore/reinstall from peer (like Carrier deployment) [TT#44969]
  • Implement always-transcode flag in rtpengine as preference [TT#37690]
  • Implement recordings deletion from the physical NFS mount in the UI/API [TT#40453]
  • Implement ngcp-service sync-state to ensure all necessary components are up and running. ngcpcfg apply executes ngcp-service sync-state at the end. It will stop unnecessary components and start all the necessary ones. [TT#43700]
  • [PRO/Carrier] Add GRML rescue option to PRO/Carrier iPXE boot menu [TT#45807]
  • Update kamailio from 5.1.4 to 5.1.6 [TT#43352]
  • Update influxdb from 1.6.1 to 1.6.4 [TT#46766]
  • Update systemd from 237-3 to 239-7 [TT#47854]
  • Move systemd coredumps from root partition to data partition [TT#46902]
  • Improved fraud check scripts performance [TT#48196]
  • /api/soundssets add support to use existing system sound sets [TT#47569]
  • [PRO/Carrier] Faxserver now supports a simplified numbers rewrite logic by default [TT#44603] Available in the config.yml as faxserver.number_rewrite_mode, where supported values are default and extended
  • [PRO/Carrier] SNMP trap behaviour for OIDs from the Sipwise MIB was fixed to be edge-triggered [TT#49848].

Please find the complete changelog in our release notes on our WEB site.

14.2. Overview

The Sipwise C5 software upgrade procedure to mr7.0.2 will perform several fundamental tasks:

  • upgrade the NGCP software packages
  • upgrade the NGCP configuration templates
  • upgrade the NGCP DB schema
  • upgrade the NGCP configuration schema
  • upgrade the base system within Debian 9 (stretch) to the latest package versions

14.3. Preparing the software upgrade

warning

Make sure that all the SIP domains and peering servers have the appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, the incorrect network interface may be used for sending and receiving RTP traffic after the software upgrade.

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

14.3.1. Log into the C5 server

tip

Use the static server IP address so you can switch between the nodes.

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

14.3.2. Check the overall system status

Check the overall system status:

ngcp-status --all

14.3.3. Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

warning

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once this is done, archive the new customtt files to deploy the new templates after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

warning

Starting from version mr7.0.1 a new kamailio module called "pv_headers" has been introduced. This new module enables storing all headers in XAVP to freely modify them in the kamailio logic and only apply them once when it’s time for the packet to be routed outside. The main goal of the module is to offload the intermediate header processing into the XAVP dynamic container as well as provide with high level methods and pseudovariables to simplify SIP message header modifications. The module is enabled by default in kamailio proxy and all the templates have beed updated to use this new logic. Before proceeding with the upgrade it is important that the customtt you have in place will be updated to this new format. At appendix Appendix D, New kamailio pv_headers module you can find additional information on the module.

14.3.4. Check system integrity

Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated *.customtt.tt2) before the software upgrade.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.

Check and remove dpkg files left from previous software upgrades.

Make sure that the list is empty before you continue:

find /etc/ngcp-config -name \*.tt2.dpkg\*

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

14.3.5. Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Run "apt-get update" and ensure that you do not have any warnings and errors in the output.

warning

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

14.4. Upgrade from previous versions to mr7.0.2

14.4.1. Preparing for maintenance mode

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Enable maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
  • Apply configuration changes by executing:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr7.0.2'

14.4.2. Set the proper software repositories

warning

Ensure you are using the Sipwise APT repositories. Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

Execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
# Debian repositories, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr7.0.2
deb https://debian.sipwise.com/debian/ stretch main contrib non-free
#deb-src https://debian.sipwise.com/debian/ stretch main contrib non-free
#
deb https://debian.sipwise.com/debian-security/ stretch-security main contrib non-free
#deb-src https://debian.sipwise.com/debian-security/ stretch-security main contrib non-free
#
deb https://debian.sipwise.com/debian/ stretch-updates main contrib non-free
#deb-src https://debian.sipwise.com/debian/ stretch-updates main contrib non-free

deb https://debian.sipwise.com/debian-debug/ stretch-debug main contrib non-free
#deb-src https://debian.sipwise.com/debian-debug/ stretch-debug main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE
# Sipwise repository, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr7.0.2
deb https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ stretch main
#deb-src https://deb.sipwise.com/spce/${NGCP_CURRENT_VERSION}/ stretch main
EOF
warning

Do not use "ngcpcfg apply/build" after executing the steps from the above block, as otherwise the changes will be overwritten and you will have to redo these steps. Run "apt-get update" and ensure you have no warnings/errors here.

14.4.3. Switch to new repositories

To upgrade Sipwise C5 to release mr7.0.2, execute the following commands:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.0.2/" /etc/apt/sources.list.d/sipwise.list


apt-get update
apt-get install ngcp-upgrade-ce

14.4.4. Upgrade Sipwise C5

Run the upgrade script as root like this:

ngcp-upgrade
info

Sipwise C5 can be upgraded to mr7.0.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow one to choose the proper one.

info

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, just execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

14.5. Post-upgrade tasks

14.5.1. Migrate location entries from Mysql to Redis DB

Starting from mr6.2.1, location, acc and dialogs data are stored in RedisDB allowing better system performaces. Before proceed with the final upgrade steps, check if location data are still stored on MySQL DB:

ngcpcfg values "kamailio.proxy.redis.usrloc"

If the answer is yes, then skip this sub-chapter and proceed with the next one. On the contrary, an answer equals to no means that the migration process has not been completed. This happens because, to be more flexible and to reduce the downtime of the system, only acc and dialogs data have been moved to RedisDB during the upgrade. To proceed with the migration and complete the process, execute the following commands:

  • Enable location data storage on RedisDB:
ngcpcfg set /etc/ngcp-config/config.yml "kamailio.proxy.redis.usrloc=yes"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Enable location data storage on RedisDB'
  • Migrate all location data from MySQL to Redis DB using an adhoc script:
ngcp-location-migrate -a
  • Restart kamailio proxy service to load migrated location data
ngcp-service proxy restart

14.5.2. Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
  • Apply the changes to configuration templates:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr7.0.2'

14.5.3. Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

info

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr7.0.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/backup/ngcp-mr7.0.2-*/upgrade.log.

14.6. Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest appling the latest hotfixes and critical bug fixes.

Execute all steps as described in Section 14.3, “Preparing the software upgrade”. They include the system checks, customtt preparation and others. It is important to execute all the steps from the above chapter.

14.6.1. Apply hotfixes

ngcp-update

14.6.2. Recheck or update the custom configuration tempates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'applying customtt after installing the latest packages'

Execute the final checks as described in the Post-upgrade checks section.