Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S ngcp-pre-upgrade-checks

Become root inside your screen session:

sudo -s

Check the overall system status:

ngcp-status --all

Make sure that the cluster health status is OK: Check the nodes in parallel, using the clish command:

# ngcp-clish
Entering 'clish-enable' view (press Ctrl+Z to exit)...
# exit
#

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

Create tickets to Sipwise developers to make relevant custom modifications part of the product in future releases. This allows you to get rid of the customtt files one day.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once this is done, archive the new customtt files to deploy the new templates after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated *.customtt.tt2) before the software upgrade.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.

Check and remove dpkg files left from previous software upgrades.

Make sure that the list is empty before you continue:

find /etc/ngcp-config -name \*.tt2.dpkg\*

Log into all the servers.

Open separate windows for all the servers inside your "screen" session. (Press Ctrl+a + c to open a new window, Ctrl+a + a or Ctrl+a + [0-9] to change the window. Ctrl+a + " shows the list of all your windows. Use Ctrl+a + A to change the window names to corresponding hosts).

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Check the replication on both nodes. The result must always show:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0

Test the cluster failover to see if everything works fine on the second node as well. On the standby node execute:

ngcp-make-active

Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.

Run "apt-get update" on all nodes and ensure that you do not have any warnings and errors in the output.

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

ngcpcfg pull

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr6.2.1'
ngcpcfg push all

To specify the new list of APT data sources, execute the following commands on both nodes:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr6.2.1/" /etc/apt/sources.list.d/sipwise.list

To download the new packages into the approx cache, execute the following command on the standby node. This will ensure that both nodes have identical packages:

ngcp-approx-cache-helper --auto --node localhost

Run the following commands on both nodes to install the package responsible for upgrading C5 to a newer release:

apt-get update
apt-get install ngcp-upgrade-pro

Execute the upgrade script on the standby node as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

Merge/add the custom configuration templates if needed. Apply the changes to configuration templates and send them to the shared storage and the other node:

ngcpcfg apply 'The first node upgraded'
ngcpcfg push --nobuild --noapply

Execute on the current standby node as root:

ngcp-make-active

Go to the new standby node. Run the upgrade script as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

Starting from mr6.2.1, location, acc and dialogs data are stored in RedisDB allowing better system performaces. In order to be more flexible and to reduce the downtime of the system, only acc and dialogs data have been moved to RedisDB during the upgrade. To complete the process with the location data, execute the following commands:

ngcpcfg pull

ngcpcfg set /etc/ngcp-config/config.yml "kamailio.proxy.redis.usrloc=yes"

ngcpcfg apply 'Enable location data storage on RedisDB'

ngcp-location-migrate -a

ngcp-make-active

ngcpcfg push

In order to disable the maintenance mode, do the following:

ngcpcfg pull

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"

ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr6.2.1'
ngcpcfg push all

When everything has finished successfully, check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.

ngcp-approx-cache-helper --auto --node localhost

ngcp-update

Execute on the standby node as root:

ngcp-make-active

Check in a minute that the node became active:

ngcp-check-active

ngcp-update

Execute the final checks as described in the Post-upgrade checks section.