ngcp-upgrade optionsThe sip:provider PRO version mr5.5.10 has the following important changes:
Performance tuning:
Linux sysctl changes:
Please find the complete changelog in our release notes on our WEB site.
| warning | |
Debian 9 (stretch) provides OpenSSH 7 with the ssh-dss (DSA) public key algorithm disabled (see more information here). Please ensure you are NOT using ssh-dss (DSA) based SSH keys, otherwise migrate to modern ssh-ed25519 (ED25519) algorithm BEFORE the upgrade. |
| warning | |
The license key for the sip:provider PRO must be installed and applied on all the servers before the software upgrade. Please contact the Sipwise support team to receive the license key. |
Specify the provided license key in /etc/ngcp-config/config.yml
general: license_key: XXX_XXX_XXX
| warning | |
Make sure that all the SIP domains and peering servers have the appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, incorrect network interface may be used for sending and receiving RTP traffic after the software upgrade. |
It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause service downtime, so it is safe to execute them in peak hours.
Check the overall system status:
ngcp-status --all
Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary):
ngcp-status --integrity
For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).
Find local changes to the template files by executing:
ngcp-customtt-diff-helper
ngcp-customtt-diff-helper can help you to download the templates for you target release. To do so, you can run the following command:
ngcp-customtt-diff-helper -d
In the tmp folder provided by the script you can, now, merge the current customtt with the new tt2, creating the new customtt.tt2 files. Once did that, you can TAR the new customtts in a tar.gz file and you can use that after the upgrade to deploy the new customtt.
ngcp-customtt-diff-helper -t
You can check further details regarding the ngcp-customtt-diff-help with the option "-h".
Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated *.customtt.tt2) before the software upgrade.
Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.
find /etc/ngcp-config -name \*.tt2.dpkg-dist
Check/clean old dpkg backup files.
Make sure that the list is empty before you continue:
find /etc/ngcp-config -name \*.tt2.dpkg\*
Log in to both servers. Use their real IP addresses so you can switch over between the nodes later on.
Make sure the cluster status is OK: on both nodes execute:
Slave_IO_Running: Yes Slave_SQL_Running: Yes Seconds_Behind_Master: 0
Test the cluster failover to see if everything works fine on the second node as well. On the standby node execute:
ngcp-make-active
Afterwards, check monit once again: ngcp-check-active and ngcp-collective-check.
Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.
Run "apt-get update" and ensure that you have no warnings/errors in the output.
| warning | |
If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones! |
Make sure you are prepared to spend about two hours upgrading the system. Note that a short service downtime is possible during the services switchover to the upgraded node.
The sip:provider PRO software upgrade to mr5.5.10 procedure will perform several fundamental tasks:
Assuming that you have a running sip:provider PRO system and want to upgrade it, start with the software upgrade on the standby node. Then switch the services over to the upgraded node and upgrade the other (now standby) node, as described in the steps below.
Sipwise NGCP introduced Maintenance Mode with its mr5.4.1 release. The maintenance mode of NGCP will disable some background services (for instance, mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.
| important | |
For sip:provider PRO systems running NGCP release older than mr5.4.1: As upgrading from an earlier release to mr5.4.1 or later will result in a system being aware and making use of the maintenance mode, it is necessary to prepare the NGCP for it before the software upgrade. |
Enable maintenance mode:
ngcpcfg pull
Edit /etc/ngcp-config/config.yml file:
maintenance: yes" in the general section of the configuration file
general.maintenance
parameter to yes
general: maintenance: yes
Disabling background services. These tasks are for systems currently running NGCP release older than mr5.4.1. The maintenance mode of NGCP is not available in your system prior to the software upgrade. It is therefore necessary to manually disable the following background services to avoid a potential system inconsistency:
Execute the following:
Stop HB watchdog on all nodes:
monit stop hb_watchdog
Stop mediator and rate-o-mat services on both nodes :
monit stop mediator monit stop rate-o-mat
Prevent background services from (re)starting. You need to change
configuration in the /etc/ngcp-config/config.yml file: set the enable
parameter to "no":
heartbeat:
hb_watchdog:
enable: no
...
mediator:
enabled: no
...
rateomat:
enable: noIn order to disable the CDR exporter process, add a new custom template file because there is no such file by default:
config="/etc/cron.d/ngcp-cdr-exporter"
customtt="/etc/ngcp-config/templates/etc/cron.d/ngcp-cdr-exporter.customtt.tt2"
[ -f "${customtt}" ] || cp "${config}" "${customtt}"
vim "${customtt}"Comment out this single line in the file:
# 25,55 * * * * root . /etc/default/ngcp-roles; if ...
Please don’t forget to reverse the manual changes after the software upgrade as described in Enabling background services section of the handbook!
For all NGCP systems, regardless of their version:
Apply configuration changes by executing s:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr5.5.10' ngcpcfg push
To specify the new list of APT data sources, execute the following commands on both nodes:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.5.10/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/jessie/stretch/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
[ -r /etc/default/ngcp-proxy ] && source /etc/default/ngcp-proxy
mgmt_node=$(cat /etc/ngcp_mgmt_node)
grep -q "debian-debug" /etc/apt/sources.list.d/debian.list || \
echo "deb http://${mgmt_node:-sp}:${APPROX_RO_PORT:-9998}/debian-debug/ stretch-debug main contrib non-free" >> /etc/apt/sources.list.d/debian.list
grep -q "debian-debug" /etc/approx/approx.conf || \
echo "debian-debug https://debian.sipwise.com/debian-debug" >> /etc/approx/approx.conf| warning | |
Do not use "ngcpcfg apply/build" after executing the above commands, as otherwise the changes will be overwritten and you will have to redo this step. |
To download the new packages into the approx cache, execute the following command on the standby node. This will ensure that both nodes have identical packages:
ngcp-approx-cache-helper --auto --node localhost
Run the following commands on both nodes to install the package responsible for upgrading C5 to a newer release:
apt-get update apt-get install ngcp-upgrade-pro
The following options in ngcp-upgrade can be specially useful in some
instances of upgrade:
--step-by-step: confirm before proceeding to next step. With this option
the upgrade operation is performed confirming every step before execution,
with the possibility to instruct to continue without confirming further steps
until the end (if confirmation is only needed for some steps at the
beginning).
--pause-before-step STEP_NAME: pause execution before step, given by the
name of the script (e.g. "backup_mysql_db"). This option can be useful in
several scenarios, for example:
--step-by-step), or just continue without stop until the end
--skip-db-backup: This will speed-up the process in cases where it’s
deemed unnecessary, and this is very likely in the upgrade of nodes other than
the first.
Execute the upgrade script on the standby node as root:
ngcp-upgrade
| info | |
sip:provider PRO can be upgraded to mr5.5.10 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one. |
| info | |
If there is an error during the upgrade, the ngcp-upgrade script will
request you to solve it. Once you’ve fixed the problem, just execute
|
The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.
The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).
The following options in ngcp-upgrade can be useful for this phase of
upgrades:
--step-by-step: confirm before proceeding to next step.
--pause-before-step STEP_NAME: pause execution before step, given by the
name of the script (e.g. "backup_mysql_db").
See a more detailed description of the options in: ngcp-upgrade options
Merge/add the custom configuration templates if needed. Apply the changes to configuration templates and send them to the shared storage and the other node:
ngcpcfg apply 'The first node upgraded' ngcpcfg push --nobuild --noapply
Execute on the current standby node as root:
ngcp-make-active
Go to the new standby node. Run ngcp-upgrade as root:
ngcp-upgrade
The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.
The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).
The following options in ngcp-upgrade can be useful for this phase of
upgrades, because it is very likely that the backup was already performed in the
upgrade of the first node:
--skip-db-backup: This will speed-up the process in cases where it’s
deemed unnecessary.
See a more detailed description of the options in: ngcp-upgrade options
In order to disable the maintenance mode, do the following:
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
Enabling background services. If you upgraded from NGCP release earlier than mr5.4.1, you have to manually reverse the changes you applied before the upgrade in the (Disabling background services) section.
Enable background services:
ngcpcfg set /etc/ngcp-config/config.yml "heartbeat.hb_watchdog.enable=yes" ngcpcfg set /etc/ngcp-config/config.yml "mediator.enabled=yes" ngcpcfg set /etc/ngcp-config/config.yml "rateomat.enable=yes"
In order to enable the CDR exporter process, remove the corresponding customtt file:
rm /etc/ngcp-config/templates/etc/cron.d/ngcp-cdr-exporter.customtt.tt2
For all NGCP systems, regardless of their previous version:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr5.5.10' ngcpcfg push
When everything has finished successfully, check that replication is running.
Check ngcp-status --all.
Finally, do a basic functionality test.
Check the web interface, register two test subscribers and perform a test call
between them to ensure call routing works.
| info | |
You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr5.5.10-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr5.5.10-*/upgrade.log. |