The Sipwise C5 version mr7.5.11 has the following important changes:
Please find the complete changelog in our release notes on our WEB site.
The Sipwise C5 software upgrade procedure to mr7.5.11 will perform several fundamental tasks:
Sipwise C5 is a PRO-style system that has "A" and "B" sets of nodes with specific roles. The number of nodes can differ between installations and must be clarified before the upgrade at the planning stage.
The software upgrade is usually performed by Sipwise engineers according to the following steps:
| warning | |
The only allowed software upgrade path is the one described above. All the other theoretically possible upgrade scenarios can lead to unpredictable results. |
| warning | |
Nodes "A" and "B" MUST be used as described in this document. It is NOT allowed to swap them unless proxy replication (of MySQL on port 3308) is configured on the db01b node. |
Confirm the following information:
It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.
| tip | |
Use the static server IP address so you can switch between the nodes. |
Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):
screen -S my_screen_name_for_ngcp_upgrade
Become root inside your screen session:
sudo -s
Check the overall system status:
ngcp-status --all
Make sure that the cluster health status is OK: Check the nodes in parallel, using the clish command:
| info | |
Software must be identical on all nodes (before and after the upgrade!) |
| info | |
to exit from ngcp-clish press Ctrl+Z (or type exit): |
# ngcp-clish Entering 'clish-enable' view (press Ctrl+Z to exit)... # exit #
For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).
Create tickets to Sipwise developers to make relevant custom modifications part of the product in future releases. This allows you to get rid of the customtt files one day.
| warning | |
If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable. |
Find the local changes to the template files:
ngcp-customtt-diff-helper
The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:
ngcp-customtt-diff-helper -d
In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:
ngcp-customtt-diff-helper -t
Find all available script options with the "-h" parameter.
| warning | |
Starting from version mr7.0.1 a new kamailio module called "pv_headers" has been introduced. This new module enables storing all headers in XAVP to freely modify them in the kamailio logic and only apply them once when it’s time for the packet to be routed outside. The main goal of the module is to offload the intermediate header processing into the XAVP dynamic container as well as provide with high-level methods and pseudovariables to simplify SIP message header modifications. The module is enabled by default in kamailio proxy and all the templates have been updated to use this new logic. Before proceeding with the upgrade it is essential that the customtt/patchtt you have in place are updated to this new format. At appendix you can find additional information on the module. |
Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt/patchtt files. If you find any *.tt2.dpkg-dist files, treat the corresponding tt2 files as if they were customtt.tt2 and introduce the changes from the existing tt2 files into the new templates (create associated customtt.tt2 or patchtt.tt2) before the software upgrade.
find /etc/ngcp-config -name \*.tt2.dpkg-dist
Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade as they prevent the upgrade script from updating the tt2 files.
Check and remove dpkg files left from previous software upgrades.
Make sure that the list is empty before you continue:
find /etc/ngcp-config -name \*.tt2.dpkg\*
Log into all the servers.
Open separate windows for all the servers inside your "screen" session.
(Press Ctrl+a + c to open a new window, Ctrl+a + a or Ctrl+a + [0-9] to
change the window. Ctrl+a + " shows the list of all your windows.
Use Ctrl+a + A to change the window names to corresponding hosts).
Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:
ngcp-status --integrity
Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:
ngcpcfg status
Check the replication on both central DB servers and on ports 3306 and 3308 of all the proxy servers. Ensure that all the proxy nodes replicate the read-only DB (127.0.0.1:3308) from the db01a node. Otherwise, discuss a special plan to address your particular configuration. The result must always show:
Slave_IO_Running: Yes Slave_SQL_Running: Yes Seconds_Behind_Master: 0
Test the cluster failover to see if everything works fine on "B" nodes as well. On all the standby nodes execute:
ngcp-make-active
Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.
Run "apt-get update" on all nodes and ensure that you do not have any warnings and errors in the output.
| warning | |
If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones! |
| warning | |
Upgrade to mr7.5 is allowed with kamailio.location in Redis DB only! It is necessary for the proper migration of location records from datetime to bigint/epoch. |
Starting from mr6.2.1, location, acc and dialogs data are stored in RedisDB allowing better system performances. Before proceed with the upgrade, check if location data are already migrated to RedisDB:
ngcpcfg values "kamailio.proxy.redis.usrloc"
If the answer is yes, then skip this sub-chapter and proceed with the next one. On the contrary, an answer equals to no means that the migration process has not been completed. This happens because, to be more flexible and to reduce the downtime of the system, only acc and dialogs data have been moved automatically to RedisDB. Migration of location data is a step that should be done manually and it was probably skipped during previous upgrades.
To proceed with the migration and complete the process, execute the following commands during out of business hours.
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "kamailio.proxy.redis.usrloc=yes"
ngcpcfg apply 'Enable location data storage on RedisDB'
ngcpcfg push db01a lb01a prx01a prx02a ...
| important | |
Execute the following 3 steps one after another with as short as possible delay between them. |
ngcp-location-migrate
ngcp-location-sync
ngcp-make-active
ngcpcfg push db01b lb01b prx01b prx02b ...
ngcp-location-migrate -c
Ensure that both management nodes have access to deb.sipwise.com by executing the following commands on a management node:
ngcp-approx-cache-helper --check --node sp1 ngcp-approx-cache-helper --check --node sp2
If --check is not recognised, as it likely happens when upgrading from mr6.5
and early mr7.x versions, use the short form instead:
ngcp-approx-cache-helper -c --node sp1 ngcp-approx-cache-helper -c --node sp2
The checks must show only 200 OK results. If you see cannot connect!, Received 404 or any other error, check these possible causes:
| info | |
The document describes upgrade from all mr7.4.* and mr6.5.* LTS releases. |
Log in to all nodes and execute the checks from Section 12.4, “Preparing the software upgrade” again. This will ensure that nothing was broken since the preparation steps were finished. Also, execute ngcpcfg show and ngcpcfg status to check the latest configuration changes.
Perform the BFT test.
The Sipwise C5 — starting from mr6.5.1 release — enforce software licensing restrictions in form of a regular comparison of the licensed services and capacities against the actual usage patterns of the platform. In case some functionalities are enabled but not licensed, an error in syslog will be reported and the impacted services will be automatically deactivated.
Before proceeding with the upgrade, please take some time to check that all the modules not licensed are actually disabled in config.yml file. To verify if they are enabled execute the following commands:
ngcpcfg values sems.prepaid.enable ngcpcfg values sems.prepaid.inew.enable ngcpcfg values pbx.enable ngcpcfg values pushd.enable ngcpcfg values intercept.enable ngcpcfg values voisniff.admin_panel ngcpcfg values voisniff.daemon.li_x1x2x3.enable ngcpcfg values voisniff.daemon.start
If the output of one of the commands is yes but the module is not licensed, you have to deactivate it. For example, in case of pre-paid billing module execute:
ngcpcfg set /etc/ngcp-config/config.yml sems.prepaid.enable=no ngcpcfg apply 'Disable prepaid module' ngcpcfg push all
| warning | |
Please, pay particular attention to pre-paid billing module because it is enabled by default. |
Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr7.5.11' ngcpcfg push all
To upgrade Sipwise C5 CARRIER to mr7.5.11 release, execute the following commands on the standby management "A" node:
The following options in ngcp-upgrade can be specially useful in some
instances of upgrade:
--step-by-step: confirm before proceeding to next step. With this option
the upgrade operation is performed confirming every step before execution,
with the possibility to instruct to continue without confirming further steps
until the end (if confirmation is only needed for some steps at the
beginning).
--pause-before-step STEP_NAME: pause execution before step, given by the
name of the script (e.g. "backup_mysql_db"). This option can be useful in
several scenarios, for example:
--step-by-step), or just continue without stop until the end
--skip-db-backup: This will speed-up the process in cases where it’s
deemed unnecessary, and this is very likely in the upgrade of nodes other than
the first.
| warning | |
Customers with far-sighted software upgrade policies usually have pre-production installations to test the services in their environment before upgrading the production platform. In this case, the approx cache should be updated on both platforms simultaneously to synchronize the package versions between them, hence consider carefully before updating the approx cache. |
| info | |
Sometimes the DB and MGMT roles are assigned to the same host. This is OK. |
| warning | |
Do NOT execute the software upgrade on web01a and db01a in parallel! |
To download the latest packages metadata into the approx cache, execute the following command on the standby management node. This action ensures that all nodes within the platform have identical packages after the software upgrade:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.5.11/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/stretch/buster/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
ngcp-approx-cache-helper --auto --node localhost
apt-get update
apt-get install ngcp-upgrade-pro| info | |
Don’t worry, ngcp-upgrade-carrier does not exist, use ngcp-upgrade-pro as outlined above. |
| warning | |
Do not use "ngcpcfg apply/build" after executing the steps from the above section, otherwise the changes will be overwritten and you will have to redo these steps. The same applies to similar sections below. |
Run the upgrade script on the standby node as root:
ngcp-upgrade
| info | |
Sipwise C5 can be upgraded to mr7.5.11 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and makes it possible to choose the proper one. |
| info | |
If there is an error during the upgrade, the ngcp-upgrade script will
request you to solve it. Once you’ve fixed the problem, just execute
|
Merge/add the custom configuration templates if needed.
Apply the changes to configuration templates:
ngcpcfg apply 'apply customtt/patchtt for new the release mrX.X on xxx01a'
Send the new templates to the shared storage and the other nodes
ngcpcfg push --nobuild --noapply all
| warning | |
Do NOT execute ngcpcfg push --shared-only at this stage, as it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glusterfs. |
| info | |
If the DB and MGMT roles are assigned to the same host, then skip this step as you have already upgraded the standby MGMT node "A" above. |
Run the following commands to upgrade the standby DB node "A" (select the same release version as above and follow the on-screen recommendations):
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.5.11/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/stretch/buster/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade| info | |
It is important to upgrade db01a node before upgrading any proxy nodes.
Otherwise, the "local" MySQL (127.0.0.1:3308) on proxy nodes may become out of
sync in case the new release has |
Run the below commands selecting the same release version and follow the on-screen recommendations:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.5.11/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/stretch/buster/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgradeThe following options in ngcp-upgrade can be useful for this phase of
upgrades, because it is very likely that the backup was already performed:
--skip-db-backup: This will speed-up the process in cases where it’s
deemed unnecessary.
See a more detailed description of the options in: ngcp-upgrade options
| warning | |
Ensure that all standby nodes "A" are:
* upgraded to the new release (check /etc/ngcp_version or use |
On all "A" nodes run:
ngcp-make-active
Ensure that the "A" nodes became active, by executing the 'ngcp-status' and 'ngcp-clish' commands described above.
Ensure that ALL "B" nodes are standby now!
Run the following commands selecting the same release version and following the on-screen recommendations:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/${NGCP_CURRENT_VERSION}/mr7.5.11/" /etc/apt/sources.list.d/sipwise.list
sed -i "s/stretch/buster/g" /etc/apt/sources.list.d/sipwise.list /etc/apt/sources.list.d/debian.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade| info | |
You can upgrade all standby "B" nodes simultaneously (including the ones with the mgmt and db roles). |
The following options in ngcp-upgrade can be useful for this phase of
upgrades:
--step-by-step: confirm before proceeding to next step.
--pause-before-step STEP_NAME: pause execution before step, given by the
name of the script (e.g. "backup_mysql_db").
See a more detailed description of the options in: ngcp-upgrade options
Starting from mr7.5.1, location entries are saved on RedisDB using timestamp time format (expires and last_modified columns). This allows to handle better the daylight saving time adjustments that occurs every years.
The conversion was automatically performed during the upgrade process, anyhow it could happen that some of the records are still using old format due to REGISTER messages received right after the conversion script was executed. To double check all the location entries and finalize the conversion please execute on the management node the following script:
/usr/sbin/ngcp-upgrade-redis-usrloc --finalize
In order to disable the maintenance mode, do the following:
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr7.5.11' ngcpcfg push all
When everything has finished successfully, check that replication is running.
Check ngcp-status --all.
Finally, do a basic functionality test.
Check the web interface, register two test subscribers and perform a test call
between them to ensure call routing works.
| info | |
You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr7.5.11-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/backup/ngcp-mr7.5.11-*/upgrade.log. |
If your current release is already the latest or you prefer to be on the LTS release, we still suggest appling the latest hotfixes and critical bug fixes.
Execute all steps as described in Section 12.4, “Preparing the software upgrade”. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.
It is suggested to promote B-nodes to active and start the update with A-nodes.
The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.
ngcp-approx-cache-helper --auto --node localhost
Merge/add the custom configuration templates if needed.
Apply the changes to configuration templates:
ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'
Send the new templates to the shared storage and the other nodes.
ngcpcfg push --nobuild --noapply all
Execute on the standby nodes as root:
ngcp-make-active
Check in a minute that the nodes became active:
ngcp-check-active