The sip:carrier version mr6.1.2 has the following important changes:
Please find the complete changelog in our release notes on our WEB site.
The sip:carrier software upgrade procedure to mr6.1.2 will perform several fundamental tasks:
sip:carrier is a PRO-style system that has "A" and "B" sets of nodes with specific roles. The number of nodes can differ between installations and must be clarified before the upgrade at the planning stage.
The software upgrade is performed by Sipwise engineers with the following steps:
| warning | |
The only allowed software upgrade path is the one described above. All the other theoretically possible upgrade scenarios can lead to unpredictable results. |
| warning | |
Nodes "A" and "B" MUST be used as described in this document. It is NOT allowed to swap them unless proxy replication (of MySQL on port 3308) is configured on the db01b node. |
Have the following information prepared in writing:
It is recommended to execute all preparation steps in this chapter a few days before the actual software upgrade. They do not cause service downtime, so it is safe to execute them in peak hours.
| tip | |
Use the physical IP addresses instead of the shared one, so you can jump between the nodes later on. |
Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):
screen -S ngcp-upgrade
Become root inside your screen session:
sudo -s
Check the overall system status:
ngcp-status --all
Ensure that all the proxy nodes replicate the read-only DB (127.0.0.1:3308) from the db01a node. Otherwise, discuss a special plan to address this particularity.
For the following steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).
Create tickets to Sipwise developers to make relevant custom modifications part of the product in the future releases. This will allow you to get rid of the customtt files.
Find local changes to the template files by executing:
ngcp-customtt-diff-helper
ngcp-customtt-diff-helper can help you to download the templates for you target release. To do so, you can run the following command:
ngcp-customtt-diff-helper -d
In the tmp folder provided by the script you can, now, merge the current customtt with the new tt2, creating the new customtt.tt2 files. Once did that, you can TAR the new customtts in a tar.gz file and you can use that after the upgrade to deploy the new customtt.
ngcp-customtt-diff-helper -t
You can check further details regarding the ngcp-customtt-diff-help with the option "-h".
Check if there are any *.tt2.dpkg-dist files among the templates. They usually appear when tt2 files are modified directly instead of creating customtt files. If you find any *.tt2.dpkg-dist files, then treat the corresponding tt2 files as if they were custom templates and introduce the changes from the existing tt2 files into the new templates (by creating associated *.customtt.tt2) before the software upgrade.
find /etc/ngcp-config -name \*.tt2.dpkg-dist
Note that in the end all *.tt2.dpkg-dist files must be removed before the software upgrade.
If you changed the configuration (e.g. added custom templates or changed the existing ones), then the system must be thoroughly tested when these changes are applied. Continue with the software upgrade preparation only when the results of the tests are successful.
Check and remove dpkg files left from previous software upgrades.
Make sure that the list is empty before you continue:
find /etc/ngcp-config -name \*.tt2.dpkg\*
| warning | |
If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones! |
Open separate windows for all the servers inside your "screen" session.
(Press Ctrl+a + c to open a new window, Ctrl+a + a or Ctrl+a + [0-9] to
change the window. Ctrl+a + " shows the list of all your windows.
Use Ctrl+a + A to change the window names to corresponding hosts).
Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary) on all servers:
ngcp-status --integrity
Make sure the cluster status is OK: on all nodes manually run:
Check all nodes in parallel, using clish and parallel-ssh:
ngcp-clish "ngcp version package check" - ensure that all nodes have the identical Debian package installed
| info | |
Software on all nodes must be identical before and after the upgrade! |
| info | |
to exit from ngcp-clish press Ctrl+Z (or type exit): |
root@web01b:~# ngcp-clish Entering 'clish-enable' view (press Ctrl+Z to exit)... # exit root@web01b:~#
Run "apt-get update" on all nodes. Ensure that you have no warnings/errors here.
Test the cluster failover to see if everything works fine on "B" nodes as well. On all the standby nodes execute:
ngcp-make-active
Afterwards, check ngcp-status --all again.
Log in to all nodes and execute the checks from Section 12.4, “Preparing the software upgrade” again. This will ensure that nothing was broken since the preparation steps were finished. Also, execute ngcpcfg show and ngcpcfg status to check the latest configuration changes.
Perform the BFT test.
Sipwise NGCP introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of NGCP will disable some background services (for instance: mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr6.1.2' ngcpcfg push all
To upgrade the sip:carrier to mr6.1.2 release, execute the following commands on the standby management "A" node:
| info | |
Sometimes the DB and MGMT roles are assigned to the same host. This is OK. |
| warning | |
Do NOT execute the software upgrade on web01a and db01a in parallel! |
The main goal of the following commands is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr6.1.2/" /etc/apt/sources.list.d/sipwise.list ngcp-approx-cache-helper --auto --node localhost apt-get update apt-get install ngcp-upgrade-pro
| info | |
Don’t worry, ngcp-upgrade-carrier does not exist, use ngcp-upgrade-pro as outlined above. |
| warning | |
Do not use "ngcpcfg apply/build" after executing the steps from the above section, otherwise the changes will be overwritten and you will have to redo these steps. The same applies to similar sections below. |
Execute ngcp-upgrade on the standby node as root:
ngcp-upgrade
| info | |
sip:carrier can be upgraded to mr6.1.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one. |
| info | |
If there is an error during the upgrade, the ngcp-upgrade script will
request you to solve it. Once you’ve fixed the problem, just execute
|
Merge/add the custom configuration templates if needed.
Apply the changes to configuration templates:
ngcpcfg apply 'applying customtt for new release mrX.X on node xxx01a'
Send the new templates to the shared storage and the other nodes
ngcpcfg push --nobuild --noapply all
| warning | |
Do NOT execute ngcpcfg push --shared-only at this stage, as it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glusterfs. |
| info | |
If the DB and MGMT roles are assigned to the same host, then skip this step as you have already upgraded the standby MGMT node "A" above. |
Run the following commands to upgrade the standby DB node "A" (select the same release version as above and follow the on-screen recommendations):
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr6.1.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| info | |
It is important to upgrade db01a node before upgrading any proxy nodes.
Otherwise, the "local" MySQL (127.0.0.1:3308) on proxy nodes may become out of
sync in case the new release has |
Run the below commands selecting the same release version and follow the on-screen recommendations:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr6.1.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| warning | |
Ensure that all standby nodes "A" are:
* upgraded to the new release (check /etc/ngcp_version or use |
On all "A" nodes run:
ngcp-make-active
Ensure that the "A" nodes became active, by executing the 'ngcp-status' and 'ngcp-clish' commands described above.
Ensure that ALL "B" nodes are standby now!
Run the following commands selecting the same release version and following the on-screen recommendations:
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr6.1.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| info | |
You can upgrade all standby "B" nodes simultaneously (including the ones with the mgmt and db roles). |
In order to disable the maintenance mode, do the following:
ngcpcfg pull
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr6.1.2' ngcpcfg push all
When everything has finished successfully, check that replication is running.
Check ngcp-status --all.
Finally, do a basic functionality test.
Check the web interface, register two test subscribers and perform a test call
between them to ensure call routing works.
| info | |
You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr6.1.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr6.1.2-*/upgrade.log. |
If your current release is already the latest or you prefer to be on the LTS release, we still suggest appling the latest hotfixes and critical bug fixes.
Execute all steps as described in ???. They include the system checks, customtt handling and others. It is important to execute all the steps from the above chapter.
It is suggested to promote B-nodes to active and start the update with A-nodes.
The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.
ngcp-approx-cache-helper --auto --node localhost
Execute on the standby nodes as root:
ngcp-make-active
Check in a minute that the nodes became active:
ngcp-check-active