The sip:carrier version mr5.2.2 has several important changes comparing to the previous release:
| important | |
Due to migrating to bcypt hashing of admin and reseller passwords both on the admin panel and the API, password authentication via the API will take ~500ms for each request. It is highly advised to use ssl client certificate based authentication instead of passwords on the API for performance reasons! |
Please find the complete changelog in our release notes on our WEB site.
The sip:carrier system upgrade to mr5.2.2 will perform a couple of fundamental tasks:
sip:carrier is a PRO-style system which has "A" and "B" pairs of nodes which execute specific roles. The nodes amount here is different and must be clarified ahead of the upgrade on the planning stage.
The way to upgrade sip:carrier is clean and simple:
| warning | |
the only allowed way to upgrade sip:carrier is described above. All the other theoretically possible upgrade scenarios can lead to unpredictable results. |
| warning | |
Nodes "A" and "B" MUST be used as described in this document. It is NOT allowed to swipe them unless proxy replication (of MySQL on port 3308) is configured on the db01b node. |
Have a written answer on the following questions:
| tip | |
Use their real IP so you can switch the cluster forth and back later on. |
Switch to the terminal multiplexer under the user sipwise (to reuse Sipwise .screenrc settings which are user-friendly for handling upgrade in multiple windows):
screen -S ngcp-upgrade
Become a root inside your screen session:
sudo -s
Check the system overall status:
ngcp-status --all
Ensure that all proxy nodes replicate read-only DB (MySQL on 127.0.0.1:3308) from the node db01a. Otherwise, inform your manager about the special state here.
Try to find local changes to the template files by issuing:
find /etc/ngcp-config -name \*customtt.tt2
You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.
find /etc/ngcp-config -name \*.tt2.dpkg-dist
Also, please check/clean old dpkg backup files (just in case if another engineer did the previous step not carefully enough).
Normally the list should be empty:
find /etc/ngcp-config -name \*.tt2.dpkg\*
You will have to understand why the changes are there and if they are still needed after the upgrade.
You must create a ticket in the bug tracker to include customtt changes in the following releases (to remove customtt one day).
| warning | |
Installation may use locally specified apt Debian mirrors. Discuss with a customer possibility to switch on Sipwise APT repositories (at least for the time of upgrades), the public Debian mirrors may not provide packages for old Releases anymore or be at least outdated! |
Open separate windows for all the servers inside your screen session.
(Press Ctrl+a + c to open new window, Ctrl+a+a or Ctrl+a + [0-9] to
change the window. Ctrl+a + " can open list of all windows for you.
C+a + A can be used to change the screen name, so you can mark hosts here).
Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary) on all servers:
ngcp-status --integrity
Make sure the cluster status is ok - on all nodes issue manually:
Can be checked on all nodes in parallel, using clish and parallel-ssh:
ngcp-clish "ngcp version package check" - ensure all nodes have identical list of debian package installed.
| info | |
All nodes must be identical before and after the upgrade! |
| info | |
to exit from ngcp-clish press Ctrl+Z (or type exit): |
root@web01b:~# ngcp-clish Entering 'clish-enable' view (press Ctrl+Z to exit)... # exit root@web01b:~#
Run "apt-get update", ensure you have no warnings/errors here.
A cluster failover could be a good idea to see if everything works on the second node too. On the standby node issue:
/usr/share/heartbeat/hb_takeover
Afterwards, again check ngcp-status --all.
Create two test subscribers or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.
For upgrading the sip:carrier to mr5.2.2 release, execute the following commands on inactive management "A" node:
| info | |
sometimes DB and MGMT roles are assigned to the same host. It is OK. |
| warning | |
do NOT execute the current step on web01a and db01a in parallel! |
The main goal here is to fill the approx cache with new version of packages. So all the other nodes will get identical version of packages as the first one.
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr5.2.2/" /etc/apt/sources.list.d/sipwise.list ngcp-approx-cache-helper --auto --node localhost apt-get update apt-get install ngcp-upgrade-pro
| info | |
do NOT worry, ngcp-upgrade-carrier does not exist, use ngcp-upgrade-pro above. |
Execute ngcp-upgrade in inactive node as root:
ngcp-upgrade
| info | |
sip:carrier can be upgraded to mr5.2.2 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one. |
| info | |
If there is an error during upgrade, the ngcp-upgrade script will request
you to solve it. Once you’ve fixed the problem just re-execute |
Merge/add the customtt configuration templates if needed. Apply the changes to configuration templates if any:
ngcpcfg apply 'applying customtt for new release mrX.X on node xxx01a'
Send new templates to the shared storage and the other nodes
ngcpcfg push --nobuild --noapply all
| info | |
do NOT execute ngcpcfg push --shared-only on this stage, it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glustefs. |
| info | |
If DB and MGMT roles are assigned to the same host, skip this step as you have upgraded inactive MGMT node "A" above already. |
Run the following commands to upgrade inactive DB node "A" (choose the same release version as above and follow on-screen recommendations):
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr5.2.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| info | |
it is important to upgrade db01a node before upgrading any proxy nodes.
Otherwise "local" MySQL (127.0.0.1:3308) on proxy nodes may be out of sync if
new release has |
Run the following commands here (choose the same release version and follow on-screen recommendations):
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr5.2.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| warning | |
ensure all inactive nodes "A" are: |
ngcp-clish)
Run on all "A" nodes:
/usr/share/heartbeat/hb_takeover
Ensure "A" node became active, feel free to reuse 'ngcp-status' and 'ngcp-clish' commands described above.
Ensure ALL "B" nodes are inactive now!
Run the following commands here (choose the same release version and follow on-screen recommendations):
NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version) sed -i "s/$NGCP_CURRENT_VERSION/mr5.2.2/" /etc/apt/sources.list.d/sipwise.list apt-get update apt-get install ngcp-upgrade-pro ngcp-upgrade
| info | |
you can upgrade all inactive "B" nodes together (including mgmt and db roles). |
When all finishes successfully check that replication is running.
Check ngcp-status --all.
Finally, do a basic functionality test.
Check web interface, register two test subscribers and perform a test
call between them to ensure call routing works.
| info | |
You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr5.2.2-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr5.2.2-*/upgrade.log. |