Use their real IP so you can switch the cluster forth and back later on.

Switch to the terminal multiplexer under the user sipwise (to reuse Sipwise .screenrc settings which are user-friendly for handling upgrade in multiple windows):

screen -S ngcp-upgrade

Become a root inside your screen session:

sudo -s

Check the system overall status:

ngcp-status --all

Ensure that all proxy nodes replicate read-only DB (MySQL on 127.0.0.1:3308) from the node db01a. Otherwise, inform your manager about the special state here.

Try to find local changes to the template files by issuing:

find /etc/ngcp-config -name \*customtt.tt2

You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Also, please check/clean old dpkg backup files (just in case if another engineer did the previous step not carefully enough).

Normally the list should be empty:

find /etc/ngcp-config -name \*.tt2.dpkg\*

You will have to understand why the changes are there and if they are still needed after the upgrade.

You must create a ticket in the bug tracker to include customtt changes in the following releases (to remove customtt one day).

	Installation may use locally specified apt Debian mirrors. Discuss with a customer possibility to switch on Sipwise APT repositories (at least for the time of upgrades), the public Debian mirrors may not provide packages for old Releases anymore or be at least outdated!

Open separate windows for all the servers inside your screen session. (Press Ctrl+a + c to open new window, Ctrl+a+a or Ctrl+a + [0-9] to change the window. Ctrl+a + " can open list of all windows for you. C+a + A can be used to change the screen name, so you can mark hosts here).

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary) on all servers:

ngcp-status --integrity

Make sure the cluster status is ok - on all nodes issue manually:

Can be checked on all nodes in parallel, using clish and parallel-ssh:

	to exit from ngcp-clish press Ctrl+Z (or type exit):

root@web01b:~# ngcp-clish
Entering 'clish-enable' view (press Ctrl+Z to exit)...
# exit
root@web01b:~#

Run "apt-get update", ensure you have no warnings/errors here.

A cluster failover could be a good idea to see if everything works on the second node too. On the standby node issue:

/usr/share/heartbeat/hb_takeover

Afterwards, again check ngcp-status --all.

Create two test subscribers or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.

	sometimes DB and MGMT roles are assigned to the same host. It is OK.

	do NOT execute the current step on web01a and db01a in parallel!

The main goal here is to fill the approx cache with new version of packages. So all the other nodes will get identical version of packages as the first one.

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.1/" /etc/apt/sources.list.d/sipwise.list
ngcp-approx-cache-helper --auto --node localhost
apt-get update
apt-get install ngcp-upgrade-pro

	do NOT worry, ngcp-upgrade-carrier does not exist, use ngcp-upgrade-pro above.

Execute ngcp-upgrade in inactive node as root:

ngcp-upgrade

	sip:carrier can be upgraded to mr4.5.1 from previous release or previous build only. The script ngcp-upgrade will find all the possible destination releases for the upgrade and allow to choose the proper one.

	If there is an error during upgrade, the ngcp-upgrade script will request you to solve it. Once you fix the problem, just re-execute ngcp-upgrade again and it will continue from the previous step.

Merge/add the customtt configuration templates if needed. Apply the changes to configuration templates if any:

ngcpcfg apply 'applying customtt for new release mrX.X on node xxx01a'

Send new templates to the shared storage and the other nodes

ngcpcfg push --nobuild --noapply all

	do NOT execute ngcpcfg push --shared-only on this stage, it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glustefs.

	If DB and MGMT roles are assigned to the same host, skip this step as you have upgraded inactive MGMT node "A" above already.

Run the following commands to upgrade inactive DB node "A" (choose the same release version as above and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.1/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

	it is important to upgrade db01a node before upgrading any proxy nodes. Otherwise "local" MySQL (127.0.0.1:3308) on proxy nodes may be out of sync if new release has _not_replicated.up DB statements.

Run the following commands here (choose the same release version and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.1/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

	ensure all inactive nodes "A" are:

Run on all "A" nodes:

/usr/share/heartbeat/hb_takeover

Ensure "A" node became active, feel free to reuse 'ngcp-status' and 'ngcp-clish' commands described above.

Ensure ALL "B" nodes are inactive now!

Run the following commands here (choose the same release version and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.1/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

	you can upgrade all inactive "B" nodes together (including mgmt and db roles).