11. Software Upgrade

11.1. Release Notes
11.2. Preparing for a Software Upgrade
11.3. Upgrade sip:provider PRO from previous mr4.4/mr4.5 versions to mr4.5.13 LTS version
11.3.1. Switch to the new repositories
11.3.2. Download the new packages into the approx cache (on standby node only)
11.3.3. Install the package used to upgrade C5
11.3.4. Upgrade the first PRO node
11.3.5. The customtt files handling (if necessary)
11.3.6. Promote inactive node to active.
11.3.7. Upgrade the second PRO node
11.4. Upgrade sip:provider PRO from previous mr3.8 LTS version to mr4.5.13 LTS version
11.4.1. Verify APT source lists
11.4.2. Switch to new repositories
11.4.3. Upgrade the first PRO node
11.4.4. The customtt files handling (if necessary)
11.4.5. Promote inactive node to active.
11.4.6. Upgrade the second PRO node

11.1. Release Notes

The sip:provider PRO version mr4.5.13 has several important changes comparing to the previous release. Please find the complete changelog in our release notes on our WEB site.

11.2. Preparing for a Software Upgrade

[Warning]

Make sure that all the SIP domains and peering servers have appropriate rtp_interface option (e.g. ext) selected in the NAT and Media Flow Control section. If you leave default there, then incorrect network interface may be used for sending and receiving RTP media after the software upgrade.

Make sure you’re prepared to spend two hours or so upgrading the system. There can be service interruptions, so also notify the customer and get their approval.

Check the system overal status:

ngcp-status --all

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary):

ngcp-status --integrity

Try to find local changes to the template files by issuing:

find /etc/ngcp-config -name \*customtt.tt2

You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Also, please check/clean old dpkg backup files (just in case if previous person did the previous step not carefully enough). Normally the list should be empty:

find /etc/ngcp-config -name \*.tt2.dpkg\*

You will have to understand why the changes are there and if they are still needed after the upgrade. You should create a ticket in the bugtracker if there isn’t one yet.

Log into the two servers. Use their real IPs so you can switch the cluster forth and back later on. Make sure the cluster status is ok - on both nodes issue:

  • monit summary - one should be running all services, the other all but rtpengine, lb, proxy, sbc, mediator and rate-o-mat
  • cl_status rscstatus - one (with all services running) should print "all", the other "none"
  • mysql -e "show slave status\G" - look for the following:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
  • ngcpcfg status - should print OK all the times
  • ngcp-collective-check - should not report any problems.

A cluster fail-over could be a good idea to see if everything works on the second node too. On the standby node issue:

/usr/share/heartbeat/hb_takeover

Afterwards again check monit, cl_status and ngcp-collective-check.

Create two test subscribers, or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.

Run "apt-get update", ensure you have no warnings/errors here.

[Warning]

Installation may use locally specified mirrors. Discuss with a customer possibility to switch on Sipwise APT repositories (at least for the time of upgrades), the public Debian mirrors may not provide packages for old Releases anymore or be at least outdated!

11.3. Upgrade sip:provider PRO from previous mr4.4/mr4.5 versions to mr4.5.13 LTS version

[Warning]

ensure you have read and performed all the steps described in the section "Preparing for a Software Upgrade" above!

The sip:provider PRO system upgrade to mr4.5.13 will perform a couple of fundamental steps:

  • Upgrade NGCP software packages
  • Upgrade NGCP configuration templates
  • Upgrade NGCP DB schema
  • Upgrade the base system within Debian (v8) to the latest package versions

So assuming you have a running sip:provider PRO system and want to upgrade it, start on the inactive node by upgrading software, then take over from the other node and then upgrade the other (now inactive) node, as detailed in the steps below.

11.3.1. Switch to the new repositories

To specify the new list of APT data sources, execute the following commands on both nodes:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.13/" /etc/apt/sources.list.d/sipwise.list
[Warning]

Do not use "ngcpcfg apply/build" after executing the above commands, as otherwise the changes will be overwritten and you will have to redo this step.

11.3.2. Download the new packages into the approx cache (on standby node only)

To download the new packages into the approx cache, execute the following command on the standby node. This will ensure that both nodes have identical packages:

ngcp-approx-cache-helper --auto --node localhost

11.3.3. Install the package used to upgrade C5

Run the following commands on both nodes to install the package responsible for upgrading C5 to a newer release:

apt-get update
apt-get install ngcp-upgrade-pro

11.3.4. Upgrade the first PRO node

Execute the upgrade script on the standby node as root:

ngcp-upgrade
[Note]

sip:provider PRO can be upgraded to mr4.5.13 from previous release or previous build only. The script ngcp-upgrade will find all the possible target releases for the upgrade and allow to choose the proper one.

[Note]

If there is an error during upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem just re-execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

11.3.5. The customtt files handling (if necessary)

Merge/add the custom configuration templates if needed. Apply the changes to configuration templates if any and send them to the shared storage and the other node:

ngcpcfg apply 'upgrade node'
ngcpcfg push --nobuild --noapply

11.3.6. Promote inactive node to active.

Execute on inactive node as root:

/usr/share/heartbeat/hb_takeover

11.3.7. Upgrade the second PRO node

Go to the new inactive node. Run ngcp-upgrade, as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

When all finishes successfully check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between them to ensure call routing works.

[Note]

You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr4.5.13-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr4.5.13-*/upgrade.log.

11.4. Upgrade sip:provider PRO from previous mr3.8 LTS version to mr4.5.13 LTS version

[Warning]

ensure you have read and performed all the steps described in the section "Preparing for a Software Upgrade" above!

The sip:provider PRO system upgrade to mr4.5.13 will perform a couple of fundamental steps:

  • Verify APT source lists
  • Upgrade Debian from version 7 (wheezy) to version 8 (jessie)
  • Upgrade NGCP software packages
  • Upgrade NGCP configuration templates
  • Upgrade NGCP DB schema
  • Upgrade the base system within Debian 8 (jessie) to the latest package versions

So assuming you have a running sip:provider PRO system and want to upgrade it, start on the inactive node by upgrading software, then take over from the other node and then upgrade the other (now inactive) node, as detailed in the steps below.

11.4.1. Verify APT source lists

[Warning]

ensure you are using Sipwise APT repositories. Public Debian mirrors may not provide packages for old Debian releases anymore. Also they can be outdated. Consider to use Sipwise repositories for the time of upgrade. Discuss with a customer the situation here and ensure Sipwise IPs are open on firewalls.

To switch on sipwise APT source lists, execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
## custom sources.list, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr4.5.13
#
# Debian repositories
deb http://debian.sipwise.com/debian/ wheezy main contrib non-free
#deb-src http://debian.sipwise.com/debian/ wheezy main contrib non-free
#
deb http://debian.sipwise.com/debian-security/ wheezy-security main contrib non-free
#deb-src http://debian.sipwise.com/debian-security/ wheezy-security main contrib non-free
#
deb http://debian.sipwise.com/debian/ wheezy-updates main contrib non-free
#deb-src http://debian.sipwise.com/debian/ wheezy-updates main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE - do not remove this line if it should be automatically handled
#
# Sipwise repository
deb http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ wheezy main
#deb-src http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ wheezy main
EOF

Run "apt-get update" and ensure you have no warnings/errors here.

11.4.2. Switch to new repositories

For upgrading the sip:provider PRO to the latest mr4.5.13 release, execute the following commands on both nodes:

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr4.5.13/" /etc/apt/sources.list.d/sipwise.list
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list.d/sipwise.list
rm -f /etc/apt/sources.list.d/dell.list
apt-get update
apt-get install ngcp-upgrade-pro

11.4.3. Upgrade the first PRO node

Execute ngcp-upgrade in inactive node as root:

ngcp-upgrade
[Note]

sip:provider PRO can be upgraded to mr4.5.13 from previous release or previous build only. The script ngcp-upgrade will find all the possible target releases for the upgrade and allow to choose the proper one.

[Note]

If there is an error during upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem just re-execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

11.4.4. The customtt files handling (if necessary)

Merge/add the custom configuration templates if needed. Apply the changes to configuration templates if any and send them to the shared storage and the other node:

ngcpcfg apply 'upgrade node'
ngcpcfg push --nobuild --noapply

11.4.5. Promote inactive node to active.

Execute on inactive node as root:

/usr/share/heartbeat/hb_takeover

11.4.6. Upgrade the second PRO node

Go to the new inactive node. Run ngcp-upgrade, as root:

ngcp-upgrade

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

When all finishes successfully check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between them to ensure call routing works.

[Note]

You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr4.5.13-* (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr4.5.13-*/upgrade.log.