3. Upgrading the sip:provider PRO

3.1. Preparation
3.2. Upgrade from previous release
3.3. Upgrade from 2.8 LTS

3.1. Preparation

Make sure you’re prepared to spend two hours or so upgrading the system. There will be service interruptions, so also notify the customer and get their approval.

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary):

ngcp-status --integrity

Try to find local changes to the template files by issuing:

find /etc/ngcp-config -name \*customtt.tt2

You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Also, please check/clean old dpkg backup files (just in case if previous person did the previous step not carefully enough). Normally the list should be empty:

find /etc/ngcp-config -name \*.tt2.dpkg\*

You will have to understand why the changes are there and if they are still needed after the upgrade. You should create a ticket in the bugtracker if there isn’t one yet.

Log into the two servers. Use their real IPs so you can switch the cluster forth and back later on. Make sure the cluster status is ok - on both nodes issue:

  • monit summary - one should be running all services, the other all but rtpengine, lb, proxy, sbc, mediator and rate-o-mat
  • cl_status rscstatus - one (with all services running) should print "all", the other "none"
  • mysql -e "show slave status\G" - look for the following:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
  • ngcpcfg status - should print OK three times
  • ngcp-collective-check - should not report any problems.

A cluster fail-over could be a good idea to see if everything works on the second node too. On the standby node issue:

/usr/share/heartbeat/hb_takeover

Afterwards again check monit, cl_status and ngcp-collective-check.

Create two test subscribers, or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.

Run "apt-get update", ensure you have no warnings/errors here.

[Warning]

Installation may use locally specified mirrors. Discuss with a customer possibility to switch on Sipwise APT repositories (at least for the time of upgrades), the public Debian mirrors may not provide packages for old Releases anymore or be at least outdated!

To switch on Sipwise APT repositories, execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
## custom sources.list, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr3.8.13
#
# Debian repositories
deb http://debian.sipwise.com/debian/ wheezy main contrib non-free
#deb-src http://debian.sipwise.com/debian/ wheezy main contrib non-free
#
deb http://debian.sipwise.com/debian-security/ wheezy-security main contrib non-free
#deb-src http://debian.sipwise.com/debian-security/ wheezy-security main contrib non-free
#
deb http://debian.sipwise.com/debian/ wheezy-updates main contrib non-free
#deb-src http://debian.sipwise.com/debian/ wheezy-updates main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE - do not remove this line if it should be automatically handled
# deployed via upgrade ${NGCP_CURRENT_VERSION}->mr3.8.13
#
# Sipwise repository
deb http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ wheezy main
#deb-src http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ wheezy main
EOF

Run "apt-get update" once again and ensure you have no warnings/errors here.

3.2. Upgrade from previous release

[Note]

Make sure you have finished with general Section 3.1, “Preparation” before continue.

The sip:provider PRO system upgrade to mr3.8.13 will perform a couple of fundamental tasks:

  • Update DB schema
  • Upgrade yml config schema

So assuming you have a running sip:provider PRO system and want to upgrade it, start on the inactive node by upgrading software, then take over from the other node and then upgrade the other (now inactive) node, as detailed in the steps below.

  1. Switch to new repositories

    For upgrading the sip:provider PRO to the latest mr3.8.13 release, execute the following commands on both nodes:

    NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
    sed -i "s/$NGCP_CURRENT_VERSION/mr3.8.13/" /etc/apt/sources.list.d/sipwise.list
    apt-get update
    apt-get install ngcp-upgrade-mr3.8.13-pro
  2. Execute ngcp-upgrade in inactive node.

    ngcp-upgrade
    [Note]

    If there is an error during DB schema update, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem just re-execute ngcp-upgrade again. When all finishes successfully check that replication is running.

  3. Merge/add the custom configuration templates if needed. Apply the changes to configuration templates if any and send them to the shared storage and the other node:

    ngcpcfg apply
    ngcpcfg push --nobuild --noapply
  4. Promote inactive node to active.

    /usr/share/heartbeat/hb_takeover
  5. Go to the new inactive node. Run ngcp-upgrade.

    ngcp-upgrade

    When all finishes successfully check that replication is running. Check monit summary, ngcpcfg status and ngcp-collective-check. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between the two to ensure call routing works.

    [Note]

    You can find a backup of some important configuration files of your existing installation under /var/backup/ngcp-mr3.8.13- (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr3.8.13-/upgrade.log.

3.3. Upgrade from 2.8 LTS

[Note]

Make sure you have finished with general Section 3.1, “Preparation” before continue.

The system upgrade from sip:provider PRO 2.8 LTS to mr3.8.13 will perform a couple of fundamental tasks:

  • Verify APT source lists
  • Upgrade Debian from 6.0 to 7.0
  • Update DB schema
  • Upgrade yml config schema

Run "apt-get update", ensure you have no warnings/errors here.

[Warning]

Installation may use locally specified mirrors. Discuss with a customer switching on Sipwise APT repositories. All public Debian mirrors do not provide squeeze packages anymore!

To switch on Sipwise APT repositories, execute the following commands as root:

echo "# Please visit /etc/apt/sources.list.d/ instead." > /etc/apt/sources.list

mkdir -p /etc/apt/sources.list.d
for file in /etc/apt/sources.list.d/*.list ; do mv "${file}" "${file}.DISABLED" ; done

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/debian.list << EOF
## custom sources.list, deployed via upgrade ${NGCP_CURRENT_VERSION}->mr3.8.13
#
# Debian repositories
deb http://debian.sipwise.com/debian/ squeeze main contrib non-free
#deb-src http://debian.sipwise.com/debian/ squeeze main contrib non-free
#
deb http://debian.sipwise.com/debian-security/ squeeze-security main contrib non-free
#deb-src http://debian.sipwise.com/debian-security/ squeeze-security main contrib non-free
#
deb http://debian.sipwise.com/debian/ squeeze-updates main contrib non-free
#deb-src http://debian.sipwise.com/debian/ squeeze-updates main contrib non-free
EOF

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
cat > /etc/apt/sources.list.d/sipwise.list << EOF
# NGCP_MANAGED_FILE - do not remove this line if it should be automatically handled
# deployed via upgrade ${NGCP_CURRENT_VERSION}->mr3.8.13
#
# Sipwise repository
deb http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ squeeze main
#deb-src http://deb.sipwise.com/sppro/${NGCP_CURRENT_VERSION}/ squeeze main
#
# Sipwise squeeze backports
deb http://deb.sipwise.com/squeeze-backports/ squeeze-backports main
#deb-src http://deb.sipwise.com/squeeze-backports/ squeeze-backports main
#
# Debian's squeeze-lts, using Sipwise' mirror
deb http://debian.sipwise.com/debian/ squeeze-lts main
#deb-src http://debian.sipwise.com/debian/ squeeze-lts main
#
# Percona's high performance mysql builds
deb http://deb.sipwise.com/percona/ squeeze main
#deb-src http://deb.sipwise.com/percona/ squeeze main
EOF

Run "apt-get update" once again and ensure you have no warnings/errors here.

[Warning]

be sure you are using the latest 2.8 LTS version before upgrading to mr3.8.13 LTS version. mr3.8.13 LTS version uses new repository GPG key which should be installed in 2.8 LTS version before further upgrade to mr3.8.13 LTS version. For upgrading the 2.8 LTS version to the latest 2.8 packages version, execute the following commands as root (on both nodes, using switchover and executing commands on inactive pair):

ngcpcfg pull && apt-get update && apt-get upgrade && \
ngcp-update-db-schema && ngcp-update-cfg-schema && \
ngcpcfg apply && ngcpcfg push --nobuild --noapply
[Note]

re-check you have all the latest packages installed (ensure you have 0 everywhere below):

> apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

So assuming you have a running the latest sip:provider PRO 2.8 LTS system and want to upgrade it, start on the inactive node by upgrading software, then take over from the other node and then upgrade the other (now inactive) node, as detailed in the steps below.

  1. Switch to new repositories

    For upgrading the sip:provider PRO from 2.8 LTS to the latest mr3.8.13 release, execute the following commands on both nodes:

    NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
    sed -i "s/$NGCP_CURRENT_VERSION/mr3.8.13/" /etc/apt/sources.list.d/sipwise.list
    sed -i '/squeeze-lts/d'     /etc/apt/sources.list.d/sipwise.list
    sed -i 's/squeeze/wheezy/g' /etc/apt/sources.list.d/sipwise.list
    apt-get update
    apt-get install ngcp-upgrade-mr3.8.13-pro
  2. Execute ngcp-upgrade in inactive node.

    ngcp-upgrade
    [Note]

    If there is an error during DB schema update, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem just re-execute ngcp-upgrade again. When all finishes successfully check that replication is running.

  3. Merge/add the custom configuration templates if needed. Apply the changes to configuration templates if any and send them to the shared storage and the other node:

    ngcpcfg apply
    ngcpcfg push --nobuild --noapply
  4. Promote inactive node to active.

    /usr/lib64/heartbeat/hb_takeover
  5. Go to the new inactive node. Run ngcp-upgrade.

    ngcp-upgrade

    When all finishes successfully check that replication is running. Check monit summary, ngcpcfg status and ngcp-collective-check. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between the two to ensure call routing works.

    [Note]

    You can find a backup of some important configuration files of your existing 2.8 LTS installation under /var/backup/ngcp-mr3.8.13- (where * is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /var/backup/ngcp-mr3.8.13-/upgrade.log.

[Note]

mr3.8.13 LTS release has several major changes comparing to previous 2.8 LTS release:

  1. Upgrade Debian from 6.0 to 7.0
  2. sip:provider PRO mr3.8.13 requires at least 2GB of RAM available as the minimum requirements for new WEB interface, otherwise certain features won’t work and you will run into arbitrary issues.
  3. Migrated from SOAP to REST API. SOAP API is disabled by default after upgrade and will be completely removed in mr4.0+
  4. Migrated from Apache to Nginx. Apache and related packages will NOT be deleted during upgrade. However it is highly recommended to migrate customer specific VirtualHosts (if any) from Apache to Nginx and remove Apache and related packages from sip:provider PRO system after upgrade manually. You can use the following command as root (on both nodes):
apt-get purge apache2 apache2.2-common apache2-utils apache2.2-bin && apt-get autoremove