16. Security and Maintenance

16.1. Sipwise SSH access to sip:provider PRO
16.2. Firewalling
16.3. Password management
16.4. SSL certificates.
16.5. Securing your sip:provider PRO against SIP attacks
16.5.1. Denial of Service
16.5.2. Bruteforcing SIP credentials
16.6. Backup and recovery
16.6.1. Backup
16.6.2. Recovery
16.7. Reset database
16.8. Synchronize database
16.9. System requirements and performance
16.10. Troubleshooting
16.10.1. Collecting call information from logs
16.10.2. Collecting SIP traces

Once the sip:provider PRO is in production, security and maintenance becomes really important. In this chapter, we’ll go through a set of best practices for any production system.

16.1. Sipwise SSH access to sip:provider PRO

The sip:provider PRO provides SSH access to the system for Sipwise operational team for debugging and final tuning. Operational team uses user sipwise which can be logged in through SSH key only (password access is disabled) from dedicated access server jump.sipwise.com only.

To completely remove Sipwise access to your system, please execute as user root:

root@myserver:~# ngcp-support-access --disable && apt-get install ngcp-support-noaccess
[Note]

you have to execute the command above on each node of your sip:provider PRO system!

[Warning]

please ensure that the script complete successfully:

  * Support access successfully disabled.

If you need to restore Sipwise access to the system, please execute as user root:

root@myserver:~# apt-get install ngcp-support-access && ngcp-support-access --enable
[Warning]

please ensure that the script complete successfully:

  * Support access successfully enabled.

16.2. Firewalling

The sip:provider PRO runs a wide range of services. Some of them need to interact with the user, while some others need to interact with the administrator or with nobody at all. Assuming that we trust the sip:provider PRO server for outgoing connections, we’ll focus only on incoming traffic to define the services that need to be open for interaction.

Table 11. Subscribers

ServiceDefault portConfig option

Customer self care interface

443 TCP

www_admin→http_csc→port

SIP

5060 UDP, TCP

kamailio→lb→port

SIP over TLS

5061 TCP

kamailio→lb→tls→port + kamailio→lb→tls→enable (Disabled by default)

RTP

30000-40000 UDP

rtpproxy→minport + rtpproxy→maxport

XCAP

1080 TCP

kamailio→proxy→presence→enable + nginx→xcap_port (Disabled by default)

XMPP

5222 and 5269 TCP

None, standard XMPP ports for clients (5222) and federation (5269)


Table 12. Administrators

ServiceDefault portConfig option

SSH/SFTP

22 TCP

NA

Administrator interface

1443 TCP

www_admin→http_admin→port

Provisioning interfaces

2443 TCP

ossbss→apache→port


[Caution]

To function correctly, the rtpengine requires an additional iptables rule installed. This rule (with a target of RTPENGINE) is automatically installed and removed when the rtpengine starts and stops, so normally you don’t need to worry about it. However, any 3rd party firewall solution can potentially flush out all existing iptables rules before installing its own, which would leave the system without the required RTPENGINE rule and this would lead to decreased performance. It is imperative that any 3rd party firewall solution either leaves this rule untouched, or installs it back into place after flushing all rules out. The complete parameters to install this rule (which needs to go into the INPUT chain of the filter table) are: -p udp -j RTPENGINE --id 0

16.3. Password management

The sip:provider PRO comes with some default passwords the user should change during the deployment of the system. They have been explained in the previous chapters of this document.

  • The login for the system account cdrexport is disabled by default. Although this is a jailed account, it has access to sensitive information, namely the Call Detail Records of all calls. SSH keys should be used to login this user, or alternatively a really strong password should be used when setting the password via passwd cdrexport.
  • The root user in MySQL has no default password. A password should be set using the mysqladmin password command.
  • The administrative web interface has a default user administrator with password administrator. It should be changed within this interface.
  • Generate new password for user ngcpsoap to access the provisioning interfaces, see the details in Section 12, “Provisioning interfaces”.
[Important]

Many NGCP services use MySQL backend. Users and passwords for these services are created during the installation. These passwords are unique for each installation, and the connections are restricted to localhost. You should not change these users and passwords.

16.4. SSL certificates.

The sip:provider PRO provides default, self-signed SSL certificates for SSL connections. These certificates are common for every installation. Before going to production state, the system administrator should provide SSL certificates for the web services. These certificates can either be shared by all web interfaces (provisioning, administrator interface and customer self care interface), or separate ones for each them can be used.

  • Generate the certificates. The customer self care interface certificate should be signed by a certification authority to avoid browser warnings.
  • Upload the certificates to the system
  • Set the path to the new certificates in /etc/ngcp-config/config.yml:

    • ossbssapacheautoprovsslcertfile and ossbssapacheautoprovsslcertkeyfile for the provisioning interface.
    • ossbssapacherestapisslcertfile and ossbssapacherestapisslcertkeyfile for the REST interface.
    • www_adminhttp_adminsslcertfile and www_adminhttp_adminsslcertkeyfile for the admin interface.
    • www_adminhttp_cscsslcertfile and www_adminhttp_cscsslcertkeyfile for the customer self care interface.
  • Apply the configuration changes with ngcpcfg apply 'added web ssl certs'.

The sip:provider PRO also provides the self-signed SSL certificates for SIP over TLS services. The system administrator should replace them with certificates signed by a trusted certificate authority if he is going to enable it for the production usage (kamailiolbtlsenable (disabled by default)).

  • Generate the certificates.
  • Upload the certificates to the system
  • Set the path to the new certificates in /etc/ngcp-config/config.yml:

    • kamailiolbtlssslcertfile and kamailiolbtlssslcertkeyfile .
  • Apply the configuration changes with ngcpcfg apply 'added kamailio certs'.

16.5. Securing your sip:provider PRO against SIP attacks

The sip:provider PRO allows you to protect your VoIP system against SIP attacks, in particular Denial of Service and brute-force attacks. Let’s go through each of those attacks and let’s see how to configure your system in order to face such situations and react against them.

16.5.1. Denial of Service

As soon as you have packets arriving on your sip:provider PRO server, it will require a bit of time of your CPU. Denial of Service attacks are aimed to break down your system by sending floods of SIP messages in a very short period of time and keep your system busy to handle such huge amount of requests. sip:provider PRO allow you to block such kind of attack quite easily, by configuring the following section in your /etc/ngcp-config/config.yml:

security:
   dos_ban_enable: 'yes'
   dos_ban_time: 3600
   dos_reqs_density_per_unit: 50
   dos_sampling_time_unit: 2

Basically, as soon as sip:provider PRO receives more than 50 messages from the same IP in a time window of 2 seconds, that IP will be block for 3600 sec, and you will see in the the kamailio-lb.log a line saying:

Nov 9 00:11:53 sp1 lb[41958]: WARNING: <script>: IP '1.2.3.4' is blocked and banned - R=<null> ID=304153-3624477113-19168@tedadg.testlab.local

The banned IP will be stored in kamailio memory, you can check the list via web interface or via the following command:

# ngcp-kamctl lb fifo sht_dump ipban

16.5.2. Bruteforcing SIP credentials

This is a very common attack you can easily detect checking your /var/log/ngcp/kamailio-proxy.log. You will see INVITE/REGISTER messages coming in with strange usernames. Attackers is trying to spoof/guess subscriber’s credentials, which allow them to call out. The very first protection against these attacks is: ALWAYS USE STRONG PASSWORD. Nevertheless sip:provider PRO allow you to detect and block such attacks quite easily, by configuring the following /etc/ngcp-config/config.yml section:

  failed_auth_attempts: 3
  failed_auth_ban_enable: 'yes'
  failed_auth_ban_time: 3600

You may increase the number of failed attempt if you want (in same cases it’s better to be safed, some users can be banned accidentally because they are not writing the right password) and adjust the ban time. If a user try to authenticate an INVITE (or REGISTER) for example and it fails more then 3 times, the "user@domain" (not the IP as for Denial of Service attack) will be block for 3600 seconds. In this case you will see in your /var/log/ngcp/kamailio-lb.log the following lines:

Nov 9 13:31:56 sp1 lb[41952]: WARNING: <script>: Consecutive Authentication Failure for 'sipvicous@mydomain.com' UA='sipvicous-client' IP='1.2.3.4' - R=<null> ID=313793-3624525116-589163@testlab.local

Both the banned IPs and banned users are shown in the Admin web interface, you can check them by accessing the Security Bans section in the main menu. You can check the banned user as well by retrieving the same info directly from kamailio memory, using the following commands:

# ngcp-kamctl lb fifo sht_dump auth

16.6. Backup and recovery

16.6.1. Backup

The sip:provider PRO can be integrated with most of the existing backup solutions. While it does not provide any backup system by default, any Debian compatible system can be installed. It’s not the scope of this chapter to go through backup system configuration. We’ll focus on which information needs to be saved.

The minimum set of information to be backed up is:

  • The database information.

This is the most important data in the system. All subscriber information, billing, CDRs, user preferences etc. are stored in the MySQL server. A periodical dump of all the databases should be performed.

  • System configuration options

/etc/ngcp-config/config.yml, /etc/ngcp-config/constants.yml, /etc/ngcp-config/network.yml and /etc/mysql/sipwise.cnf files, where your specific system configurations are stored, should be included in the backup as well. Saving the entire /etc/ngcp-config folder is a good idea in general.

  • Optional: Exported CDRs

The directory /home/jail/home/cdrexport contains the exported CDRs the system has generated so far. It depends on your local call data retention policy whether or not to remove these files after exporting them to an external system.

  • Optional: Custom files

Any custom configurations, like modified templates or additionally implemented services which are not provided by the sip:provider PRO

16.6.2. Recovery

In the worst case scenario, when the system needs to be recovered from a total loss, you only need 4 steps to get back online:

  • Install the sip:provider PRO as explained in chapter 2.
  • Restore config.yml, constants.yml, network.yml and sipwise.cnf from the backup, overwriting your local files.
  • Restore the database dump.
  • Execute ngcpcfg apply 'my commit message'.

16.7. Reset database

To reset database to its original state you can use the script provided by CE: * Execute ngcp-reset-db. It will assign new unique password for the NGCP services and restart all services. IMPORTANT: All existing data will be wiped out without possibility of restoring.

16.8. Synchronize database

In case of unresolvable database replication issues or to copy mysql data between a pair of hosts (usually a pair of sp1 and sp2 nodes).

There is a script for that: ngcp-sync-db.

To synchronize databases you need to run the script on your target host.

  • Definitions:

    • master - remote/master host (the database is dumped from there)
    • local - target/local host (the database is imported onto)
  • Usage:
[Important]

Your existing database on local will be completely wiped. The script provides a possibility to backup both master and local databases during the procedure.

You can run the script with -h or --help to check its options or use man ngcp-sync_db

If you run it without any options it automatically calculates master hostname (e.g. if you run it on sp2 then sp2==local and sp1==master).

The script also requires mysql credentials and if none provided it uses username=sipwise and the password is picked from /etc/mysql/sipwise.cnf. You can specify user and/or password for both master and local.

Before the actual start it produces a summary with settings used to the procedure and a confirmation prompt to prevent accidental usage. Making use of --force option" however suppresses the confirmation prompt. By default no messages are printed on STDOUT (compliant to be integrated into another tools) and with -v or --verbose options you enable debugging where all the ongoing steps will be printed to STDOUT.

There are 2 modes available for synchronization, online and backup. By default online is used where the procedure does not create any backups and everything goes on the fly. That is useful for large databases where creating backups would require solid amounts of available free disk space. With the backup mode master db is dumped into a backup file on local first (default directory: /var/backup/ngcp-sync-db) and imported upon the backup completion.

Mysql database connection to the master db and the local db is the essential part and by default the script tries to establish direct mysql connection however that may not be possible due to the access restrictions. To overcome that you can use --ssh-tunnel option and specifying there a local custom free port (e.g. --ssh-tunnel=33125) in this case an ssh tunnel will be created to master and used to establish the db connection on the localhost behalf (NOTE: Public key based ssh negotiation is required for the tunnel as the script does not suppot ssh credentials for security reasons).

Backups may be a subject to create during synchornization for possible rollbacks. To create the local db backup you should add --local-backup. The master db backup is automatically created only using --sync-mode=backup. Upon completion all those created backups are deleted and if you need to keep them please use --keep-backups option (NOTE: In case of errors during synchronization and when backups are created they are NOT automatically deleted. Therefore, if the script had failed with an error and afterwards completed successully you may want to manually remove the remaining backups from /var/backup/ngcp-sync-db).

  • Examples:

Normal online mode synchronization sp1sp2.

sp2> ngcp-sync-db

Normal backup mode synchronization sp1sp2.

sp2> ngcp-sync-db --sync-mode=backup

Forced online mode synchronization sp1sp2. USE WITH CARE as there will be no confirmation prompts.

sp2> ngcp-sync-db --force

Direct mysql db access is not possible. SSH tunnel is initialised to local port 33125 and forwards all connections 127.0.0.1:33125 → sp1:3306.

sp2> ngcp-sync-db --ssh-tunnel=33125

Custom mysql credentials for the master db connection (by default: sipwise:/etc/mysql/sipwise.cnf)

sp2> ngcp-sync-db --master-user=frank --master-pass=dbconnect

Normal online mode synchronization sp1sp2 with the local db backup and retaining the backup. (no master backup in this case as it is only available with --sync-mode=backup).

sp2> ngcp-sync-db --local-backup --keep-backups

Normal online mode synchronization custom-nodesp2 with ssh tunnel

sp2> ngcp-sync-db --master-host=custom-node --ssh-tunnel=45001

Forced syncrhonization custom-nodesp2 with ssh tunnel, backup sync mode, local backup, custom master and local db credentials and ports as well as a different backup dir

sp2> ngcp-sync-db --force --sync-mode=backup --master-host=custom-node --master-port=3308 --ssh-tunnel=45001 --master-user=frank --master-pass=dbconnect --local-user=john --local-pass=dblocal --local-backup --keep-backups --backup-dir=/home/barry/backups

16.9. System requirements and performance

The sip:provider PRO is a very flexible system, capable of serving from hundreds to several tens of thousands of subscribers in a single node. The system comes with a default configuration, capable of serving up to 50.000 subscribers in a normal environment. But there is no such thing as a normal environment. And the sip:provider PRO has sometimes to be tunned for special environments, special hardware requirements or just growing traffic.

[Note]

If you have performance issues with regards to disk I/O please consider enabling the noatime mount option for the root filesystem. Sipwise recommends the usage of noatime, though remove it if you use software which conflicts with its presence.

In this section some parameters will be explained to allow the sip:provider PRO administrator tune the system requirements for optimum performance.

Table 13. Requirement_options

OptionDefault valueRequirement impact

cleanuptools→binlog_days

15

Heavy impact on the harddisk storage needed for mysql logs. It can help to restore the database from backups or restore broken replication.

database→bufferpoolsize

1/2 * Total system RAM

The installer will calculate the total system RAM and dedicate 50% to the mysql innodb buffer. This value won’t be changed in case the system RAM changes so it’s up to the administrator to adjust it. For test systems or low RAM systems, lowering this setting is one of the most effective ways of releasing RAM. The administrator can check the innodb buffer hit rate on production systems; a hit rate over 99% is desired to avoid bottlenecks.

kamailio→lb→pkg_mem

16

This setting affects the amount of RAM the system will use. Each kamailio-lb worker will have this amount of RAM reserved. Lowering this setting up to 8 will help to release some memory depending on the number of kamailio-lb workers running. This can be a dangerous setting as the lb process could run out of memory. Use with caution.

kamailio→lb→shm_mem

1/16 * Total System RAM

The installer will set this value to 1/16 of the total system RAM. This setting does not change even if the system RAM does so it’s up to the administrator to tune it. It has been calculated that 1024 (1GB) is a good value for 50K subscriber environment. For a test environment, setting the value to 64 should be enough. "Out of memory" messages in the kamailio log can indicate that this value needs to be raised.

kamailio→lb→tcp_children

8

Number of TCP workers kamailio-lb will spawn per listening socket. The value should be fine for a mixed UDP-TCP 50K subscriber system. Lowering this setting can free some RAM as the number of kamailio processes would decrease. For a test system or a pure UDP subscriber system 2 is a good value. 1 or 2 TCP workers are always needed.

kamailio→lb→tls→enable

yes

Enable or not TLS signaling on the system. Setting this value to "no" will prevent kamailio to spawn TLS listening workers and free some RAM.

kamailio→lb→udp_children

8

See kamailio→lb→tcp_children explanation

kamailio→proxy→children

8

See kamailio→lb→tcp_children explanation. In this case the proxy only listens udp so these children should be enough to handle all the traffic. It could be set to 2 for test systems to lower the requirements.

kamailio→proxy→*_expires

Set the default and the max and min registration interval. The lower it is more REGISTER requests will be handled by the lb and the proxy. It can impact in the network traffic, RAM and CPU usage.

kamailio→proxy→natping_interval

30

Interval for the proxy to send a NAT keepalive OPTIONS message to the nated subscriber. If decreased, this setting will increase the number of OPTIONS requests the proxy needs to send and can impact in the network traffic and the number of natping processes the system needs to run. See kamailio→proxy→natping_processes explanation.

kamailio→proxy→natping_processes

7

Kamailio-proxy will spawn this number of processes to send keepalive OPTIONS to the nated subscribers. Each worker can handle about 250 messages/second (depends on the hardware). Depending the number of nated subscribers and the kamailio→proxy→natping_interval parameter the number of workers may need to be adjusted. The number can be calculated like nated_subscribers/natping_interval/pings_per_second_per_process. For the default options, assuming 50K nated subscribers in the system the parameter value would be 50.000/30/250 = (6,66) 7 workers. 7 is the maximum number of processes kamailio will accept. Raising this value will cause kamailio not to start.

kamailio→proxy→shm_mem

1/16 * Total System RAM

See kamailio→lb→shm_mem explanation.

rateomat→enable

yes

Set this to no if the system shouldn’t perform rating on the CDRs. This will save CPU usage.

rsyslog→external_log

0

If enabled, the system will send the log messages to an external server. Depending on the rsyslog→external_loglevel parameter this can increase dramatically the network traffic.

rsyslog→ngcp_logs_preserve_days

93

This setting will set the number of days ngcp logs under /var/log/ngcp will be kept in disk. Lowering this setting will free a high amount of disk space.


[Tip]

In case of using virtualized environment with limited amount of hardware resources, you can use the script ngcp-toggle-performance-config to adjust sip:provider PRO configuration for high/low performance:

root@spce:~# /usr/sbin/ngcp-toggle-performance-config
/usr/sbin/ngcp-toggle-performance-config - tool to adjust sip:provider configuration for low/high performance

  --help                Display this usage information
  --high-performance    Adjust configuration for system with normal/high performance
  --low-performance     Adjust configuration for system with low performance (e.g. VMs)

root@spce:~#

16.10. Troubleshooting

The sip:provider PRO platform provides detailed logging and log files for each component included in the system via rsyslog. The main folder for log files is /var/log/ngcp/, it contains a list of self explanatory log files named by component name.

The sip:provider PRO is a high performance system which requires compromise between traceability (maximum amount of debug information being written to hard drive) and productivity (minimum load on IO subsystem). This is the reason why different log levels are configured for the provided components by default.

Most log files are designed for debugging sip:provider PRO by Sipwise operational team while main log files for daily routine usage are:

Log fileContentEstimated size

/var/log/ngcp/api.log

API logs providing type and content of API requests and responses as well as potential errors

medium

/var/log/ngcp/panel.log /var/log/ngcp/panel-debug.log

Admin Web UI logs when performing operational tasks on the ngcp-panel

medium

/var/log/ngcp/cdr.log

mediation and rating logs, e.g. how many CDRs have been generated and potential errors in case of CDR generation or rating fails for particular accounting data

medium

/var/log/ngcp/ha.log

fail-over related logs in case a node in a pair loses connection to the other side, when a standby node takes over or an active node goes standby due to intra-node communication issues or external ping node connection issues

small

/var/log/ngcp/kamailio-proxy.log

Overview of SIP requests and replies between lb, proxy and sems processes. It’s the main log file for SIP overview

huge

/var/log/ngcp/kamailio-lb.log

Overview of SIP requests and replies along with network source and destination information flowing through the platform

huge

/var/log/ngcp/sems.log

Overview of SIP requests and replies between lb, proxy and sems processes

small

/var/log/ngcp/rtp.log

rtpengine related log, showing information about RTP communication

small

[Warning]

it is highly NOT recommended to change default log levels as it can cause system IO overloading which will affect call processing.

[Note]

the exact size of log files depend on system type, system load, system health status and system configuration, so cannot be estimated with high precision. Additionally operational network parameters like ASR and ALOC may impact the log files' size significantly.

16.10.1. Collecting call information from logs

The easiest way to fetch information about a single call among the log files is the search for the SIP CallID (a unique identifier for a SIP dialog). The call ID is used as call marker in almost all the voip related log file, such as /var/log/ngcp/kamailio-lb.log , /var/log/ngcp/kamailio-proxy.log , /var/log/ngcp/sems.log or /var/log/ngcp/rtp.log. Example of kamailio-proxy.log line:

Nov 19 00:35:56 sp1 proxy[7475]: NOTICE: <script>: New request on proxy - M=REGISTER R=sip:sipwise.local
F=sip:jdoe@sipwise.local T=sip:jdoe@sipwise.local IP=10.10.1.10:5060 (127.0.0.1:5060) ID=364e4676776621034977934e055d19ea@127.0.0.1 UA='SIP-UA 1.2.3.4'

The above line shows the SIP information you can find in a general line contained in /var/log/ngcp/kamailio-*:

  • M=REGISTER : The SIP Method
  • R=sip:sipwise.local : The SIP Request URI
  • F=sip:jdoe@sipwise.local : The SIP From header
  • T=sip:jdoe@sipwise.local : The SIP To header
  • IP=10.10.1.10:5060 (127.0.0.1:5060) : The source IP where the message is coming from. Between brackets it is shown the local internal IP where the message come from (in this case Load Balancer)
  • ID=364e4676776621034977934e055d19ea@127.0.0.1 : The SIP CallID.
  • UAIP=10.10.1.10 : The User Agent source IP
  • UA=SIP-UA 1.2.3.4 : The SIP User Agent header

In order to collect the full log related to a single call, it’s necessary to "grep" the /var/log/ngcp/kamailio-proxy.log using the ID= string, for example:

# grep "364e4676776621034977934e055d19ea@127.0.0.1" /var/log/ngcp/kamailio-proxy.log

16.10.2. Collecting SIP traces

The sip:provider PRO platform provides several tools to collect SIP traces. It can be used the sip:provider PRO ngrep-sip tool to collect SIP traces, for example to fetch traffic in text format from outbound and among load balancer, proxy and sems :

# ngrep-sip b

see the manual to know all the options:

# man ngrep-sip

The ngrep debian tool can be used in order to make a SIP trace and save it into a .pcap file :

# ngrep -s0 -Wbyline -d any -O /tmp/SIP_trace_file_name.pcap port 5062 or port 5060

The sngrep debian graphic tool as well can be used to visualize SIP trace and save them in a .pcap file :

# sngrep

The sip:provider PRO platform provides also the native Voip sniffer, called voisniff-ng, which provide a graphic view of all the calls passing through the platform. It can be enabled via __/etc/ngcp-config/config.yml:

voisniff:
  admin_panel: 'yes'
  daemon:
    bpf: 'port 5060 or 5062 or ip6 proto 44 or ip[6:2] & 0x1fff != 0'
    external_interfaces: 'eth0 eth1'
    filter:
      exclude:
        -
          active: 1
          case_insensitive: 1
          pattern: '\ncseq: *\d+ +(register|notify|options|subscribe)'
      include: []
    internal_interfaces: lo
    mysql_dump_threads: 4
    start: 'yes'
    threads_per_interface: 10
  partitions:
    increment: 700000
    keep: 10

admin_panel should be set to yes as well as start. Also filter.exclude.active should be set to 1 in order to avoid to sniff REGISTER, NOTIFY, OPTIONS and SUBSCRIBE messages. Then run:

ngcpcfg apply 'enable voisniff' && ngcpcfg push
[Warning]

Please notice that enabling voisniff, specially under a huge amount of traffic, may affect the system performance due to the fact that voisniff needs to save all the traffic into the database.