16. Security and Maintenance

Once the sip:provider CE is in production, security and maintenance becomes really important. In this chapter, we’ll go through a set of best practices for any production system.

16.1. Sipwise SSH access to sip:provider CE

The sip:provider CE provides SSH access to the system for Sipwise operational team for debugging and final tuning. Operational team uses user sipwise which can be logged in through SSH key only (password access is disabled) from dedicated access server jump.sipwise.com only.

To completely remove Sipwise access to your system, please execute as user root:

root@myserver:~# ngcp-support-access --disable && apt-get install ngcp-support-noaccess


	you have to execute the command above on each node of your sip:provider CE system!


	please ensure that the script complete successfully:

  * Support access successfully disabled.

If you need to restore Sipwise access to the system, please execute as user root:

root@myserver:~# apt-get install ngcp-support-access && ngcp-support-access --enable


	please ensure that the script complete successfully:

  * Support access successfully enabled.

16.2. Firewalling

The sip:provider CE runs a wide range of services. Some of them need to interact with the user, while some others need to interact with the administrator or with nobody at all. Assuming that we trust the sip:provider CE server for outgoing connections, we’ll focus only on incoming traffic to define the services that need to be open for interaction.

Table 8. Subscribers

Service	Default port	Config option
Customer self care interface	443 TCP	www_admin→http_csc→port
SIP	5060 UDP, TCP	kamailio→lb→port
SIP over TLS	5061 TCP	kamailio→lb→tls→port + kamailio→lb→tls→enable
RTP	30000-40000 UDP	rtpproxy→minport + rtpproxy→maxport
XCAP	1080 TCP	kamailio→proxy→presence→enable + nginx→xcap_port
XMPP	5222 and 5269 TCP	None, standard XMPP ports for clients (5222) and federation (5269)

Table 9. Administrators

Service	Default port	Config option
SSH/SFTP	22 TCP	NA
Administrator interface	1443 TCP	www_admin→http_admin→port
Provisioning interfaces	2443 TCP	ossbss→apache→port

To function correctly, the rtpengine requires an additional iptables rule installed. This rule (with a target of RTPENGINE) is automatically installed and removed when the rtpengine starts and stops, so normally you don’t need to worry about it. However, any 3rd party firewall solution can potentially flush out all existing iptables rules before installing its own, which would leave the system without the required RTPENGINE rule and this would lead to decreased performance. It is imperative that any 3rd party firewall solution either leaves this rule untouched, or installs it back into place after flushing all rules out. The complete parameters to install this rule (which needs to go into the INPUT chain of the filter table) are: -p udp -j RTPENGINE --id 0

16.3. Password management

The sip:provider CE comes with some default passwords the user should change during the deployment of the system. They have been explained in the previous chapters of this document.

The login for the system account cdrexport is disabled by default. Although this is a jailed account, it has access to sensitive information, namely the Call Detail Records of all calls. SSH keys should be used to login this user, or alternatively a really strong password should be used when setting the password via passwd cdrexport.
The root user in MySQL has no default password. A password should be set using the mysqladmin password command.
The administrative web interface has a default user administrator with password administrator. It should be changed within this interface.
Generate new password for user ngcpsoap to access the provisioning interfaces, see the details in Section 12, “Provisioning interfaces”.

The Vagrant/VirtualBox/VMWare sip:provider CE images come with more default credentials which should be changed immediately:

The default password of the system account root is sipwise. A password must be changed immediately using command passwd root.
SSH authorized_keys for users root and sipwise should be wiped out using command rm ~root/.ssh/authorized_keys ~sipwise/.ssh/authorized_keys for VirtualBox/VMWare images (skip the step if you use Vagrant).


	Many NGCP services use MySQL backend. Users and passwords for these services are created during the installation. These passwords are unique for each installation, and the connections are restricted to localhost. You should not change these users and passwords.

16.4. SSL certificates.

The sip:provider CE provides default, self-signed SSL certificates for SSL connections. These certificates are common for every installation. Before going to production state, the system administrator should provide SSL certificates for the web services. These certificates can either be shared by all web interfaces (provisioning, administrator interface and customer self care interface), or separate ones for each them can be used.

Generate the certificates. The customer self care interface certificate should be signed by a certification authority to avoid browser warnings.
Upload the certificates to the system
Set the path to the new certificates in /etc/ngcp-config/config.yml:
- ossbss→apache→autoprov→sslcertfile and ossbss→apache→autoprov→sslcertkeyfile for the provisioning interface.
- ossbss→apache→restapi→sslcertfile and ossbss→apache→restapi→sslcertkeyfile for the REST interface.
- www_admin→http_admin→sslcertfile and www_admin→http_admin→sslcertkeyfile for the admin interface.
- www_admin→http_csc→sslcertfile and www_csc→http_csc→sslcertkeyfile for the customer self care interface.
Apply the configuration changes with ngcpcfg apply.

The sip:provider CE also provides the self-signed SSL certificates for SIP over TLS services. The system administrator should replace them with certificates signed by a trusted certificate authority if he is going to enable it for the production usage (kamailio→lb→tls→enable (disabled by default)).

Generate the certificates.
Upload the certificates to the system
Set the path to the new certificates in /etc/ngcp-config/config.yml:
- kamailio→lb→tls→sslcertfile and kamailio→lb→tls→sslcertkeyfile .
Apply the configuration changes with ngcpcfg apply.

16.5. sip:provider CE Backup

For any service provider it is important to maintain a reliable backup policy as it enables prompt services restoration after any force majeure event. Hence, we strongly suggest you to configure a backup procedure. The sip:provider CE has a built-in solution that can help you back up the most crucial data. Alternatively, it can be integrated with any Debian compatible backup software.

16.5.1. What data to back up

The database

This is the most important data in the system. All subscriber and billing information, CDRs, user preferences, etc. are stored in the MySQL server. It is strongly recommended to have up-to-date dumps of all the databases.

System configuration

The system configuration files such as /etc/mysql/sipwise.cnf and the /etc/ngcp-config/ directory should be included in the backup as well. We suggest backing up the whole /etc folder.

Exported CDRs (optional)

The /home/jail/home/cdrexport directory contains the exported CDRs. It depends on your call data retention policy whether or not to remove these files after exporting them to an external system.

16.5.2. The built-in backup solution

The sip:provider CE comes with an easy-to-use solution that creates everyday backups of the most important data:

The system configuration files. The whole /etc directory is backed up.
Exported CDRs. The /home/jail/home/cdrexport directory with csv files.
All required databases on corresponding servers.

This functionality is disabled by default and can be enabled and configured in the backuptools subsection in the config.yml file. Please, refer to the NGCP configuration overview - backuptools Section A.1.2, “backuptools” section for the backup configuration options.

Once you set the required configuration options, apply the changes:

ngcpcfg apply 'enabled the backup feature'

Once you activate the feature, the sip:provider CE will create backups in the off-peak time and put them to the /var/backup/ngcp_backup directory. You can copy these files to your backup server using scp or ftp.


	make sure that you have enough free disk space to store the backups for the specified number of days.

16.6. Recovery

In the worst case scenario, when the system needs to be recovered from a total loss, you only need 4 steps to get the services back online:

Install the sip:provider CE as explained in chapter 2.
Restore the /etc/ngcp-config/ directory and the /etc/mysql/sipwise.cnf file from the backup, overwriting your local files.
Restore the database from the latest MySQL dump.
Apply the changes to bring the original configuration into effect:

ngcpcfg apply 'restored the system from the backup'

16.7. Reset Database


	All existing data will be wiped out! Use this script only if you want to clear all previously configured services and start configuration from scratch.

To reset database to its original state you can execute a script: ngcp-reset-db. It will assign new unique passwords for the NGCP services and reset all services. The script will also create dumps for all NGCP databases.

16.8. System requirements and performance

The sip:provider CE is a very flexible system, capable of serving from hundreds to several tens of thousands of subscribers in a single node. The system comes with a default configuration, capable of serving up to 50.000 subscribers in a normal environment. But there is no such thing as a normal environment. And the sip:provider CE has sometimes to be tunned for special environments, special hardware requirements or just growing traffic.


	If you have performance issues with regards to disk I/O please consider enabling the noatime mount option for the root filesystem. Sipwise recommends the usage of noatime, though remove it if you use software which conflicts with its presence.

In this section some parameters will be explained to allow the sip:provider CE administrator tune the system requirements for optimum performance.

Table 10. Requirement_options

Option	Default value	Requirement impact
cleanuptools→binlog_days	15	Heavy impact on the harddisk storage needed for mysql logs. It can help to restore the database from backups or restore broken replication.
database→bufferpoolsize	64MB	For test systems or low RAM systems, lowering this setting is one of the most effective ways of releasing RAM. The administrator can check the innodb buffer hit rate on production systems; a hit rate over 99% is desired to avoid bottlenecks.
kamailio→lb→pkg_mem	16	This setting affects the amount of RAM the system will use. Each kamailio-lb worker will have this amount of RAM reserved. Lowering this setting up to 8 will help to release some memory depending on the number of kamailio-lb workers running. This can be a dangerous setting as the lb process could run out of memory. Use with caution.
kamailio→lb→shm_mem	1/16 * Total System RAM	The installer will set this value to 1/16 of the total system RAM. This setting does not change even if the system RAM does so it’s up to the administrator to tune it. It has been calculated that 1024 (1GB) is a good value for 50K subscriber environment. For a test environment, setting the value to 64 should be enough. "Out of memory" messages in the kamailio log can indicate that this value needs to be raised.
kamailio→lb→tcp_children	8	Number of TCP workers kamailio-lb will spawn per listening socket. The value should be fine for a mixed UDP-TCP 50K subscriber system. Lowering this setting can free some RAM as the number of kamailio processess would decrease. For a test system or a pure UDP subscriber system 2 is a good value. 1 or 2 TCP workers are always needed.
kamailio→lb→tls→enable	yes	Enable or not TLS signaling on the system. Setting this value to "no" will prevent kamailio to spawn TLS listening workers and free some RAM.
kamailio→lb→udp_children	8	See kamailio→lb→tcp_children explanation
kamailio→proxy→children	8	See kamailio→lb→tcp_children explanation. In this case the proxy only listens udp so these children should be enough to handle all the traffic. It could be set to 2 for test systems to lower the requirements.
kamailio→proxy→*_expires		Set the default and the max and min registration interval. The lower it is more REGISTER requests will be handled by the lb and the proxy. It can impact in the network traffic, RAM and CPU usage.
kamailio→proxy→natping_interval	30	Interval for the proxy to send a NAT keepalive OPTIONS message to the nated subscriber. If decreased, this setting will increase the number of OPTIONS requests the proxy needs to send and can impact in the network traffic and the number of natping processes the system needs to run. See kamailio→proxy→natping_processes explanation.
kamailio→proxy→natping_processes	7	Kamailio-proxy will spawn this number of processes to send keepalive OPTIONS to the nated subscribers. Each worker can handle about 250 messages/second (depends on the hardware). Depending the number of nated subscribers and the kamailio→proxy→natping_interval parameter the number of workers may need to be adjusted. The number can be calculated like nated_subscribers/natping_interval/pings_per_second_per_process. For the default options, asuming 50K nated subscribers in the system the parameter value would be 50.000/30/250 = (6,66) 7 workers. 7 is the maximum number of processes kamailio will accept. Raising this value will cause kamailio not to start.
kamailio→proxy→shm_mem	1/16 * Total System RAM	See kamailio→lb→shm_mem explanation.
rateomat→enable	yes	Set this to no if the system shouldn’t perform rating on the CDRs. This will save CPU usage.
rsyslog→external_log	0	If enabled, the system will send the log messages to an external server. Depending on the rsyslog→external_loglevel parameter this can increase dramatically the network traffic.
rsyslog→ngcp_logs_preserve_days	93	This setting will set the number of days ngcp logs under /var/log/ngcp will be kept in disk. Lowering this setting will free a high amount of disk space.


	In case of using virtualized environment with limited amount of hardware resources, you can use the script ngcp-toggle-performance-config to adjust sip:provider CE configuration for high/low performance:

root@spce:~# /usr/sbin/ngcp-toggle-performance-config
/usr/sbin/ngcp-toggle-performance-config - tool to adjust sip:provider configuration for low/high performance

  --help                Display this usage information
  --high-performance    Adjust configuration for system with normal/high performance
  --low-performance     Adjust configuration for system with low performance (e.g. VMs)

root@spce:~#

16.9. Troubleshooting

The sip:provider CE platform provides detailed logging and log files for each component included in the system via rsyslog. The main folder for log files is /var/log/ngcp/, it contains a list of self explanatory log files named by component name.

The sip:provider CE is a high performance system which requires compromise between traceability (maximum amount of debug information being written to hard drive) and productivity (minimum load on IO subsystem). This is the reason why different log levels are configured for the provided components by default.

Most log files are designed for debugging sip:provider CE by Sipwise operational team while main log files for daily routine usage are:

Log file	Content	Estimated size
/var/log/ngcp/api.log	API logs providing type and content of API requests and responses as well as potential errors	medium
/var/log/ngcp/panel.log	Admin Web UI logs when performing operational tasks on the ngcp-panel	medium
/var/log/ngcp/cdr.log	mediation and rating logs, e.g. how many CDRs have been generated and potential errors in case of CDR generation or rating fails for particular accounting data	medium
/var/log/ngcp/kamailio-lb.log	Overview of SIP requests and replies along with network source and destination information flowing through the platform	huge


	it is highly NOT recommended to change default log levels as it can cause system IO overloading which will affect call processing.


	the exact size of log files depend on system type, system load, system health status and system configuration, so cannot be estimated with high precision. Additionally operational network parameters like ASR and ALOC may impact the log files' size significantly.