Once the sip:provider CE is in production, security and maintenance becomes really important. In this chapter, we’ll go through a set of best practices for any production system.
The sip:provider CE provides SSH access to the system for Sipwise operational team for debugging and final tuning. Operational team uses user sipwise which can be logged in through SSH key only (password access is disabled) from dedicated access server jump.sipwise.com only.
To completely remove Sipwise access to your system, please execute as user root:
root@myserver:~# ngcp-support-access --disable && apt-get install ngcp-support-noaccess
info | |
you have to execute the command above on each node of your sip:provider CE system! |
warning | |
please ensure that the script complete successfully: |
* Support access successfully disabled.
If you need to restore Sipwise access to the system, please execute as user root:
root@myserver:~# apt-get install ngcp-support-access && ngcp-support-access --enable
warning | |
please ensure that the script complete successfully: |
* Support access successfully enabled.
The sip:provider CE runs a wide range of services. In order to secure the platfrom while allowing access to the sip:provider CE the NGCP configuration framework provides a set of predefined network zones. Services are aggregated into appropriate zones by default. Zones are assigned to network interfaces (and VLANs if applicable) in /etc/ngcp-config/network.yml.
caution | |
Though the default firewall setup provided by the NGCP configuration framework provides a save setup for sip:provider CE security audits of the platform performed by qualified engineers before commissioning the platform into service are strongly recommended. Customization of the setup requires in depth konwledge of firewalling principles in general and the netfilter facility in particular. |
Table 13. NGCP network zones
Zone name | Description |
---|---|
ha_int | Internal cluster interface providing internal cluster communications between cluster pairs (heartbeat) and synchronization of data and configuration |
mon_ext | Interface to conect external monitoring appliances (SNMP) |
rtp_ext | Interface for external RTP media relay between sip:provider CE and endpoints (e.g. user agents, peers) |
sip_ext | Interface for external SIP signalling between sip:provider CE and endpoints (e.g. user agents, peers) |
sip_int | Interface for internal signalling, e.g. between load-balancers, proxies and applications servers |
ssh_ext | Interface providing external access to the sip:provider CE command line interface |
web_ext | Interface providing access to the customers' self-care Web panel |
web_int | Interface for access to the administrative Web panel, its REST APIs and internal API communications |
info | |
Additional custom zones may be configured, but will not be automatically integrated into the firewall configuration. |
To facilitate firewall functionality sip:provider CE uses the Kernel’s netfilter facility and iptables-persistent as an interface to netfilter. Netfilter is using tables and within that chains to store rules in this hierarchy: table → chain → rule. Default firewall setups of sip:provider CE do not use netfilter tables nat and raw, but only default table filter.
info | |
Custom nat rules for IPv4 and IPv6 may be added in file /etc/ngcp-config/config.yml in sections security→firewall→nat_rules4 and security→firewall→nat_rules6. |
Each chain deploys a default policy handling packets which did not trigger and rule in a prticular chain.
Table 14. NGCP netfilter default policies
Chain | Default policy | Description |
---|---|---|
INPUT | DROP | Handling all packets directly destined for a sip:provider CE node (only packets matching a rule are allowed) |
FORWARD | DROP | Handling all packets received by a sip:provider CE node and destined for another, non-local IP destination (no default rules added) |
OUTPUT | ACCEPT | Handling all packets originating on a sip:provider CE node (no default rules added) |
rtpengine | N/A | Container for rptengine rule to allow the rule to persist even when the Kernel module is unloaded (e.g. during upgrades) |
The default firewall setup provided by sip:provider CE:
The sip:provider CE comes with a preconfigured set of firewall rules, which can be enabled and configured in /etc/ngcp-config/config.yml
in section security→firewall
. Refer to Section 1.29, “security” for available configuration options.
Firewall configuration is applied by running ngcpcfg apply
. However, this will not activate new rules automatically to avoid inadvertent self-lockout. To finally activate new firewall rules run iptables-apply
. This will prompt for another system logon to verify access remains available. If the prompt is not confirmed, firewall rules will automatically be reverted to the previous state re-enabling access to the command line.
caution | |
The NGCP firewall subsystem by default is disabled in /etc/ngcp-config/config.yml key |
The following set of rules is added by the system upon activation of the firewall subsystem. Individual system rules are configured in /etc/ngcp-config/templates/etc/iptables/rules.v4.tt2 and /etc/ngcp-config/templates/etc/iptables/rules.v6.tt2
Table 15. Firewall system rules
Zone | Chain | Target | Rule | Description |
---|---|---|---|---|
all | INPUT | rtpengine |
| Redirects all incoming UDP packets to chain rtpengine (putting RTPENGINE rule into a dedicated chain allows for the rule to persist even when the Kernel module gets unloaded, e.g. during upgrades) |
all | rtpengine | RTPENGINE |
| Feeds all RTP packets to RTPENGINE Kernel module |
n/a | INPUT | ACCEPT |
| Accept all packets received by local loopback interface |
all | INPUT | ACCEPT |
| Accept all incoming packets tied to related or established connections |
all | INPUT (IPv4) | ACCEPT |
| Accept all ICMP echo messages |
all | INPUT (IPv4) | ACCEPT |
| Accept all ICMP echo reply messages |
all | INPUT (IPv6) | ACCEPT |
| Accept all ICMPv6 messages |
all | INPUT | cluster |
| Divert all incoming packets to the cluster chain |
all | cluster | ACCEPT |
| Set of rules white-listing all IP-addresses owned by the NGCP platform for incoming traffic |
api_int | INPUT | ACCEPT |
| Set of rules for all api_int interfaces accepting all incoming packets for API port defined in /etc/ngcp-config/config.yml with key ossbss.port |
mon_ext | INPUT | ACCEPT | +-p udp -s <snmpclient_ip> --dport 161 -j ACCEPT | Set of rules for all mon_ext interfaces based on a list of IPs for all SNMP communities configured in checktools.snmpd.communities |
rtp_ext | INPUT | ACCEPT |
| Set of rules for all rtp_ext interfaces accepting all incoming packets for RTP port range defined in /etc/ngcp-config/config.yml with keys rtpproxy.minport and rtpproxy.maxport |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.port (UDP) |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.port (TCP) |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.tls.port (TCP/TLS) |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets on TCP port 5222 (XMPP client) |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets on TCP port 5269 (XMPP server) |
sip_ext | INPUT | ACCEPT |
| Set of rules for all sip_ext interfaces accepting all packets incoming for the pushd server port configured in /etc/ngcp-config/config.yml with key pushd.port |
ssh_ext | INPUT | ACCEPT |
| List of rules to accept incoming packets for SSH on all ssh_ext interfaces from hosts configured in /etc/ngcp-config/config.yml with key sshd.permit_support_from |
web_ext | INPUT | ACCEPT |
| List of rules to accept incoming packets for the Customer Self Care interface defined in /etc/ngcp-config/config.yml with key www_admin.http_csc.port on all web_ext interfaces |
web_int | INPUT | ACCEPT |
| List of rules to accept incoming packets for the Admin Panel interface defined in /etc/ngcp-config/config.yml with key www_admin.http_admin.port on all web_int interfaces |
caution | |
To function correctly, the rtpengine requires an additional iptables rule installed. This rule (with a target of |
info | |
Some of the parameters used to populate the firewall rules automatically may contain hostnames instead of IP addresses. Since firewall rules need to be configured based on IP addresses by design, the NGCP configuration framework will lookup such hostnames during ngcpcfg apply and expand them to the IP addresses as returnd by gethostbyname. If DNS resolving changes for such hostnames due to changes to DNS the rules will not update automatically. Another run of ngcpcfg apply will be needed to reperform the lookup and update the rules to reflect chages in DNS. If this step is omitted, clients may be locked out of the system. |
The NGCP configuration framework allows to add custom rules to the firewall setup in /etc/ngcp-config/config.yml. The custom rules are added after the system rules. Hence, they apply for packets not matched by the systems rules only.
Example custom rule to whitelist all IPv4 traffic from network interface eth1.301 effectively making VLAN 301 a trusted network:
rules4: - '-A INPUT -i eth1.301 -j ACCEPT'
Example custom rule to accept incoming traffic from monitoring station 203.0.113.93 for an optionally installed check_mk agent:
rules4: - '-A INPUT -p tcp -s 203.0.113.93 --dport 6556 -j ACCEPT'
To add hosts or networks to the SSH whitelist they can be either added to key sshd.permit_support_from in /etc/ngcp-config/config.yml or a custom rule may be used:
rules4: - '-A INPUT -s 198.51.100.0/24 --dport 22 - j ACCEPT' - '-A INPUT -s 203.0.113.93 --dport 22 -j ACCEPT'
info | |
In custom rules keys from /etc/ngcp-config/config.yml cannot be referenced. Thus, the values need to be manually looked up, hard coded, and kept in sync manually. This is by design of YAML. |
An example for NGCP firewall configuration in /etc/ngcp-config/config.yml enabling both the firewall subsystem and the logging facility may look like:
security: firewall: enable: 'yes' logging: enable: 'yes' file: '/var/log/firewall.log' tag: 'NGCPFW' policies: input: 'DROP' forward: 'DROP' output: 'ACCEPT' rules4: - '-A INPUT -i eth0 -j ACCEPT'
The sip:provider CE comes with some default passwords the user should change during the deployment of the system. They have been explained in the previous chapters of this handbook.
The Vagrant/VirtualBox/VMWare sip:provider CE images come with more default credentials which should be changed immediately:
important | |
Many NGCP services use MySQL backend. Users and passwords for these services are created during the installation. These passwords are unique for each installation, and the connections are restricted to localhost. You should not change these users and passwords. |
The sip:provider CE provides default, self-signed SSL certificates for SSL connections. These certificates are common for every installation. Before going to production state, the system administrator should provide SSL certificates for the web services. These certificates can either be shared by all web interfaces (provisioning, administrator interface and customer self care interface), or separate ones for each them can be used.
Set the path to the new certificates in /etc/ngcp-config/config.yml:
The sip:provider CE also provides the self-signed SSL certificates for SIP over TLS services. The system administrator should replace them with certificates signed by a trusted certificate authority if he is going to enable it for the production usage (kamailio→lb→tls→enable (disabled by default)).
Set the path to the new certificates in /etc/ngcp-config/config.yml:
The sip:provider CE allows you to protect your VoIP system against SIP attacks, in particular Denial of Service and brute-force attacks. Let’s go through each of those attacks and let’s see how to configure your system in order to face such situations and react against them.
As soon as you have packets arriving on your sip:provider CE server, it will require a bit of time of your CPU. Denial of Service attacks are aimed to break down your system
by sending floods of SIP messages in a very short period of time and keep your system busy to handle such huge amount of requests.
sip:provider CE allows you to block such kind of attacks quite easily, by configuring the following section in your /etc/ngcp-config/config.yml
:
security: dos_ban_enable: 'yes' dos_ban_time: 3600 dos_reqs_density_per_unit: 50 dos_sampling_time_unit: 2 dos_whitelisted_ips: [] dos_whitelisted_subnets: []
Basically, as soon as sip:provider CE receives more than 50 messages from the same IP in a time window of 2 seconds, that IP will be blocked for 3600 sec, and you will see in the the kamailio-lb.log
a line saying:
Nov 9 00:11:53 sp1 lb[41958]: WARNING: <script>: IP '1.2.3.4' is blocked and banned - R=<null> ID=304153-3624477113-19168@tedadg.testlab.local
The banned IP will be stored in kamailio memory, you can check the list via web interface or via the following command:
# ngcp-kamctl lb fifo sht_dump ipban
Excluding SIP endpoints from banning
There may be some SIP endpoints that send a huge traffic towards NGCP from a specific IP address. A typical example is a SIP Peering Server.
caution | |
sip:provider CE supports handling such situations by excluding all defined SIP Peering Servers from DoS protection mechanism. |
The NGCP platform administrator may also add whitelisted IP addresses manually in
/etc/ngcp-config/config.yml
at kamailio.lb.security.dos_whitelisted_ips
and
kamailio.lb.security.dos_whitelisted_subnets
parameters.
This is a very common attack you can easily detect checking your /var/log/ngcp/kamailio-proxy.log. You will see INVITE/REGISTER messages coming in with strange usernames. Attackers is trying to spoof/guess subscriber’s credentials, which allow them to call out. The very first protection against these attacks is: ALWAYS USE STRONG PASSWORD. Nevertheless sip:provider CE allow you to detect and block such attacks quite easily, by configuring the following /etc/ngcp-config/config.yml section:
failed_auth_attempts: 3 failed_auth_ban_enable: 'yes' failed_auth_ban_time: 3600
You may increase the number of failed attempt if you want (in same cases it’s better to be safed, some users can be banned accidentally because they are not writing the right password) and adjust the ban time. If a user try to authenticate an INVITE (or REGISTER) for example and it fails more then 3 times, the "user@domain" (not the IP as for Denial of Service attack) will be block for 3600 seconds. In this case you will see in your /var/log/ngcp/kamailio-lb.log the following lines:
Nov 9 13:31:56 sp1 lb[41952]: WARNING: <script>: Consecutive Authentication Failure for 'sipvicous@mydomain.com' UA='sipvicous-client' IP='1.2.3.4' - R=<null> ID=313793-3624525116-589163@testlab.local
Both the banned IPs and banned users are shown in the Admin web interface, you can check them by accessing the Security Bans section in the main menu. You can check the banned user as well by retrieving the same info directly from kamailio memory, using the following commands:
# ngcp-kamctl lb fifo sht_dump auth
The sip:provider CE is a very flexible system, capable of serving from hundreds to several tens of thousands of subscribers in a single node. The system comes with a default configuration, capable of serving up to 50.000 subscribers in a normal environment. But there is no such thing as a normal environment. And the sip:provider CE has sometimes to be tunned for special environments, special hardware requirements or just growing traffic.
info | |
If you have performance issues with regards to disk I/O please consider enabling the noatime mount option for the root filesystem. Sipwise recommends the usage of noatime, though remove it if you use software which conflicts with its presence. |
In this section some parameters will be explained to allow the sip:provider CE administrator tune the system requirements for optimum performance.
Table 16. Requirement_options
Option | Default value | Requirement impact |
---|---|---|
cleanuptools→binlog_days | 15 | Heavy impact on the harddisk storage needed for mysql logs. It can help to restore the database from backups or restore broken replication. |
database→bufferpoolsize | 64MB | For test systems or low RAM systems, lowering this setting is one of the most effective ways of releasing RAM. The administrator can check the innodb buffer hit rate on production systems; a hit rate over 99% is desired to avoid bottlenecks. |
kamailio→lb→pkg_mem | 16 | This setting affects the amount of RAM the system will use. Each kamailio-lb worker will have this amount of RAM reserved. Lowering this setting up to 8 will help to release some memory depending on the number of kamailio-lb workers running. This can be a dangerous setting as the lb process could run out of memory. Use with caution. |
kamailio→lb→shm_mem | 1/16 * Total System RAM | The installer will set this value to 1/16 of the total system RAM. This setting does not change even if the system RAM does so it’s up to the administrator to tune it. It has been calculated that 1024 (1GB) is a good value for 50K subscriber environment. For a test environment, setting the value to 64 should be enough. "Out of memory" messages in the kamailio log can indicate that this value needs to be raised. |
kamailio→lb→tcp_children | 8 | Number of TCP workers kamailio-lb will spawn per listening socket. The value should be fine for a mixed UDP-TCP 50K subscriber system. Lowering this setting can free some RAM as the number of kamailio processes would decrease. For a test system or a pure UDP subscriber system 2 is a good value. 1 or 2 TCP workers are always needed. |
kamailio→lb→tls→enable | yes | Enable or not TLS signaling on the system. Setting this value to "no" will prevent kamailio to spawn TLS listening workers and free some RAM. |
kamailio→lb→udp_children | 8 | See kamailio→lb→tcp_children explanation |
kamailio→proxy→children | 8 | See kamailio→lb→tcp_children explanation. In this case the proxy only listens udp so these children should be enough to handle all the traffic. It could be set to 2 for test systems to lower the requirements. |
kamailio→proxy→*_expires | Set the default and the max and min registration interval. The lower it is more REGISTER requests will be handled by the lb and the proxy. It can impact in the network traffic, RAM and CPU usage. | |
kamailio→proxy→natping_interval | 30 | Interval for the proxy to send a NAT keepalive OPTIONS message to the nated subscriber. If decreased, this setting will increase the number of OPTIONS requests the proxy needs to send and can impact in the network traffic and the number of natping processes the system needs to run. See kamailio→proxy→natping_processes explanation. |
kamailio→proxy→natping_processes | 7 | Kamailio-proxy will spawn this number of processes to send keepalive OPTIONS to the nated subscribers. Each worker can handle about 250 messages/second (depends on the hardware). Depending the number of nated subscribers and the kamailio→proxy→natping_interval parameter the number of workers may need to be adjusted. The number can be calculated like nated_subscribers/natping_interval/pings_per_second_per_process. For the default options, assuming 50K nated subscribers in the system the parameter value would be 50.000/30/250 = (6,66) 7 workers. 7 is the maximum number of processes kamailio will accept. Raising this value will cause kamailio not to start. |
kamailio→proxy→shm_mem | 1/16 * Total System RAM | See kamailio→lb→shm_mem explanation. |
rateomat→enable | yes | Set this to no if the system shouldn’t perform rating on the CDRs. This will save CPU usage. |
rsyslog→external_log | 0 | If enabled, the system will send the log messages to an external server. Depending on the rsyslog→external_loglevel parameter this can increase dramatically the network traffic. |
rsyslog→ngcp_logs_preserve_days | 93 | This setting will set the number of days ngcp logs under /var/log/ngcp will be kept in disk. Lowering this setting will free a high amount of disk space. |
tip | |
In case of using virtualized environment with limited amount of hardware resources, you can use the script ngcp-toggle-performance-config to adjust sip:provider CE configuration for high/low performance: |
root@spce:~# /usr/sbin/ngcp-toggle-performance-config /usr/sbin/ngcp-toggle-performance-config - tool to adjust sip:provider configuration for low/high performance --help Display this usage information --high-performance Adjust configuration for system with normal/high performance --low-performance Adjust configuration for system with low performance (e.g. VMs) root@spce:~#
The sip:provider CE platform provides detailed logging and log files for each component included in the system via rsyslog. The main folder for log files is /var/log/ngcp/, it contains a list of self explanatory log files named by component name.
The sip:provider CE is a high performance system which requires compromise between traceability (maximum amount of debug information being written to hard drive) and productivity (minimum load on IO subsystem). This is the reason why different log levels are configured for the provided components by default.
Most log files are designed for debugging sip:provider CE by Sipwise operational team while main log files for daily routine usage are:
Log file | Content | Estimated size |
---|---|---|
/var/log/ngcp/api.log | API logs providing type and content of API requests and responses as well as potential errors | medium |
/var/log/ngcp/panel.log /var/log/ngcp/panel-debug.log | Admin Web UI logs when performing operational tasks on the ngcp-panel | medium |
/var/log/ngcp/cdr.log | mediation and rating logs, e.g. how many CDRs have been generated and potential errors in case of CDR generation or rating fails for particular accounting data | medium |
/var/log/ngcp/kamailio-proxy.log | Overview of SIP requests and replies between lb, proxy and sems processes. It’s the main log file for SIP overview | huge |
/var/log/ngcp/kamailio-lb.log | Overview of SIP requests and replies along with network source and destination information flowing through the platform | huge |
/var/log/ngcp/sems.log | Overview of SIP requests and replies between lb, proxy and sems processes | small |
/var/log/ngcp/rtp.log | rtpengine related log, showing information about RTP communication | small |
warning | |
it is highly NOT recommended to change default log levels as it can cause system IO overloading which will affect call processing. |
info | |
the exact size of log files depend on system type, system load, system health status and system configuration, so cannot be estimated with high precision. Additionally operational network parameters like ASR and ALOC may impact the log files' size significantly. |
The easiest way to fetch information about a single call among the log files is the search for the SIP CallID (a unique identifier for a SIP dialog). The call ID is used as call marker in almost all the voip related log file, such as /var/log/ngcp/kamailio-lb.log , /var/log/ngcp/kamailio-proxy.log , /var/log/ngcp/sems.log or /var/log/ngcp/rtp.log. Example of kamailio-proxy.log line:
Nov 19 00:35:56 sp1 proxy[7475]: NOTICE: <script>: New request on proxy - M=REGISTER R=sip:sipwise.local F=sip:jdoe@sipwise.local T=sip:jdoe@sipwise.local IP=10.10.1.10:5060 (127.0.0.1:5060) ID=364e4676776621034977934e055d19ea@127.0.0.1 UA='SIP-UA 1.2.3.4'
The above line shows the SIP information you can find in a general line contained in /var/log/ngcp/kamailio-*:
In order to collect the full log related to a single call, it’s necessary to "grep" the /var/log/ngcp/kamailio-proxy.log using the ID= string, for example:
# grep "364e4676776621034977934e055d19ea@127.0.0.1" /var/log/ngcp/kamailio-proxy.log
The sip:provider CE platform provides several tools to collect SIP traces. It can be used the sip:provider CE ngrep-sip tool to collect SIP traces, for example to fetch traffic in text format from outbound and among load balancer, proxy and sems :
# ngrep-sip b
see the manual to know all the options:
# man ngrep-sip
The ngrep debian tool can be used in order to make a SIP trace and save it into a .pcap file :
# ngrep -s0 -Wbyline -d any -O /tmp/SIP_trace_file_name.pcap port 5062 or port 5060
The sngrep debian graphic tool as well can be used to visualize SIP trace and save them in a .pcap file :
# sngrep