For any service provider it is important to maintain a reliable backup policy as it enables prompt services restoration after any force majeure event. Hence, we strongly suggest you to configure a backup procedure. The Sipwise C5 can be integrated with any Debian compatible backup software.
This is the most important data in the system. All subscriber and billing information, CDRs, user preferences, etc. are stored in the MySQL server. It is strongly recommended to have up-to-date dumps of all the databases .
The system configuration folder /etc/ngcp-config/ must be included in the backup as well. It contains the system specific configuration (like SSL keys). Also you might have some local modifications. We suggest backing up the whole /etc folder to preserve the etckeeper history to be able to answer when and who changed particular configuration files in the past.
The /home/jail/home/cdrexport directory contains the exported CDRs. It depends on your call data retention policy whether or not to remove these files after exporting them to an external system.
In the worst case scenario, when the system needs to be recovered from a total loss, you only need 4 steps to get the services back online:
ngcpcfg apply 'restored the system from the backup'
| important | |
All existing data will be wiped out! Use this script only if you want to clear all previously configured services and start configuration from scratch. |
To reset database to its original state you can use a script provided by CE: * Execute ngcp-reset-db. It will assign new unique passwords for Sipwise C5 services and reset all services. The script will also create dumps for all Sipwise C5 databases.
Sipwise Sipwise C5 offers ways to cleanup, backup or archive old accounting
data — i.e. CDRs — that is not necessary for further processing any more, or must
be deleted according to the law. There are some Sipwise C5 components designed for this
purpose and they are commonly called cleanuptools. These are configurable
scripts that interact with NGCP’s accounting and kamailio databases, or remove
exported CDR files in order to clean or archive the unnecessary data.
The configuration parameters of cleanuptools are located in the main Sipwise C5 configuration
file: /etc/ngcp-config/config.yml. Please refer to the config.yml file description:
Cleanuptools Configuration Data for configuration parameter
details.
In case the system administrator needs to modify some configuration value, the new configuration must be activated in the usual way, by running the following commands:
> ngcpcfg apply 'Modified cleanuptools config'
As a result new configuration files will be generated for the accounting database and the exported CDR cleanup tools. Please read detailed description of those tools in subsequent sections of the handbook.
The Sipwise C5 system administrator can also select the time when cleanup scripts are
run, by modifying the schedule here: /etc/cron.d/cleanup-tools
The script responsible for cleaning up the database is: ngcp-cleanup-acc
The configuration file used by the script is: /etc/ngcp-cleanup-tools/acc-cleanup.conf
An extract from a sample configuration file is provided here:
############ batch = 10000 archive-target = /ngcp-data/backup/cdr compress = gzip username = dbcleaner password = rcKamRdHhx7saYRbkJfP host = localhost port = 3306 redis-batch = 10000 redis-port = 6379 connect accounting keep-months = 2 use-partitioning = yes timestamp-column = cdr_start_time backup cdr_cash_balance_data backup cdr_time_balance_data backup cdr_relation_data backup cdr_tag_data backup cdr_mos_data backup cdr_export_status_data backup cdr_group timestamp-column = first_cdr_start_time backup cdr_period_costs timestamp-column = start_time backup cdr archive-months = 2 archive cdr_cash_balance_data archive cdr_time_balance_data archive cdr_relation_data archive cdr_tag_data archive cdr_mos_data archive cdr_export_status archive cdr_group archive cdr_period_costs archive cdr cleanup-days = 1 use-partitioning = no timestamp-column = cdr_start_time cleanup int_cdr_cash_balance_data cleanup int_cdr_time_balance_data cleanup int_cdr_relation_data cleanup int_cdr_tag_data cleanup int_cdr_group cleanup int_cdr_export_status timestamp-column = start_time cleanup int_cdr connect kamailio time-column = time cleanup-days = 90 cleanup acc connect-redis 21 connect kamailio time-column = time_hires cleanup-days = 3 cleanup-mode = mysql cleanup-redis acc:entry::* # Clean up after mediator by deleting old leftover acc entries and deleting # old entries out of acc_trash and acc_backup connect kamailio time-column = time cleanup-days = 30 cleanup acc_trash cleanup acc_backup maintenance = no
The configuration file itself contains a detailed description of how database cleanup script works. It consists of a series of statements, one per line, which are going to be executed in sequence. A statement can either only set a variable to some value, or perform an action.
There are 4 types of actions the database cleanup script can take:
These actions are discussed in following sections.
A generic action is connecting to the proper database: connect <database name>
The database cleanup tool can create monthly backups of data in the accounting
database tables by moving old records to separate tables named: cdr_YYYYMM.
The instruction in the configuration file looks like: backup <table name>,
by default and typically it is: backup cdr
Configuration values that govern the backup procedure are:
time-column: The name of the column in the table to use for determining which month a record belongs to. Must be a "datetime" column.
timestamp-column: The name of the column in the table to use for determining which month a record belongs to. Must be a "decimal(13,3)" column.
use-partitioning: If a table is partitioned using the time-column (timestamp-column) and the value is set
to "yes", then moving/deleting records are instant operations, done by managing the partitions.
Otherwise the usual method is used to delete/move records by chunks.
batch: How many rows to include per transaction when processing in chunks. If unset or ⇐ 0, it does them all at once.
keep-months: How many months worth of records to keep in the table and not move into the monthly backup tables.
IMPORTANT: Months are always processed as a whole and this specifies how many months to keep AT MOST. In other words, if the script is started on December 15th and this value is set to "2", then all of December and November is kept, and all of October will be moved out.
The database cleanup tool can archive (dump) old monthly tables. The statement
used for this purpose is: archive <table name>, by default and typically it is:
archive cdr
This creates an SQL dump out of older tables created by the backup statement
and drop them from database afterwards. Archiving uses the following configuration
values:
archive-months: Uses the same logic as the "keep-months" variable.
If set to "12" and the script was started on December 15th, it will start
archiving with the December table of the previous year. Archiving continues
month by month, going backwards in time, until the script encounters a
missing table.
archive-target: Target directory for writing the SQL dump files. If
explicitly specified as "/dev/null", then no actual archiving will be
performed, but instead the tables will only be dropped.
compress: If set to "gzip", then gzip the dump files after creation. If
unset, do not compress.
host, "username" and "password": As dumping is performed by an external
command, those variables are reused from the "connect" statement.
The database cleanup tool may do database table cleanup without performing backup.
In order to do that, the statement: cleanup <table name> is used. Typically this
has to be done in kamailio database, examples:
cleanup acc
cleanup acc_trash
cleanup acc_backup
The cleanup statement works exactly like the backup statement, but doesn’t
actually backup anything, but rather only deletes old records. Additional configuration parameters
required by the cleanup procedure:
cleanup-days: Any record older than these many days will be deleted.
With the advent of persisting kamailio.acc record data in a redis keystore,
a separate cleanup-redis <key pattern> statement was introduced. It will remove
old redis entries, whose keys match the given redis SCAN pattern. Typically this
has to be done for the redis 21 database (acc records), eg.:
cleanup-redis acc:entry::*
connect-redis <redis database number> has to be used instead of connect +<mariadb database name>, which is needed to initially connect before any <backup>, <archive> and <cleanup> operations.
The cleanup-redis statement works similar to the cleanup statement, with some additional options below:
time-column: The name of the field in a redis entry denoting the record timestamp in epoch seconds.
cleanup-mode: If set to "delete", aged entires will be simply removed. If set to "mysql", they will be inserted into a database table first. The latter requires connect to open the database additionally.
redis-batch: Chunk size of redis entries with matching keys to look at. Note that redis processing always works in chunks, there is no partitioning.
cleanup-days: Any entry older than these many days will be deleted.
The script responsible for cleaning up exported CDR files is: ngcp-cleanup-cdr-files
The configuration file used by exported CDR cleanup script is: /etc/ngcp-cleanup-tools/cdr-files-cleanup.yml
A sample configuration file is provided here:
enable: no
max_age_days: 30
paths:
-
path: /home/jail/home/*/20[0-9][0-9][0-9][0-9]/[0-9][0-9]
wildcard: yes
remove_empty_directories: yes
max_age_days: ~
-
path: /home/jail/home/cdrexport/resellers/*/20[0-9][0-9][0-9][0-9]/[0-9][0-9]
wildcard: yes
remove_empty_directories: yes
max_age_days: ~
-
path: /home/jail/home/cdrexport/system/20[0-9][0-9][0-9][0-9]/[0-9][0-9]
wildcard: yes
remove_empty_directories: yes
max_age_days: ~The exported CDR cleanup tool deletes CDR files in the directories provided in the configuration file, if those have already expired.
Configuration values that define the files to be deleted:
enable: Enable (yes) or disable (no) exported CDR cleanup.
max_age_days: Gives the expiration time of the exported CDR files in days.
There is a general value which may be overridden by a local value provided at a
specific path. The local value is valid for the particular path only.
paths: an array of path definitions
path: a path where CDR files are to be found and deleted; this may contain wildcard
characters
wildcard: Enable (yes) or disable (no) using wildcards in the path
remove_empty_directories: Enable (yes) or disable (no) removing empty
directories if those are found in the given path
max_age_days: the local expiration time value for files in the particular path