arthurguru: Integrating PowerDNS with NicTool Server

Integrating PowerDNS with NicTool Server

Intro

This solution came about when I was removing every single DJB-ism (qmail, tinydns, daemontools) from an organisation. PowerDNS was the obvious replacement to tinydns as it has many nice features like modular back-ends, reporting, and excellent caching mechanisms. I also observed a significant improvement in speed when I switched over to it, and it also proved to be very reliable.
NicTool is an interesting beast. It's design is well thought out and it is simple to use. Its best features are the ability to delegate editing control of DNS zones to target users, export select DNS zones to select DNS servers, and export to multiple back-ends. My only criticism of NicTool is that it uses some old Perl modules that may require you to fetch the sources then build and install them, but I hear a rumour that the author is going to address these shortcomings.
There isn't much documentation on integrating NicTool with PowerDNS, and although the author did help me out with some integration issues and provided me with a Perl script, I decided to use a different approach altogether. One of my strict requirements for a DNS server was that it must continue to operate as a standalone service even after a failure of NicTool or its database. This meant that each of my DNS servers will have their own database and a means of refreshing them with the latest data from the source database that is managed by NicTool.

Unix: Choosing a PowerDNS back-end.

PowerDNS supports a number of back-ends.
My first configuration attempt was investigating the SQL back-end for PowerDNS. Installing a local SQL server on a DNS server is overkill but you could pull it off with a lightweight SQL server like SQLLite or BerkelyDB. The only challenge is writing a method of some kind that keeps these local databases fresh with updates from the source NicTool Server database, which isn't trivial.
My second configuration attempt was to install a customised version of a Perl script that the author of NicTool gave me to use as a PowerDNS back-end that connected to the NicTool database with read-only access and translated the NicTool database queries into PowerDNS compatible output. It was always my intention to mitigate the depedency on a central database server by enabling huge local cache resources on the PowerDNS host. This sort of worked ok but I didn't want to go into production with it as the biggest limiting factors were the first DNS queries for hosts were painfully slow until the PowerDNS cache built up, and I needed to install the SQL client and Perl modules on the PowerDNS host for the Perl script to work.
My third configuration attempt involved moving the complexity of that Perl script to the more powerful NicTool web server itself as that host already had the SQL client and Perl modules installed, and then run that Perl script (in a wrapper) as an inetd service. This reduced the complexity of the PowerDNS back-end shell script to just a couple of lines and improved the speed of the initial DNS queries significantly, but I was still not content to put this solution into production.
DNS servers are a critical piece of infrastructure, more critical than people give them credit for. After a complete power cycle in your computer room they should be the very first hosts to be brought back online and should be running before your application servers come online, otherwise all sorts of anomalies can occur. The last two attempts above put the PowerDNS source database on a web application server, and with the local PowerDNS cache cleared after a reboot you now have a catch 22 situation, that is, how does the NicTool application find its SQL server if the DNS server can't find its NicTool Server host? Your DNS servers need to be fully functional and independent of all other servers, this means they will each need to have their own permanent local databases.
Fortunately PowerDNS also supports BIND as a back-end, which is also convenient as the NicTool Sever also supports exporting to a BIND back-end. Unlike the original BIND software which is buggy and problematic, PowerDNS's implementation of BIND is quite reliable. This makes the solution much simpler as all we need to do now is configure the NicTool Server to export its database into a BIND folder on a regular basis, and if there are any changes then remote copy them to the BIND folders on your PowerDNS servers.

Unix: Configure the NicTool Server to export to a BIND folder

Use the NicTool Client to create a DNS server with a BIND back-end specifying a full pathname to the BIND folder. Create that folder on your NicTool Server host. Remember to note the pathname you used because NicTool writes that pathname into a BIND conf file called named.conf.nictool which is what you will point the PowerDNS back-end to. This means you will also have to use the same pathname for your PowerDNS hosts as you have configured in the NicTool Server.
It's more secure running the export as a non privileged user so create a user and group called nt_export:nt_export and give it write permission to the pathname folder above.
Copy a script called nt_export.pl from the NicTool Server bin folder to the parent folder of the pathname above.
NicTool DNS servers are referred to by their nsid rather than their host name (once you get familiar with the NicTool database schema you find that it is much easier working with the nsid rather than the nsname). Use the NicTool Client to list your NicTool DNS servers to identify their nsid values.
Manually run the NicTool nt_export.pl script using the nsid of your DNS server as an argument. This will perform a DNS server export using the method you defined in the NicTool configuration, which in our case will be BIND format.
For example, I performed the following steps on the NicTool Server host for a name server called ns1.mycompany.com:
addgroup nt_export
adduser nt_export

mkdir -p /usr/local/dns/ns1.mycompany.com/data-ns1.mycompany.com
chgrp -R nt_export /usr/local/dns/ns1.mycompany.com
chmod -R g+w /usr/local/dns/ns1.mycompany.com

mkdir -p /usr/local/dns/etc
echo "some_secret_password" > /usr/local/dns/etc/rsyncd.secrets.nt_export
chown root:nt_export /usr/local/dns/etc/rsyncd.secrets.nt_export
chmod 640 /usr/local/dns/etc/rsyncd.secrets.nt_export

# This script usually needs some modifications in relation to paths, make a copy and edit.
cp -p <path_to_NicTool_Server_bin_folder>/nt_export.pl /usr/local/dns/ns1.mycompany.com/

cd /usr/local/dns/ns1.mycompany.com/
sudo -u nt_export ./nt_export.pl -nsid <nsid> -force 
Now check the contents of the pathname you took note of above (the BIND folder) and ensure an export of DNS data actually occurred.

Unix: Configure the PowerDNS host to use BIND

Remote copy the BIND folder exported from the NicTool Server to your DNS host. Use the same pathname on your DNS host as you have on your NicTool Server host.
Configure PowerDNS to use BIND and point it to the named.conf.nictool file.
Edit pdns.conf (usually /etc/pdns/pdns.conf) with the following settings:
launch=bind
bind-config=/usr/local/dns/ns1.mycompany.com/data-ns1.mycompany.com/named.conf.nictool
bind-check-interval=300
While your editing this file you may also want to modify some other settings like cache, performance and reporting as follows:
cache-ttl=60
query-cache-ttl=60

local-address=<external_IP_address>

log-dns-details=off
log-failed-updates=off
loglevel=3
query-logging=yes

version-string=anonymous

webserver=yes
webserver-address=<internal_IP_address>
webserver-password=password
Start PowerDNS and make sure DNS lookups work (this assumes you already have a zone called mycompany.com defined in your NicTool Server).
service pdns start
service pdns status

nslookup mycompany.com ns1.mycompany.com
If you enabled the web server feature of PowerDNS you can point a web browser to it as http://<internal_IP_address>:8081/ and check out the report.

Unix: Synchronising your PowerDNS servers with the latest NicTool data

Write a shell script that performs the following:

Call NicTool's nt_export.pl script with the nsid as an argument to perform the DNS server export to a working BIND folder.

Check to see if there have been any changes inside the working BIND folder.

If there have been changes then remote copy the working BIND folder to the target DNS server associated with the nsid value.

If required then cycle the PowerDNS service.

I use rsync to do the remote copy of the BIND folder securely by using IP address restrictions and a secrets file. I also use rsync on the destination host to cycle PowerDNS if required. Here is how I did it.
On the NicTool Server host I created a script called /usr/local/sbin/nt_zonexfer.sh that looks as follows:
#!/bin/sh
#
# BSD License for nt_zonexfer.sh 
# Copyright (c) 2013, Arthur Gouros
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without 
# modification, are permitted provided that the following conditions are met:
# 
# - Redistributions of source code must retain the above copyright notice, 
#   this list of conditions and the following disclaimer.
# - Redistributions in binary form must reproduce the above copyright notice, 
#   this list of conditions and the following disclaimer in the documentation 
#   and/or other materials provided with the distribution.
# - Neither the name of Arthur Gouros nor the names of its contributors 
#   may be used to endorse or promote products derived from this software 
#   without specific prior written permission.
# 
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 
# POSSIBILITY OF SUCH DAMAGE.
#
#
# Extract fresh zone files and only upload and log if there has been a change.
#
# Requires 2 arguments:
#    arg1: nsid value. E.g. 4
#    arg2: Name Server fqdn host name. E.g. ns1.mycompany.com
#
# Must be run as the user nt_export from cron.
#
#
# Author: Arthur Gouros - 17/10/2012

# Globals
NSID="$1"
HOSTNAME="$2"

WORKDIR="/usr/local/dns/${HOSTNAME}"
ZONEDIR="data-${HOSTNAME}"
TRIGGER_FILE="${WORKDIR}/${ZONEDIR}/.pdns_cycle"
CHECKSUM="${WORKDIR}/.checksum_zones.md5"
CHECKSUM_NAMED="${WORKDIR}/.checksum_named.md5"
LOGTEMP="/var/tmp/nictool_${HOSTNAME}.log"
RSYNC="/usr/bin/rsync"
RSYNC_SECRETS="/usr/local/dns/etc/rsyncd.secrets.nt_export"
OPENSSL="/usr/bin/openssl"

# init
echo "--------------------------------" > ${LOGTEMP}
echo "`date`" >> ${LOGTEMP}

# Historically the nt_export.pl script was run from the working directory (i.e. a copy of the original was placed there).
cd ${WORKDIR}
if test $? -ne 0
then
  echo "ERROR: Could not change directory to ${WORKDIR}" >> ${LOGTEMP}
  exit 1
fi

# nt_export.pl leaves behind old zone files so purge everything and recreate.
/bin/rm -fr ${WORKDIR}/${ZONEDIR}

# Extract fresh zone files from the NicTool database.
./nt_export.pl -nsid ${NSID} -force >> ${LOGTEMP}
echo "" >> ${LOGTEMP}

# Check whether anything changed.
if test -s "${CHECKSUM}"
then
  prev_md5="`cat ${CHECKSUM}`"
else
  prev_md5="0"
fi

# Calculate a checksum from the latest data.
echo "Calculating new checksum." >> ${LOGTEMP}
new_md5="`cat ${WORKDIR}/${ZONEDIR}/* 2>/dev/null | ${OPENSSL} md5`"

if test "${new_md5}" != "${prev_md5}"
then
  # Transfer the zone files to the DNS server.
  echo "Checksum is different so transfering across zone files." >> ${LOGTEMP}

  # Check whether named.conf.nictool changed.
  if test -s "${CHECKSUM_NAMED}"
  then
    prev_named_md5="`cat ${CHECKSUM_NAMED}`"
  else
    prev_named_md5="0"
  fi

  # Calculate a checksum from the latest named.conf.nictool file.
  new_named_md5="`cat ${WORKDIR}/${ZONEDIR}/named.conf.nictool 2>/dev/null | ${OPENSSL} md5`"

  if test "${new_named_md5}" != "${prev_named_md5}"
  then
    # Set the trigger file to cycle pdns.
    echo "named.conf.nictool has changed, setting trigger file." >> ${LOGTEMP}
    touch ${TRIGGER_FILE}
    # Trigger file is always removed upon next invocation.

    # Update the named checksum file.
    echo "${new_named_md5}" > ${CHECKSUM_NAMED}
  fi

  # Note: The zone file path is hard coded in the named.conf.nictool file so we copy to a similar destination.
  ${RSYNC} -rz --delete-during --checksum --password-file=${RSYNC_SECRETS} ${ZONEDIR}/ nt_export@${HOSTNAME}::ns_zonedir/ >> ${LOGTEMP}
  
  # Update the checksum file.
  echo "${new_md5}" > ${CHECKSUM}

  # Display the logtemp file if there is a DNS change, hopefully you are sending all stdout/err to a log file in your crontab.
  cat ${LOGTEMP}
else
  echo "Checksum is the same so no zone transfer performed." >> ${LOGTEMP}
fi
On the NicTool Server host I run this script with 2 arguments via cron every 5 minutes.
*/5 * * * * nt_export /usr/local/sbin/nt_zonexfer.sh 4 ns1.mycompany.com >> /var/log/nt_zonexfer_ns1.log 2>&1
On the PowerDNS host (ns1.mycompany.com) I have the following definition in rsyncd.conf (usually /etc/rsyncd.conf):
[ns_zonedir]
  path = /usr/local/dns/ns1.mycompany.com/data-ns1.mycompany.com
  comment = NS zone files for PowerDNS
  use chroot = no
  uid = root
  gid = wheel
  read only = no
  auth users = nt_export
  timeout = 100
  secrets file = /etc/rsyncd.secrets
  post-xfer exec = /usr/local/sbin/pdns_cycle_check.sh ns1.mycompany.com
  hosts allow = <IP_address_of_the_NicTool_Server_host>
On the PowerDNS host (ns1.mycompany.com) edit rsyncd.secrets (usually /etc/rsyncd.secrets) and add the password you specified previously for rsync:
nt_export:some_secret_password
On the PowerDNS host (ns1.mycompany.com) I have the following script /usr/local/sbin/pdns_cycle_check.sh:
#!/bin/sh
#
# BSD License for pdns_cycle_check.sh 
# Copyright (c) 2013, Arthur Gouros
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without 
# modification, are permitted provided that the following conditions are met:
# 
# - Redistributions of source code must retain the above copyright notice, 
#   this list of conditions and the following disclaimer.
# - Redistributions in binary form must reproduce the above copyright notice, 
#   this list of conditions and the following disclaimer in the documentation 
#   and/or other materials provided with the distribution.
# - Neither the name of Arthur Gouros nor the names of its contributors 
#   may be used to endorse or promote products derived from this software 
#   without specific prior written permission.
# 
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 
# POSSIBILITY OF SUCH DAMAGE.
#
#
# Nictool subprocesses generate a trigger file if named.conf.nictool changes.
# The file is included with the zone files and rsynced to the target DNS server
# along with the changed zones. This serves as a trigger to tell PDNS to cycle.
#
# Requires 1 argument:
#    arg1: Name Server fqdn host name. E.g. ns1.mycompany.com
#
# Ideally this script is run via rsync post-xfer exec option.
# 
#
# Author: Arthur Gouros - 17/10/2012

this_host="$1"
TRIGGER_FILE="/usr/local/dns/${this_host}/data-${this_host}/.pdns_cycle"

if test -f "${TRIGGER_FILE}"
then
  # Wait for rsync to finish doing it's thing.
  sleep 3
  # Remove the trigger file.
  /bin/rm -f ${TRIGGER_FILE}
  # reload named.conf.nictool
 /usr/bin/pdns_control cycle
fi

exit 0
Unix: Test that changes in NicTool actually flow through to your DNS hosts

Log into your NicTool Client and start adding/deleting/modifying zone records.
Wait 5 minutes. In our example you can tail the log file on the NicTool Server host.
tail -f /var/log/nt_zonexfer_ns1.log
If you are adding or deleting zone records you can also tail the log file on the PowerDNS host to see if it is being cycled.
tail -f /var/log/messages
Confirm the changes went across by either looking inside the BIND directory on both the NicTool Server host and the PowerDNS host, or by running a nslookup or dig query. For example:
dig newhost.mycompany.com
dig www.newzone.com
If all goes well then you can scale this solution to any number of PowerDNS hosts that are all centrally managed through the NicTool Client.

Unix: Closing the loop

The next logical step for the above solution is to close the loop by writing an automatic test which will check that central updates have actually flowed through to your DNS servers. This step is important because in largish organisations with multiple sysadmins you will invariably come across situations where the left hand doesn't know what the right hand is doing, for example with the above solution if a sysadmin accidently firewalled out the rsync protocol between the NicTool Server host and a DNS host on a separate network, then all zone updates to that DNS host will be blocked and the affected DNS host will be left serving out-of-date data until someone (usually the customer) logs a complaint.
There are many ways to test that your DNS hosts are serving the latest changes in NicTool, and in consultation with the original author of NicTool I came up with the following approach.

Create a database user that has read only access to the NicTool database e.g. nictool_report. Use this database user for all your NicTool queries.

For each name server, query the database and return the zone name of the last zone record that was changed.

NicTool always updates a zone's serial with a timestamp whenever a sysadmin performs any changes to that zone, so now run a database query to retrieve the zone's serial.

Using DNS protocols, execute a SOA record lookup of the last edited zone on the targeted DNS host and compare the serial returned from DNS with the serial returned from the database.

If the serials are not equal then generate an alert.

Create a shell script that performs the above actions automatically.

Just be aware that in my example setup here, PowerDNS is configured with an update interval of 300 seconds, and nt_zonexfer.sh in cron is configured with an update interval of 5 minutes, therefore the worst case situation for a NicTool update to become effective will be 10 minutes.

Then it occurred to me that the above could just as easily be implemented from within the NicTool Client itself, which is really the best place for such a report. So I contacted the author of NicTool and suggested this idea to him. He agreed, and hopefully it will become a feature that will be available in a near future release of NicTool.