Status: Now have secured private IP communications using LetsEncrypt ACME with DNS-01 wildcard validation

Domain Name Service (DNS) along with ARP (Address Resolution Protocol) are part of the fundamental foundations of TCP/IP protocol family. It does not matter if you are setting up Web Servers, Mail Servers, OpenStack or Kubernetes clusters, if you are doing anything non-trival then DNS will be part of the solution.

So this "diversion to DNS" is driven by a few things:

• Apple MacOS Mojave - is retiring its simple to use dns functions
• Private Kubernetes - needs to have mechanism to allocate IP addresses and also provide certificates for these
• DNS Update - as you spin up new machines you need way to get these added to DNS and the official way to do this is via DNS Update (rfc2136).

My three use case for these notes are are:

1. Provide Web Based Management interface for internal domain addressing
2. Support generation of LetsEncrypt certificates for internally addressed machines
3. Provide DNS, DCHP functions for: Kubernetes, Helm and MAAS

The non-functional need is that the solution must be able to meet expectations of an enterprise solution (i.e. is secure, uses real IP addresses and domain names.

Building Blocks:

BIND 9 - the most prevalent DNS server, with support for all required features

DNS Management Tools (I looked at a number including):

• NICTool - Web based DNS Management solution
• Admin4 - python based tool that runs on MacOS, Windows & Linux that does DNS and a number of other services administration
• gadmin-bind - Gnone admin tool with supports DNS management
• DLZ - a BIND extension that loads BIND zones directly from database

LetsEncrypt - the free and automatic certification authourity

CertBot - the agent for implmenting ACME certificate issuing protocol

Apache - my default web server

FreeBSD - as NICTool requires source installation, I am using FreeBSD (Free Berkeley Software Distribution) as it goes naturally with BIND (Berkeley Internet Name Domain (Daemon)) ;-)

Linux / MacOS - to test the non web based DNS Management tools

Architecture - is something like this...

The architecture to support managed private IP address space for (microk8s) Kubernetes with Registry and Help will look something like this. I say something, because the final way it looks will be driven by its implementation.

As can be seen from this high level view DNS and its management requires quite a few resources (multiple public services, a private internal subnet master and a management machine).

NOTE: While illustrated as "Machines" all of this is managed virtually and some of VMs will collapse into Kubernetes deployable containers.

DNS Management Tools

I tested a number of these. My initial plan was to use NICTool (based on the Wikipedia DNS Management Software listings) got thwarted due to issues with either user or software, I did not have time determine the cause of fault.

Here is summary of result (and links to more detailed instructions):

NICTool - I had issues possibly due to using more recent version of MySQL DB.

Admin4 - I kept getting DNS Update error, even though "tsig-key" was correct and verified using "dnsupdate utility".

Gadmin-bind - requires admin to be on same machine as BIND server, so I did not try this, it also appears be not maintained...

dlz - DLZ (Dynamically Loaded Zones) is an extension that works directly with BIND. It provides direct integration from a database (BDB, PostgreSQL, MySQL and others) to BIND to support dynamic loading of data.  While not a DNS Management tool itself by having zone data managed directly in database you can use general DB admin tool to manage your zone data and this is then loaded into BIND.  I have yet to test this approach.

LetsEncrypt Wildcard Certificates and BIND (DNS Update)

1. Build the LetsEncypt agent available. For wildcard certificates (required for private IP non internet facing machines, the port you need is in "/usr/ports/security/py-certbot-dns-rfc2136/"
2. Configure BIND DNS server. In my example I am using "ME.in.domain.com" for DNS server (built as above). As DNS server is behind NAT'ing firewall you need to configure port forwarding from Firewall to DNS server. You also need to configure zone delegation record in name server for "in.domain.com" within "domain.com" DNS. This way LetsEncrypt can find delegation record for internal DNS server and its contact IP address.
3. Generate the security key for dns-update policy. Is this example this is done using "tsig-key" utility (as "dnssec-keygen" is deprecated with BIND 9.14 and you get error:  

# dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST dnssec-key.
dnssec-keygen: fatal: unknown algorithm HMAC-SHA512
#
# ... deprecated, use tsig-key instead
#
# tsig-keygen -a hmac-sha512 tsig-key.acme. > /usr/local/etc/namedb/tsig-key.acme.key
#
# cat tsig-key.acme.key
key "tsig-key.acme." {
	algorithm hmac-sha512;
	secret "z/f8JKIJoXwhuuSwCwEEBu8l1Sg0MKA==";
};

Now you have your BIND key you need to configure your server to use this. The configuration file for this is: named.conf, here is example configuration referencing the key file (snippet) above:

include "/usr/local/etc/namedb/tsig-key.acme.key";

zone "YOUR.DOMAIN" {
  type master;
  file "/usr/local/etc/namedb/dynamic/db.YOUR.DOMAIN";
  allow-transfer {
    XXX.XXX.XXX.XXX;
  };
  update-policy {
    grant tsig-key.acme. name _acme-challenge.YOUR.DOMAIN. TXT;
  };
  allow-query {
    any;
  };
};

You will note that the generated key file (called: tsig-key.acme.key in this example) is now included into named.conf and the key name (called: "tsig-key.acme." in this example) is referred to in the "update-policy" directive. The policy directive is restricted to only allow changes to "IN TXT" record for: "_acme-challenge.YOUR.DOMAIN". This is the record that LetsEncrypt will use for its validation process.

3. Restart you bind server and test that your DNS update is working:

#
# // Write al small update script:
# cat test-nsupdate-01.txt
server XXX.XXX.XXX.XXX
debug yes
zone YOUR.ZONE.
update delete _acme-challenge.YOUR.DOMAIN. TXT
update add _acme-challenge.YOUR.DOMAIN. 10800 IN TXT "test record"
show
send
#
# nsupdate -k tsig-key.acme.key -v test-nsupdate-01.txt
...
...
;; TSIG PSEUDOSECTION:
tsig-key.acme. 0	ANY	TSIG	hmac-sha512. 1570962126 300 64 bskfj$@$E+CVA== 13742 NOERROR 0 
#
# // check you dns server
#
# dig -t TXT _acme-challenge.YOUR.DOMAN
...
...
;; ANSWER SECTION:
_acme-challenge.YOUR.DOMAIN. 10800 IN TXT "test record"
...
#
# // Ok result is good so lets get certificate
#

4. Get your wild card certificate (which requires DNS-01 validation). This requires another configuration file to provide LetsEncrypt with target server and key data. See example below:

# cat rfc2136.ini
dns_rfc2136_server = YOUR.TARGET.DNS.SERVER
dns_rfc2136_port = 53
dns_rfc2136_name = tsig-key.acme.
dns_rfc2136_secret = kOF75Hw==
dns_rfc2136_algorithm = HMAC-SHA512
#
# // Note: key name is same as prior examples
#
# certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /usr/local/etc/namedb/rfc2136.ini --dns-rfc2136-propagation-seconds 10 --debug-challenges
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): *.YOUR.DOMAIN
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for YOUR.DOMAIN
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /usr/local/etc/letsencrypt/live/in.YOUR.DOMAIN/fullchain.pem
   Your key file has been saved at:
   /usr/local/etc/letsencrypt/live/YOUR.DOMAIN/privkey.pem
   Your cert will expire on 2020-01-11. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

5. That required quite a bit of configuration.... just check the letsencypt directory for your certificate and key files (/usr/local/etc/letsencrypt/live/YOUR.DOMAIN/).

References & Links:

LetsEncrypt - your go to place for certificates, not just free but automated as well

Let's Encrypt Wildcard Certificates on FreeBSD with BIND DNS Validation - useful set of instructions for getting DNS-01 Validation going

NICTool - DNS Management Software

DLZ - BIND extention to load data from database rather than zone file

ACME Protocol - Underlying protocol used by LetsEncrypt and CertBot

MySQL TIMESTAMP DEFAULT - Info about MySQL 5.7 sql_mode issue.

DNS and BIND (5th Ed) - By Cricket Liu & Paul Albitz (picture is from cover)

nsupdate / tsig-gen / ddns-confgen - bind utilities

rfc2136 - Dynamic Updates in the Domain Name System (DNS UPDATE)

DNS Management Software - Wikipedia section provide summary of some DNS Management software (not comprehensive)

DNS Tools - from isc.org page

Gadmin - set of admin tools for Gnome, appears to be unmaintained

NOTE 1: Setting up authoritative and secure DNS with management of private certificates is non-trivial, so allow some time.