Some DNS automation (bind)

Dear Loyal Readers,

I’ve been using bind in AWS for a while now.  Initially route53 created publicly available DNS entries, but recently, AWS added the private/vpc-aware Route 53.  This makes our security team happy! Sadly, the record propagation delay was causing us problems (long delays) in our ability to quickly spin up new instances.

We might have been able to live with long instance creation times, but then we had the problem of being on a private network.  Our project was not completely self-contained, and we needed to interact with other groups.  Just as we needed to know their records, other groups would need to know ours.  So we needed to push them up to the main DNS server.

I was able to train some people to manually edit zone files, but first they kept forgetting to make the reverse lookup (PTR) records. So I scripted this for them:

#!/usr/bin/env ruby
revlook={}
require 'date'
subdomain="chowski.local"
ls=IO.readlines(subdomain);
ARECORD_REX = /^(?\S*)\s*(!?IN\s*)?A\s*(?\d+\.\d+\.\d+)\.(?\d+)$/
revlook={}
ls.reject! { |l| l =~ /^[#;].*/ }.each { |l|
  m = l.match( ARECORD_REX )
  if m
    key = m[:subnet].split(".").reverse.join(".")
    revlook[key]=[] if revlook[key].nil?
    revlook[key] << [ m[:forth], "#{m[:host]}.#{subdomain}" ]
  end
}                                                                                                                     
 
 
#el#136.103.10.in-addr.arpa.        IN      NS      ns.aws.chowski.local.
#el#37      IN      PTR     el2-gss-chf1.aws.chowski.local.

puts "Make sure you have (/etc/named.conf):\n"
revlook.keys.each { |revsubdomain|
  revfile=File.open("#{revsubdomain}.in-addr.arpa","w")
  id=DateTime.now.strftime("%Y%m%d%H%m")
 
  header =<<MULTI
@ IN SOA #{revsubdomain}.in-addr.arpa. el2-gss-dns1.#{subdomain}. (
#{id}; Serial
43200 ; Refresh
3600 ; Retry
3600000 ; Expire
2592000 ) ; Minimum
 
#{revsubdomain}.in-addr.arpa.        IN      NS      el2-gss-dns1.#{subdomain}. 
MULTI 
  revfile.puts header
  revlook[revsubdomain].each { |l| revfile.puts "#{l.first}\tIN\tPTR\t#{l.last}." }
  revfile.close
  puts namedconf
} #keys.each

Next, we needed dynamic DNS (1) (2), because the networking (including DNS) needs to be setup (automatically) before our Configuration Management tool can begin to install the middleware.

RHEL Tip: 
  chown named:named /var/named/
  sudo chmod 770 /var/named

Then last week happened.

The A records from AWS’s databases (RDS) and load balancers (ELB) have a low expiration (60 seconds).  I guess AWS needs its SaaS offerings to be more dynamic as they reserve the right to rebuild the front end if they detect a problem.  Also, ELBs will scale up if need be (and so cover more IPs).  This constant cache expiration exacerbated a problem between my local bind server and the upstream private DNS cluster.  Just like in testing, I want to reduce my dependence on any external factors.  I have no control over the upstream system, and I always want to be in control.

My solution:  Cache the DNS hits myself. In the following script, I’m going to make A records in my sub-domain for every ELB/RDS instance in AWS (instead of CNAMEs).

1. If I get a lookup miss, or the ip hasn’t changed, then I can just not update it, and bind continues to serve up the last known ip.
2. If I get an difference, I’ll build the nsupdate script, and execute it against the local bind server (using the private key for authentication).

#!/usr/bin/env ruby
require 'resolv'
require 'tempfile'
require 'socket'

SUBDOMAIN="aws.chowski.local"
aws_alist = %w{ myexampledb.a1b2c3d4wxyz.us-west-2.rds.amazonaws.com myexampleelb.a1b2c3d4wxyz.us-east-1.rds.amazonaws.com }

Dir.chdir( File.dirname $0 )

if ARGV.include?("debug")
  jdebug=true
  outstream=STDOUT
  outstream.puts "STDOUT used in debug"
  ARGV.delete("debug")
else
  jdebug=false
  outstream=File.open("#{$0}.log","w")
end
outstream.sync=true

if ARGV.include?("force")
  jforce=true
outstream.puts "Force update enabled, skipping cache check"
  ARGV.delete("force")
else
  jforce=false
end

unless ARGV.count == 1
  outstream.puts "Usage: #{$0} uploader.key"
exit 1
  else
  keypath=ARGV[0]
end
raise "keyfile(#{keypath}) does not exist" unless File.exists?(keypath)

myip = Socket.ip_address_list.select { |ai| ai.ipv4? }.map { |ai| ai.ip_address }.reject { |s| s == "127.0.0.1" }.first
outstream.puts "Start: #{Time.new}"
outstream.puts "Using #{myip}"
r_resolver=Resolv::DNS.new( :nameserver => [ '10.79.154.80','10.239.138.80' ], :search=>[SUBDOMAIN], :ndots=>1 )
l_resolver=Resolv::DNS.new( :nameserver => [ "#{myip}"], :search=>[SUBDOMAIN], :ndots=>1 )

h={}
aws_alist.each { |arecord|
  host = arecord.split(".")[0]
  outstream.print "Resolving #{host}: "
  new_ips = r_resolver.getaddresses(arecord).map { |cls| cls.to_s }.sort
  m = host.match /internal-(.*)-\d+/
  host=m[1] if m #rewrite for internal elb shortname
  old_ips =  l_resolver.getaddresses(host).map { |cls| cls.to_s}.sort
  outstream.print "-- OLD(#{old_ips.join(",")}) & NEW(#{new_ips.join(",")}) -- " if jdebug
  if jforce
    outstream.puts " forced"
    h[ arecord ] = { :host=>host, :ip=>new_ips }
  end

  if new_ips.empty?
    outstream.puts "No response, skipping"
    next
  elsif old_ips == new_ips
    outstream.puts "Same response, skipping"
    next
  else
    outstream.puts new_ips.join(", ")
    h[ arecord ] = { :host=>host, :ip=>new_ips }
  end #empty?
}

## zone aws.chowski.local
# nsupdate -k Klocalhost.+157+61824.private  -v

nsupdate_s= "server #{myip}\n"
h.each { |arecord, arecord_val |
  nsupdate_s+= "update delete #{ arecord_val[:host] }.#{SUBDOMAIN} a\n"
  nsupdate_s+= "update delete #{ arecord_val[:host] }.#{SUBDOMAIN} cname\n"
  arecord_val[:ip].each { |ip|
    nsupdate_s+= "update add #{ arecord_val[:host] }.#{SUBDOMAIN} 150 a #{ip}\n"
  }
}

nsupdate_s += "send\n"
outstream.puts "Writting update script to ./nsupdate.send"
IO.write("nsupdate.send", nsupdate_s)
cmd="nsupdate -k #{keypath} -v nsupdate.send"
outstream.puts "executing: #{cmd}"
outstream.puts `#{cmd}`
outstream.puts "Stop: #{Time.new}"
outstream.close

Until the Robots reduce us to oil for fuel (like the dinosaurs),
Jonathan Malachowski