To sum up: the primary issue was that whitespace was interfering with the knife upload command. To bypass this, my colleague, Rich, wrote a custom script. This script leveraged Ruby, specific Chef gems, and direct Chef API calls to manipulate files and create the data bag directly, skipping the standard upload process entirely.
Seeing how well this worked, I decided to adopt the same pattern to manage the .p12 certificates on my current project. I copied his script, ran it locally, and it appeared to be a success—I could even see the certificate stored in the data bag in its binary form. I did the standard victory dance, ready to buy Rich a drink for saving me a day’s worth of work. However, when I confidently ran my Chef recipe to deploy the certs, everything went south. I was met with a "Chef explosion" and the following error:
FATAL: Yajl::ParseError: lexical error: invalid bytes in UTF8 string.
{"json_wrapper":"0\b\u0007 \u0003\u0002\u000
(right here) ------^
The type of PEM file Rich was working with is text at heart. The cert is encoded as base64 text with some plain text metadata. The only issue that Rich was having had to do with white space. After analyzing the error and the contents of the file I made an educated guess that since a p12 is a binary file, there is probably a character in the p12 cert that the json lexical analyzer is throwing an error on while decoding the databag. To test this theory I took the sane cert and base64 encoded it. I then used Rich’s script to upload it and ran the chef script again. While my explanation may not be 100% correct my chef script is now executing correctly with the base64 text. Unfortunately, I now have two issues:
1. I need to base64 encode the cert
2. The cert is not usable in the format my script is leaving it at the moment.
To solve the whole problem we need to make a few assumptions. First you need to have the ruby base64 library and secondly you need to have a command line base64 utility on the system. To solve the first issue we need to change the knife script. Since we are encrypting the files, we are already reading them into the knife script. On line 41 of the original script, sure enough, we are reading in the whole file. With a slight modification from:
content = File.read(filename)
bag_item[:content] = content
to:
content = File.read(filename)
base64_content = Base64.encode64(content)
bag_item[:content] = base64_content
We will now make the encrypted databag contain a base64 version of what ever it is we want to upload. This is great for getting the data in and out of a databag, but now when the data is used I need to account for the base64ness. While I am sure there is many cleaner chef like ways to implement what I want, I was limited on time so I needed to settle for the following chef code in my recipe:
execute "decode #{cert}" do
command "base64 -d /tmp/#{cert}.base64 > /tmp/#{cert}.pem"
end
And viola, I can use databags to manage my binary files.
