A Jenkins Tip for complicated executes

Shares a practical Jenkins scripting tip for preserving fail-fast behavior when shell steps are moved into standalone scripts. Recommends using bash flags like `-e` and `-x` to keep deployments safer and easier to debug.

Coveros Staff

April 30, 2015

Dear Loyal Readers,

Everyone uses Jenkins.  It is like Pringles on any project; once you start using it, who doesn’t want a nice framework/web UI for scheduling and polling tasks!?

My first use of Jenkins was a simple CI build where every job simply executed the classic Makefile:

make make test make install

But as we started using Jenkins for more and more complicated tasks, and especially for push-button deployments, our execution shell block got longer and longer.  There are some cool plugins to handle more complicated jobs- This one saves all config changes internally, and this one syncs the config changes to a git repo.

But this is Jenkins, so why not just wrap up our shell commands into a single script and put it in a repository? That way it will be tracked in source control and not in the Jenkins config.xml.

We ran into one problem. We lost my favorite Jenkins feature: every line error checking. Jenkins will fail at the FIRST error. In Jenkins, an execute-shell block like this:
echo "Starting" false echo "Finishing"
would fail:

+ echo Starting
Starting
+ false
Build step 'Execute shell' marked build as failure
Finished: FAILURE

But once I put that all in a single script, we lost the error checking:

+ ./runme.sh
Starting Build
We should never get here
Finished: SUCCESS

For deployments, this is an important feature we don’t want to lose.  If any preceding command fails, we need to stop and fail immediately. We now have to write a bunch of ugly bash error checking to replace the whole execute block with “./build_me.sh” …or do we?

#!/bin/bash -e

I was so happy to find this bash flag: -e

This shebang line option tells bash to return non-zero at the first sign of trouble. Now all my Jenkins scripts die as soon as there is a problem 🙂

There is only one step left that all ancient bash nerds know: -x

The “-x” flag is bash’s debug mode that _echo_s every command.  Now my scripts can be saved from that extra crappy bash error checking and those extra _echo_s I would have to add.

Coveros Staff

Coveros Staff

This post represents the collective insights of the Coveros team. Our staff consists of software experts who bring deep experience in secure agile development, DevOps, testing, and software quality. Over the past 20 years, Coveros has trained more than 30,000 professionals and worked with half of the Fortune 100 companies on mission-critical software development challenges. We draw on this extensive experience to share practical insights, proven strategies, and real-world solutions that help organizations build better software faster and more securely.