There is an old saying in testing: Automating a bad test only gives you bad results faster. The same thing applies to DevOps.
Lots of DevOps initiatives focus on speed and frequency of deployment without an emphasis on quality. Bad testing practices in DevOps only deploys buggy software faster. Here are some ideas about how to avoid this situation.
Produce Working Software
Agile principles make it clear that the only measure of progress is the frequent delivery of working software. This means a user story is not complete until it has been built according to its acceptance criteria, adequately tested, and approved by the product owner.
Make sure your definition of done for both individual user stories and entire sprints includes adequate functional and nonfunctional testing against story, integration, and end-to-end testing criteria. Use continuous regression testing to make sure new code does not break old code. Also, demand that your developers do adequate unit testing so implementation bugs are caught early in the process.
Move toward Producing Releasable Software
DevOps expands on that last principle by advocating that all software also be releasable—including being adequately tested in a production (or production-like) environment.
Traditionally, access to production environments has been out of the reach of development teams, but this is changing quickly. By embracing a DevOps philosophy that integrates release management, environment and production support, and application monitoring into your teams, frequent testing on production environments is now more realistic and can be done as part of each sprint.
If your production environments are too costly or complex to fully replicate, establish production-like environments that still allow some late-lifecycle testing to be shifted left and performed continuously. And you don’t need to solve all your environment access issues at once in order to make gains toward producing releasable software. Your goal should be to make as many of your environments as production-like as possible to allow earlier detection of defects that have typically not been found until late in the software release process.
Leverage Test Automation
Test automation is critically important when seeking to deploy software frequently into production. When beginning an effort to increase test automation to support DevOps delivery, focus on areas that provide the biggest bang for the buck first:
- Unit testing to identify defects early and validate code changes during check-ins
- Smoke testing to make sure testing environments are properly set up and your software is ready for testing
- Frequent regression testing across your code base to detect code that has broken something that used to work
Be sure to look at the entire test process. Establishing automation to provision environments, install software and databases, and tear them down after testing is completed can save a lot of time during testing and support the goal of continuously delivering software to downstream test environments and, ultimately, into production.
Use these tips to move toward a more effective testing process that supports a continuous delivery approach—without sacrificing quality.
