I recently had the opportunity to work on a software development project as part of a 300 person team. On this team there were two types of developers: user interface (UI) developers and service developers. In order to evaluate the effect of a code change, UI developers needed to deploy their modified JavaScript files to Tomcat, while service developers had to deploy their changed services to a JBoss server. Although deploying files to Tomcat is a relatively quick process, the service developers had to restart JBoss after making a service change, which could take as long as 30 minutes. In this post I am going to explain why having such a long feedback loop is detrimental to a software development project and then discuss some strategies for coping with long feedback loops.
These long feedback loops are detrimental to software development projects for two reasons. The first is that they interrupt “cognitive flow” and make it harder for developers to complete their work. A developer may forget exactly why he or she made a change by the time they see its effect. This is an inefficient way to develop software — the developer must constantly refresh his or her memory before evaluating the effect of the change and continuing to work. As a result, the worker may quickly become bored or frustrated, which may cause them to check in code that has not been compiled or tested.
In addition to lowering employee morale and decreasing the quality of the software that they produce, these long feedback loops also have serious financial implications for software development projects. Consider a team of 300 individuals with an equal number of UI and service developers. If a UI developer waits for two minutes to see the effect of a change and makes a testable change 48 times per day (once every 10 minutes), then he or she will spend 1.5 hours (19%) of their working day waiting. Service developers spend even more time waiting: although they will test their changes less frequently (say six times per day), they must wait 30 minutes for JBoss to start. Over the course of an eight hour work day, a service developer will spend 3 hours (38% of their day) waiting. Given this information, all of the UI developers on the project combined will spend 225 hours per day waiting, while the service developers will spend 450 hours per day waiting. This amounts to 675 hours (84 man days) spent waiting over the course of one day. If all developers work 20 days per month for an entire year, then the total time spent waiting is 162,000 hours (20,250 man days) per year. If each developer is paid $40 per hour, then $6.5 million per year is spent on developers waiting to evaluate the effect of their changes.
Even though this long feedback loop was detrimental to the software development process, our team found ways to both reduce and make better use of the wasted time. One strategy is to write out the steps that you plan to take before starting your work. This allows you to remember what you were doing quickly after waiting for a server restart to complete. If possible we also tried to test the effect of several changes at once, especially when developing service code. Although this strategy is risky (we might be unsure if the effect can be attributed to one or a combination of changes), it can save time if used for changes that are routine or likely to work. A final strategy is to use this waiting time to check email and plan next steps so that you can spend the rest of your working time focused on development.
Long feedback loops can waste a significant amount of development effort and money and can make it harder for developers to perform their jobs well. In addition to coping with these long feedback loops by using the strategies described in the previous paragraph, it is important to consider whether it is possible to reduce the feedback time and thus improve developers’ efficiency and save money. For example, if one person spends one week of effort (5 man days) reducing the JBoss server startup time to 20 minutes, then the service developers will save 36,000 hours (4,500 man days) of effort, or $1.4 million, per year. This time savings could likely be achieved by moving non-JBoss jars from the base lib folder to the server-specific lib folder, as storing these non-JBoss jars in this location slows startup time. UI developers could also reduce the length of their feedback loop by selectively re-deploying modified JavaScript files to Tomcat, perhaps by using TortoiseSVN to determine whether each JavaScript file has been modified. If it takes someone one day to implement a selective Tomcat deployment tool that reduces wait time from two minutes to one minute, then the UI developers will collectively save 27,000 hours (3,375 man days) of effort, or $1.1 million, per year. Implementing both of these strategies could save $2.5 million of wasted effort per year.
Although the potential cost savings are large, many projects will not take advantage of this benefit due to a short-term point of view. Developers are often forced to work on “high-priority” tasks, and are told that now is not the time to improve the way in which they work. Learning to take a long-term point of view by investing time now to save time in the future is a valuable skill for both managers and developers in software development projects.
