Options for Promoting and Controlling Changes in Risk Averse Environments

The difference between an SCM professional and a general SCM person s the effect they have on changing the software development process. Each software organization is inherently different. Each has different approaches to, and tolerances for, risk. Risk doesn’t just come in the form of the code being written; rather the introduction of external factors can introduce new problems into the product ecosystem. Changing environments, operating systems, and patches increase the matrix of possible effects of individual changes. They can become potentially daunting, especially in a risk adverse environment, which has a tendency to lead these environments to become change-controlled.

The same is true of legacy software. Mature products entering the latter part of their life cycle aren’t subject to the same frequency of changes as their younger counterparts, but they still change. In the more change-adverse environments, controlling these changes becomes a key component of development competency.

In these environments, we don’t want to stop change. Rather, we want to understand it, understand its associated risks, and encourage those changes deemed necessary. So let’s examine our possibilities for change control.


The Low-Key Approach: “Formalizing” Informality

One of the hallmarks of good software development process involves code-reviews, with many sets of eyes pouring over and viewing the same problems. Bringing code-reviews into the commercial development process, helps encourage the bazaar like effects mentioned by Eric S. Raymond in The Cathedral and the Bazaar. Lightly formalizing these reviews prior to a commit to a revision control server adds value in our risk-adverse environments. At the very least, it helps spread knowledge throughout the development organization, reducing the ivory tower approach associated with many slow moving commercial projects.

The best time to have these reviews is prior to changes committed to the revision control system, prior to the change affecting the universe at large.. Most modern systems support the idea of pre-commit triggers, a script that can run prior to a user’s changes being accepted by the system. As developers commit their changes to the system, checking the commit comment for a “reviewed by” statement, and rejecting those without reviewers starts the organization down the path of formalizing reviews.

Tracking the purpose of the change is another way to tackle the problem. Regardless of what type of development organization you’re a part of, there’s a high likelihood that your organization has deployed defect and/or feature tracking systems. Using the same recipe as above, it’s reasonably easy for the SCM team to ensure the revision control system verifies that defects exist for the incoming change. All we’re doing is making sure that when someone addresses bug 12345, they’re actually checking in code against bug 12345. Bonus points if your defect tracking system can link back to your revision control system.

The Interactive Approach: Formalizing Code Reviews

While informal code reviews are definitely a great way to start, trying to analyze things after the fact is somewhat difficult. How do you know what happened during the review? How can I ensure that the changes I’ve requested have been implemented? More importantly, how can I record the artifacts of a code review session? Journals aren’t searchable.

Enter review-board, the open-source code review tool, a tool that allows users to request reviews from each other, can talk to your revision control system, and stores review artifacts! Regardless of how users want to do their reviews (at each other’s desks, over the phone, or even at 5am), the tool allows users to capture their review artifacts, comments, and most importantly changes. Users can go back and forth, commenting on code and comparing their differences. Reviews can even be searched after-the-fact, with an iterative diff and review process. Perhaps the best part is that review data is stored in a relatively easy to query database. Closing the loop, and making sure that a committer’s change has been reviewed is a relatively easy task, should your environment require these guarantees.

The best part of review-board is its powerful diff and review page. ReviewBoard is a stylish web-based tool that allows users to compare differences, comment on any line of code, and is slick while doing it. Review-board makes doing code reviews easy and fun. It’s almost easier to use than an informal review at a monitor!

The Thoughtful Approach: Building a Safety Net

While code reviews definitely help contribute to code quality, they can’t necessarily catch bugs. No automated process can catch all the bugs. We can’t replace testers with machines, but we need do what we can from an SCM perspective to help reduce the bug-count over time. Most importantly, we need to help our developers build the confidence in the system, and their code, so they can do what they do best, code!

As SCM professionals, we’re always striving for the ideal, trying to build the perfect yet pragmatic system, and always practicing what we preach. As infrastructure engineers and developers, our designs also need review. Our designs must also subject to review and improve over time, with the goal of increasing value. So how do we, as a team build our safety net? The same way developers ought:

The easy rule to live by is tread lightly. Start casting your net by writing unit tests, even introducing them to existing code. Write tests for new features, and new code; simply make it a part of getting it done, and help move our product down this path.

But what about old code? For each class you visit, try writing a single test, testing a single function. When I’m visiting a class, I always try to write at least one, small, easy to deliver test. Over time we’ll build up a body of tests that can be useful in helping to direct refactoring and feature development efforts. At the outset, writing tests will take more time, but as the development team becomes accustomed to the mindset, controlling the change through testing becomes easier. After all, unit tests don’t prevent changes, but they do help contribute to a safety net for understanding the magnitude of a change. To take this a step further, start thinking about integration tests!

Another great safety net can come in the form of a dedicated staff member. Imagine having a Software Archaeologist on your team, a veritable Indiana Jones for your organization’s development needs! Such a person would delve into the history of changes, understanding their purpose, and more importantly the effect they have on customer behaviour. Like any good Archaeologist, our team member would be responsible for educating users, helping them understand the ramifications of their changes, and more importantly the history of their subject matter. Long-time customers would greatly benefit from having an advocate in the software development team, while developers would benefit from a fresh set of eyes taking a holistic approach

The Proactive Approach: Addressing the Culture of Change

Every organization has a culture of change, and part of addressing changes pro-actively involves addressing the change culture. An organization that promotes few but sweeping changes needs to encourage a development culture of reviews, sharing, and discussion. The review process and tools mentioned above can easily help get the organization on the path to instilling the change-adverse (really change cautious) culture, but ownership is an easier method.

Team members need to understand the goal, to internalize, to really commit to it. It can be difficult to get passionate about not changing something, and that will typically be the focus of developers writing code. Instead, try to coach developers towards getting passionate about the right change. Help the organization develop the “buy in” of employees; this in turn helps promote the change activity to a positive one. Call out excellent approaches to a problem, reward positive activity, and popularize difficult solutions. Slowly, over time the inclination for change will become the inclination for the right change.

The Realistic Approach: A Mixed Bag of Tricks

In reality, a development organization will adopt the practices best suited for its goals, its culture, and its personalities. The important part to remember is that you’re on a journey, and any series of baby steps helps get the organization to its destination. By starting with informal reviews, and even the occasional unit test, you’re starting down the path, helping the organization on its journey of change.