Software Design and programmers - 2

What about the role of the programmer on a maintenance project, and the resulting importance of programmer-owned design? Most of the time, I think, when developers talk about software development in the abstract and when authors write about development techniques, there is an implicit assumption that everyone is talking about the development of new software. However, many (most?) programmers are not working on the from-scratch development of brand new systems (even though that’s probably everyone’s preference). Instead, they are working on the maintenance of existing production systems, either in a bug fixing mode or in an extension mode, adding new features or otherwise evolving the software in new directions.3

It is even more important, then, that we recognize that the design skills of the programmer are of even greater import when a project is in maintenance mode. This is because, from my observations, many companies reduce the staff of a development team when a system goes into maintenance mode. Analysts and designers move on to the next assignment, while programmers are left to work on the "completed" system. In fact, when a project shifts to maintenance mode, less expensive (and probably less experienced and less skilled) programmers are often brought in to replace the original programmers. When new features need developing, or when an existing part of the system needs redesigning or performance tuning, it is often the maintenance programmers who do the design and implementation. (Incidentally, the resulting increase in learning opportunities can be one of the definite advantages of working as a maintenance developer—especially for a less experienced developer.)

In his landmark 1988 book Rethinking Systems Analysis and Design, Gerald Weinberg writes:

Successful designers have learned to avoid the temptation to design everything in one big lump. Instead, they may build one small part at a time, analyze the actual behavior, and then repeat the process for the next part. In this way, design becomes not only evolution-like, but actually evolutionary.
This process of incremental design takes place at two levels—in the mind of the designer, and in reality. When it takes place in reality, it is design as maintenance, which is the principle mode of design today. The vast majority of design decisions actually put into effect today are created by maintenance programmers, not designers.
We4 do not mean to imply that this situation is a good thing—much of the ‘design’ being done in maintenance can be equally well viewed as systematic deterioration. [Weinberg, 1988, p.103]
I could go on for ten more pages in directions suggested by this passage, but I will resist and be as brief as possible.5 First, though, an aside: notice the connection to Extreme Programming, which had not been invented when these words were written. Extreme Programming fully embraces Weinberg’s idea of "design as maintenance" and joins it together with Reeves’s assertion that "it is cheaper and simpler to just build the design and test it than to do anything else." [Reeves, 1992] The result is a process that explicitly leverages that which Weinberg (rightly) labels as risky and dangerous. What makes this work for Extreme Programming is that it uses techniques such as test first design, automated testing, short iterations, pair programming, and refactoring to simultaneously mitigate the risk and transform that risk into an advantage.

But back to the topic at hand: if Weinberg is right that "the vast majority of design decisions actually put into effect today are created by maintenance programmers, not designers," then design skills for programmers on maintenance projects (which I believe is most of us, for at least some percentage of our time) are that much more important. The ultimate care, humility, and integrity are required when extending a production software system.

The maintenance programmer/designer must constantly balance competing concerns: the need to preserve consistency within the existing system, both in the code and in the external interface; the desire to use the best and most correct solution; the desire to use the latest techniques; the desire to create the best experience for the user; the absolute requirement to not break or otherwise destabilize the production system; the critical charge to protect the production data; the desire or need to improve the design or code that already exists; and the need to craft a solution that can be deployed into the existing system with the least disruption and downtime possible. If these competing concerns are not managed with care, then Weinberg’s "systematic deterioration" (as well as "systemic" deterioration) is inevitable. Is it not ironic, then, that the job of the maintenance programmer is most often viewed as being beneath the best and brightest developers?

Given everything I’ve said up to this point, I might be giving the impression that I am lobbying against the software engineering approach—with its dependency on specialization, documentation, and hand-offs—in favor of an exclusively programmer-centric, craft-based approach. This is not my position at all. I agree wholeheartedly with Alan Cooper, who said in a 2002 interview,

These two movements in the software world—engineering and craft—appear to be moving in opposite directions. But I think they’re both right. The problem is that you can’t focus craft methods on engineering problems and vice versa. So the places they break down is misapplying RUP to something that needs three craftsmen working on it, or trying to use individual craft methods to do big projects. Where is it carved in stone that we have to use the same method for all projects? [Thé, 2002]
My point in emphasizing the role of the programmer-as-designer is not to say that specialist designers are a bad idea. Nor am I suggesting the programmers working on a project where someone else does the designing should revolt and ignore the designs they are given. My point is, other than those few projects that explicitly embrace the programmer-as-designer idea (such as XP projects), there is a danger in not recognizing the inherent design role of programmers.

The danger exists on both sides of the hand-off. Whittaker and Atkin wrote, "The nature of designs is that they abstract many details that must eventually be coded." [Whittaker, 2002, p.108] Designers then must be sensitive to the kinds of things they are abstracting in their designs (that is, what blanks they are leaving for programmers to fill), and programmers must realize that a certain amount of design is part of their job and to do their best to a) hone their design skills so that their designs will be the best possible, and b) to design consistently within the framework of the designs they are given.

On a recent project I was tasked with designing a set of PL/SQL stored procedures that would create and move a large amount of critical application data. I had three advantages: the database was designed largely by me, I had spent a lot of time in the company for which we were building this software, and I had intimate knowledge of the structure these procedures needed to have and the logic they needed to follow. I had two disadvantages: I had a lot of procedures to design under an extremely tight deadline, and the two developers who were assigned to write these procedures from my designs were brand new to the project and the domain. They had very little knowledge of the database or even what this complex system was all about. Also, having never worked with them before, I had no knowledge of what kind of coders they were. To top it off, the QA team that would be putting these procedures through their paces did not have the kind of knowledge or documentation they would need to test them.

Given this set of circumstances, I wrote very detailed designs. I spelled out every single detail, using elaborate pseudocode—even writing many of the SQL statements the programmers would need—and adding extra expository explanation wherever I could. When I delivered the designs to the developers, it was as close as it gets to those software engineering books Whittaker and Atkin were making fun of. I could not leave anything to chance in terms of the programmers’ misinterpreting the intent or logic of any part of the design. The programmers even joked about how it wasn’t too much fun writing the code when everything was spelled out for them that way. But we got those procedures done within the deadline, and they were quite solid.6

Even in this example, though, the programmers did have to interpret certain things about my designs, and they had to come back to me to get clarifications and fix mistakes I had made. Furthermore, there really was a fair amount of design decision-making for them to do. They had to turn my pseudocode into real PL/SQL code, they had to design an error handling scheme, and they had to design internal data structures.

In another situation on a different project (this one a maintenance project), I needed to design a feature that a programmer on my team was going to build. I had done the analysis with the business people, and I had a vision about how to make the new feature work. This time, however, I had a programmer who had been on the project for awhile, and he had good knowledge of the domain and the existing system. Furthermore, I knew him to be an excellent coder with great design skills and instincts. In this situation, I knew that I did not have to write a highly detailed design. I wrote up a short requirements/design document that described the feature in general terms, explained where it fit in with the existing application, spelled out a few key business rules and constraints, and described a rough user interface concept, with some specifics where necessary. I laid out some rough class and database design ideas, but explicitly spelled out in the design that the developer was free to design it the way he thought best.

The developer and I had a short meeting, and I handed the design document over to him, requesting that he keep it up to date as he finished the design and built the user interface and code. Also, I pointed him to a business person that he could go to if he needed some requirements clarifications and asked him to keep me in the loop. Here we have one of those "middle realities" we talked about earlier.

As a final example, I worked on another project where I was not the designer. I observed that the designers were not being sensitive to the kinds of concerns described in the previous two examples. The designs they created and handed to the developers were incomplete, as all designs are, but not in the appropriate ways. The developers had not been in the requirements and design meetings, they were not domain experts, and the project did not have good analysis documentation. The designers left critical business rules out of the documents, and left many crucial points open to interpretation with vague language.

As you can imagine, the developers were very frustrated in this situation. They had to scramble around asking various people questions to try to fill in the blanks. They had to go back to the designers to ask for clarifications, and the designers would get frustrated too. When the developers did make an interpretation and design things as they thought best, the designers would come back and say "No, no, that’s not the way it’s supposed to be," which of course caused even more frustration on both sides.

Now, one could argue that these designers were simply creating poor designs, but I think there was more to it than that. It was not that the designs were bad, but rather that they were incomplete at a level that was inappropriate to the situation. In my opinion, these designers were not sensitive to the fact that the developers would have responsibility for some portion of the design, and more importantly, they were not sensitive to the portions of the design for which the developers were equipped to have responsibility. The designs were not tailored to the process, the situation, or the audience, and the designers erroneously viewed their own designs as "complete."