Since I have some free time, I’ve been catching up on some of the stuff
I’ve been meaning to read. I’ve got a reading list of stuff that I’ve wanted
to look at that were written by other authors with my publisher. Yesterday, I started looking at Cucumber, which is an interesting behavior-driven development tool. This post isn’t really about Cucumber, but about something that Cucumber reminded me of.
When a competent programmer builds software, they write tests. That’s just
a given. But why do we do it? It seems like the answer is obvious: to make sure that our software works. But I’d argue that there’s another reason, which in the long run is as important as the functional one. It’s to describe what the software does. A well-written test doesn’t just make sure that the software does the right thing – it tells other programmers what the code is supposed to do.
A test is an executable specification. Specifications are a really good thing; executable specifications are even better.
In general, in software, we have a tendency to think that we can do things by the seat of our pants. In my experience, most programmers don’t do much planning before they sit down and start coding. In their heads, they think that they know what they’re building – they’ve got a model of the system, and how they’re going to put it together. But a model like that, which has never been written down, is quite like to have holes in it – because until you work through the details, you don’t necessarily even know where the hard parts are.
Let me give you an example. A while back, when I worked for IBM, I was
working on an open-source SCM (aka “version control”) system called
“Stellation”. In Stellation, we tracked the version history of all of the artifacts that made up a system: we didn’t just version text files, but directories, binaries, etc. And we supported lightweight branches and (of course) branch/change merging.
When you do versioning of directories, merging directories gets pretty complicated. To give you a sense, here’s some of the cases. Suppose you have a directory D containing a file F, and you’re merging two branches:
- Branch 1 edits F, branch 2 doesn’t. No problem – this is easy.
- Branch 1 edits F, branch 2 edits F. Standard file merge.
- Branch 1 edits F, branch 2 renames F. Still no problem.
- Branch 1 edits F, branch 2 deletes F. Tricky.
- Both branch 1 and branch 2 rename F.
- Branch 1 removes F, and branch two adds a new file named F.
- Branch 1 renames F, branch 2 adds a new file named F.
There are a bunch more cases – in fact, there are a total of 12 cases.
We built the first version of the system, and thought we’d worked out how to merge directory changes. People started using it, and found a case we’d missed, which caused Stellation to do the wrong thing. I worked out the problem, and fixed it, and released a new version. And another bug report came in – another case I’d missed. And another. And another. Directory merging was just a whole lot harder than I’d expected.
What I ended up doing was taking a software specification tool called Alloy that I’d recently seen a talk on, and spending a couple of days building a thorough, formal specification of the merge component. It helped me understand what I’d missed, and figure out a genuine, exhaustive list of all of the cases that I needed to cover. It also provided a clear, concise, thorough description of the correct behavior of the system. That specification turned out to be incredibly valuable.
Specifications are valuable beyond words. They give you clarity about what your system should do. They make sure that you understand what you’re building, and how it should behave. They force you to think clearly about what you’re building. They provide an invaluable tool for communicating with other programmers.
There are two problems with specifications. First, they’re hard to write. I love Alloy – but it’s not an easy system to use. It takes a lot of time and effort to write a specification. It’s another language that you need to learn – and you need to learn it just to be able to read a specification. Second – and more important – the specification is usually a distinct artifact. It’s not part of your source code. It’s something else – usually a separate file, written in a different syntax. And as a distinct artifact, it’s one more thing that needs to be kept in sync as you make changes. Unfortunately, that doesn’t tend to happen: when you’re in a hurry, you change the code without changing the spec, and then they get out of sync; once they’re out of sync, the spec becomes useless.
That’s where testing comes in. If you write your tests correctly, they are a specification of your system. In particular, if you use a good BDD tool, then they read as a specification! But even if you aren’t using any testing tool at all, but just writing standalone testing code, you can write your tests so that in addition to testing your system, they describe your system. So you get the advantages of specification, while also having something executable. Every time you build your system and run your tests, you’re verifying that the specification is up to date, and that your system conforms to the specification.
This is something that really fascinates me, so I’m probably going to write a series of posts about this. I’ll probably do one or two about Alloy, to give you a sense of what a real formal specification looks like, and what that can buy you – and then a few posts talking about how to write specification-tests using BDD tools.