{"id":1509,"date":"2011-08-16T13:14:14","date_gmt":"2011-08-16T17:14:14","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/?p=1509"},"modified":"2011-08-16T13:14:14","modified_gmt":"2011-08-16T17:14:14","slug":"the-value-of-tests-its-more-than-just-testing","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2011\/08\/16\/the-value-of-tests-its-more-than-just-testing\/","title":{"rendered":"The Value of Tests: It&#039;s more than just testing!"},"content":{"rendered":"<p> Since I have some free time, I&#8217;ve been catching up on some of the stuff<br \/>\nI&#8217;ve been meaning to read. I&#8217;ve got a reading list of stuff that I&#8217;ve wanted<br \/>\nto look at that were written by other authors with my publisher. Yesterday, I started looking at Cucumber, which is an interesting behavior-driven development tool. This post isn&#8217;t really about Cucumber, but about something that Cucumber reminded me of.<\/p>\n<p> When a competent programmer builds software, they write tests. That&#8217;s just<br \/>\na given. But why do we do it? It seems like the answer is obvious: to make sure that our software works. But I&#8217;d argue that there&#8217;s another reason, which in the long run is as important as the functional one. It&#8217;s to describe <em>what the software does<\/em>. A well-written test doesn&#8217;t just make sure that the software does the right thing &#8211; it tells other programmers what the code <em>is supposed to do<\/em>.<\/p>\n<p> A test is an executable specification. Specifications are a <em>really<\/em> good thing; executable specifications are even better. <\/p>\n<p><!--more--><\/p>\n<p> In general, in software, we have a tendency to think that we can do things by the seat of our pants. In my experience, <em>most<\/em> programmers don&#8217;t do much planning before they sit down and start coding. In their heads, they think that they know what they&#8217;re building &#8211; they&#8217;ve got a model of the system, and how they&#8217;re going to put it together. But a model like that, which has never been written down, is quite like to have holes in it &#8211; because until you work through the details, you don&#8217;t necessarily even <em>know<\/em> where the hard parts are.<\/p>\n<p> Let me give you an example. A while back, when I worked for IBM, I was<br \/>\nworking on an open-source SCM (aka &#8220;version control&#8221;) system called<br \/>\n&#8220;Stellation&#8221;. In Stellation, we tracked the version history of all of the artifacts that made up a system: we didn&#8217;t just version text files, but directories, binaries, etc. And we supported lightweight branches and (of course) branch\/change merging.<\/p>\n<p> When you do versioning of directories, merging directories gets pretty complicated. To give you a sense, here&#8217;s some of the cases. Suppose you have a directory D containing a file F, and you&#8217;re merging two branches:<\/p>\n<ol>\n<li> Branch 1 edits F, branch 2 doesn&#8217;t. No problem &#8211; this is easy.<\/li>\n<li> Branch 1 edits F, branch 2 edits F. Standard file merge.<\/li>\n<li> Branch 1 edits F, branch 2 renames F. Still no problem.<\/li>\n<li> Branch 1 edits F, branch 2 deletes F. Tricky.<\/li>\n<li> Both branch 1 and branch 2 rename F.<\/li>\n<li> Branch 1 removes F, and branch two adds a new file named F.<\/li>\n<li> Branch 1 renames F, branch 2 adds a new file named F.<\/li>\n<\/ol>\n<p> There are a bunch more cases &#8211; in fact, there are a total of 12 cases.<\/p>\n<p> We built the first version of the system, and thought we&#8217;d worked out how to merge directory changes. People started using it, and found a case we&#8217;d missed, which caused Stellation to do the wrong thing. I worked out the problem, and fixed it, and released a new version. And another bug report came in &#8211; another case I&#8217;d missed. And another. And another. Directory merging was just a whole lot harder than I&#8217;d expected. <\/p>\n<p> What I ended up doing was taking a software specification tool called <a href=\"http:\/\/alloy.mit.edu\/community\/\">Alloy<\/a> that I&#8217;d recently seen a talk on, and spending a couple of days building a thorough, formal specification of the merge component. It helped me understand what I&#8217;d missed, and figure out a genuine, exhaustive list of all of the cases that I needed to cover. It also provided a clear, concise, <em>thorough<\/em> description of the correct behavior of the system. That specification turned out to be incredibly valuable.<\/p>\n<p> Specifications are valuable beyond words. They give you clarity about what your system should do. They make sure that you understand what you&#8217;re building, and how it should behave. They force you to think clearly about what you&#8217;re building. They provide an invaluable tool for communicating with other programmers.<\/p>\n<p> There are two problems with specifications. First, they&#8217;re hard to write. I love Alloy &#8211; but it&#8217;s not an easy system to use. It takes a lot of time and effort to write a specification. It&#8217;s another language that you need to learn &#8211; and you need to learn it just to be able to read a specification. Second &#8211; and more important &#8211; the specification is usually a distinct artifact. It&#8217;s not part of your source code. It&#8217;s <em>something else<\/em> &#8211; usually a separate file, written in a different syntax. And as a distinct artifact, it&#8217;s one more thing that needs to be kept in sync as you make changes. Unfortunately, that doesn&#8217;t tend to happen: when you&#8217;re in a hurry, you change the code without changing the spec, and then they get out of sync; once they&#8217;re out of sync, the spec becomes useless.<\/p>\n<p> That&#8217;s where testing comes in. If you write your tests correctly, they are a specification of your system. In particular, if you use a good BDD tool, then they <em>read<\/em> as a specification! But even if you aren&#8217;t using any testing tool at all, but just writing standalone testing code, you <em>can<\/em> write your tests so that in addition to testing your system, they <em>describe<\/em> your system. So you get the advantages of specification, while <em>also<\/em> having something executable. Every time you build your system and run your tests, you&#8217;re <em>verifying<\/em> that the specification is up to date, and that your system conforms to the specification.<\/p>\n<p> This is something that really fascinates me, so I&#8217;m probably going to write a series of posts about this. I&#8217;ll probably do one or two about Alloy, to give you a sense of what a real formal specification looks like, and  what that can  buy you &#8211; and then a few posts talking about how to write specification-tests using BDD tools.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Since I have some free time, I&#8217;ve been catching up on some of the stuff I&#8217;ve been meaning to read. I&#8217;ve got a reading list of stuff that I&#8217;ve wanted to look at that were written by other authors with my publisher. Yesterday, I started looking at Cucumber, which is an interesting behavior-driven development tool. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[93],"tags":[],"class_list":["post-1509","post","type-post","status-publish","format-standard","hentry","category-program-specification"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-ol","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/1509","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=1509"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/1509\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=1509"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=1509"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=1509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}