Copy/Paste Considered Harmful

When I first started coding years ago, it was in a very early version of Basic that ran on the timesharing computer at Dartmouth. I had no formal training and committed many of the sins of self-taught programmers, including programming by copy/paste. After a few times of having to fix bugs in multiple places I started learning about functions in Basic so I could stop that nonsense. I also became aware of how messy it was to add new features in copied code.

At this point in my career I am more aware of this problem, and tend (when I see it in existing code) to refactor before going on with my planned task(s). If there are automated QA tests with the appropriate coverage I assume they will catch problems. If not, I construct a few automated tests of my own to increase my confidence that the refactoring didn't change any existing behavior.

I was talking with a co-worker a few weeks ago about coding practices, and the topic of copy/paste came up. I went into my usual rant/sermon about it, only to find out that he actually had a professor who promoted copy/paste as a method of "code reuse." I don't know where my co-worker went to school, or what this professor had a degree in, but it took me aback to hear this.

Now there are a few times where copy/paste is appropriate. For example, if we are late in the QA cycle and about to release a product and find a real show-stopper of a bug. After analyzing the bug and determing the cause and the fix, we realize that it is deep in some code that is called from many other places. Given the time frame, we're not able to test all those places to the level of quality that we would like. In this case, I have occasionally done some copy/paste to make sure that the side effects of my change are as minimal as possible. I always, however, put a TODO (or TBD, or other cleanup marker, depending on the coding standards of the company I'm at) into the copied code with a reference back to the source of the copied code. This then goes as a task at the top of my cleanup list for the beginning of the next development cycle.

I don't see how to produce long-lasting quality code without doing this.

[Minor pop quiz: where does the title of this post come from?]

Setting Expectations

I was talking with my young niece a few days ago about getting shots. She told me she really hated them. I then told her a story about my brother who had always had a deathly fear of shots. At one point he and I were talking about this (as children), and it came up that he had always thought that when the doctor pushed the plunger in, it was making the needle go further into his arm. He said he was really frightened that it would go into his bone.

At that point my niece's eyes got big and she said, "You mean, it doesn't push the needle further in?"

I was intrigued that she had the same mis-information as my brother. It made me think of how hard I work to give correct information in order to set expectations. I feel that one characteristic of good engineering is to set expectations correctly. This may sometimes go against the wishes and hopes of the marketing and sales folks, but it has to be done. I do, however, feel that (in the long run) setting expectations correctly works for both me as a producer of software and my customers as consumers of software. By "customer" here I mean both internal (e.g., within the company) and external (e.g., paying) customers.

If I set expectations too high and don't meet them, my credibility is damaged, and people have a long memory for failure.

Lowest Common Denominator Wins This One

Today I came across this article from Component Developer Magazine about ODBC. I was especially amused (and bemused) by this quote:

At the time of its release, the SQL Server team at Microsoft believed OLE DB would supersede ODBC. This is not longer the case and ODBC's future is completely secure. ODBC is more widely used that OLE DB and it is better suited to some key scenarios that I will discuss later in this article.

I was originally planning to title this post "Old APIs Never Die, And They Never Fade Away Either." The message here, though, seems to indicate (if by omission only) that OLE DB's future is not completely secure. Or maybe I'm just trying to dig too far into the sub-text.

In any case, I believe this is a case where ODBC (as clumsy as some think it is) has won precisely because it is low level, focused on one task, and does not have a lot of proprietary technology that needs to be duplicated across platforms. While ODBC on non-Windows platforms such as Solaris and Linux still has a lot of rough edges, it's generally usable except for the occasional configuration nightmare.

I wonder, on the other hand, what will happen with efforts such as Mono, which are attempting to duplicate a much higher level of functionality and more compilicated interface. As Wikipedia says (today, anyway):

The Microsoft compatibility stack provides a pathway for porting Windows .NET applications to Linux. This group of components include ADO.NET, ASP.NET, and Windows.Forms, among others. As these components are not covered by ECMA standards, some of them remain subject to patent fears and concerns.

Sometimes one's eyes are bigger than one's stomach.

Share the Wealth

When I was leaving one of my previous jobs, I was working with a younger developer who was going to take over some of my responsibilities. While I had been working there I had written documents describing procedures and processes that I had put in place for myself. I had been keeping these documents (as well as some code for personal regression tests) in a source code repository using CVS. The repository was on the laptop machine with the working copy on the desktop machine. I used CVS because the company source code system (ClearCase) wasn't set up to allow personal archives. I also prefer not to use ClearCase when possible, but I'll save that rant for another post.

The day I left I cleaned out the CVS directories in my local copy of the directory tree, zipped it up and emailed that zip file to him. I didn't want to give him the CVS repository itself because I didn't want to confuse him any more than necessary.

I also changed the password on my email (Lotus Notes) and made sure that all my emails had been put into my local email database. I then gave him a copy of that database along with the password and made sure that he could open that local database on his machine. He looked at me, slightly puzzled, and asked if there weren't emails that I didn't want him to see. I told him that I knew from the day I started that these emails were really property of the company and they didn't belong to me. I never, therefore, used it for any personal purposes or sent any emails that I wouldn't want "read out loud in the town square" (to quote my brother-in-law).

I don't know if he ever actually looked at any of this, or whether he considered it wealth or dross. I did, however, feel better leaving the information with him, rather than letting it be wiped when my work machines were returned to the company pool.

The "Dead Leaves" Guy

My family and I have lived in the same house for the past 24 years. Once or twice a year I squeeze myself between our "Florida Room" and the neighbor's garage to clean out the oak leaves that end up there. It's a tight fit, and not a task I enjoy. When it's done, I step back, dust off my hands (figuratively and literally) and take the green waste bin out to the street for pickup.

I've always gotten along well with our next door neighbors, who have been there for more than a decade. Last weekend I noticed the husband up the roof of his garage, merrily sweeping the oak leaves into the space that I have been squeezing into all these years. He looked a little sheepish, but said he had been doing this all along. I told him that if I hadn't been cleaning them out, we would have had oak trees trying to grow between our buildings. I then assured him that it was OK with me, and we went to our various tasks.

I have also been cleaning out a lot of "dead leaves" that others have put into our code at work. I am concerned when I see code that is so obviously cut-and-pasted instead of refactored. I sometimes wonder what people think when they are doing this. Are they concerned because they don't understand the code well enough to do refactoring? Are they too overloaded to think about the fact that what they put in may sprout into oak trees in the worst possible place? Are they too rushed to think a few months down the road, or a year or two?

It concerns and mystifies me.

Glosses on Linus Torvalds Coding Style

A gloss is a marginal note (see picture in the Wikipedia entry for an example). While one can't very well annotate someone else's web page, I will add my voice to the chorus of those who have commented on Linus Torvalds coding style document.

I strongly agree with almost everything said in this document, and have been glad for the "backup" when I've had to argue a point with some of my fellow workers. There are, however, a couple of items where I diverge.

The first point that I would gently argue is that 8-space tabs are too deep. I haven't "been looking at [my] screen for 20 straight hours" for a couple of years, but at the very outset I worked with 4-space tabs, and I find them very readable. I haven't worked with anyone who has strongly defended 8-space tabs. I will admit that I have had to try to persuade people to switch from 2-space to 4-space tabs, but that was back in the Apple ][ Basic days when people were trying to keep their programs as small as possible (remember cassettes?). I don't know of any readability studies on this topic, and would be interested to see them.

My second argument (or shall I say "disagreement" to be less disagreeable...) is with his distaste for mixed-case names. I find mixed-case easier to read (maybe it's just me), but, more importantly, I find it a lot easier to type than underscores. I know I'm not the best touch-typist in the world, but I always find myself looking down to make sure that my ring-finger is indeed hitting the dash/underscore key and not one of its neighbors. Using camelCase keeps my fingers on the home row much more, makes my typing faster and reduces the number of errors.

These are just friendly disagreements, however, and don't diminish the fact that I'm glad this document exists, and is available to us all.

Pythonics Anonymous

I think I need a twelve-step program to wean myself from Python.

When I first looked at Python a few years ago I was put off by the fact that white space was significant. I also thought that having to do "self." to indicate object fields (vs. global variables or parameters) was just noise. I got tired, however, of always looking up Perl syntax, and I really didn't care for the Perl TMTOWTDI (I prefer one good way to do things, not many). I then came across Eric Raymond's article and decided to learn and use Python.

Now, after a few years of working with the language I appreciate the fact that white space is significant (no more bracket-placement wars!). I also appreciate the cross-platform-ability and the fact that (even though it is open-source) it isn't GPL, which has allowed us to use it at a company I used to work for. I like having the various supporting packages (although the minidom has a problem when pretty-printing XML that I had to work around).

However, I now find myself thinking that I'd better go back to Java. I was put off Java because the reality took a really long time to catch up with the hype. Remember "Write once, run anywhere?" (which was really "Write once, debug everywhere?") It also seemed too heavyweight to me. I liked being able to run the Python interpreter in command line mode and experiment without having to fire up an editor, create files, save them and run them. It was also satisfying to be able to be able to write a one-line "Hello World".

It seems now, however, that the world (at least the world of enterprise software) has decided that It's A Java World. I guess I better get back on board. I know I'll fall off the wagon occasionally, but it's time to put Python back in the corner and get back to business.

Data Always Goes In A Database, Right?

Not necessarily.

I have worked on a number of software products where data (usually metadata) is stored in text or binary files. The code opens the file, reads from the file and parses the text (if necessary) into the structures used internally by the code. Over the years, the question has been asked more than once: why don't you put this data into a database?

There are a number of reasons not to use a database. First, all the data needs to be in memory for performance, and there is no advantage to keeping parts of it out on the disk. Second, the data is usually not appropriate for being "table-ized." That is, the data is complex and tree-structured rather than row-structured. I know that one can always flatten data into multiple tables with the appropriate keys and joins, but parsers built with tools such as Yacc and Lex will always be faster than doing multiple database joins and reads.

A third reason is that upgrading the data in the database requires scripts to add or modify tables or columns (deletes not being done to reduce the potential for referential integrity problems). Parsers can recognize a version number and do an upgrade-on-read, which doesn't require another program to run to perform the upgrade.

Fourth, files are easier for end-users to handle. For example, if a user needs to send some of his data to tech support, it's much easier for him to package up one or more files, rather than having to run some database dump utility and send the dump file. There is also no issue with (once having that database dump) the possibility that tech support does not have the particular version of the database software that the user is running.

A fifth reason (for text files at least), is that in a pinch the file can be opened and edited in a text editor if there are problems with it. With a database you have to run the appropriate database client software, then tease out the table relationships to understand where changes need to be made.

When all you know how to use is a hammer, everything looks like a nail.

Fail Early (But Not Often)

Civic-minded citizens have long been reminded to "Vote Early". A corrupt Chicago politician extended it to "Vote Early And Often." Failures should be handled as early as possible, but shouldn't (under most circumstances) happen very often.

My primary principle is "The earlier the failure, the less the consequences." For example, if I didn't check the return code on a file open, the code would merrily continue on and try to write to the bogus file descriptor returned from the "file open". Most current operating systems give you a bad handle error, but earlier ones would just return junk at best and might do something grotesque to the file system at worst. Also, the code has probably already gone past the point where it can put up a coherent error message that includes the filename and system error code. This the leads to my second principle about failure handling, which is: "The better the error message, the less tech support hates my guts." One annoying error message that I've had to deal with (as an end user) is "The [someLibrary.DLL] is in use by another program." Which program? Do I have to go through my Task Manager, or use "ps" on *nix, trying to kill one program at a time (not a good idea anyway!) to find out which is the offender?

Most contemporary programming languages have some sort of exception mechanism to help me write code to handle failures. In a language like C++ where I can throw just about anything, I create a superclass designed to handle failures (subclassed as appropriate) combined with a macro that catches all possible types that any code in my executable might throw. As long as I put my catch blocks in the right places (e.g., at the top of a thread or in top-level event handlers) I can return errors without bad side effects. Java, on the other hand, will only allow me to throw descendants of Throwable, which removes the need for the "glue" class and macro that I described for C++, but adds the complexity of having to deal with multiple subclasses of Throwable. There's no such thing as a free lunch.

What if I am working in a language that supports exceptions, but I am using some API that doesn't? For example, suppose I needed to use fopen (the low level 'C' routine) while working in C++. In this case I always check the return code from that call (as well as all other file system calls). If an error is returned from any of those calls, I convert it into a coherent exception and throw that exception.

To move farther back in the evolutionary ladder of languages, what happens when I am working in a language (such as shell script) where there is no exception mechanism at all? Here I check for errors after every possible line of shell script that can possibly fail ("assume the worst"). I keep the code readable by putting the status check off to the right of the "meaningful" code. This means that when I read the code I can concentrate on what it should be doing in most cases, rather than the edge cases of failure. For example, this shell script contains a shell function that displays an error message, the name of the script and how to get instructions for running the script.

Note that the error might not be caused by inappropriate usage, but this at least gives the person who ran the script something to see before reading the code in the script.

In the spirit of continuous improvement, I've tried moving the status test down into the shell function, but I discovered (at least on Cygwin) that making the function call changes the global status variable ($?). I even briefly considered doing something like this where the _chk_ function runs lines of script and then checks the error code.

_chk_( 'first line of script' )
_chk_( 'second line of script' )

I rejected this, however, because (a) the noise level is too high, and (b) I had serious concerns about quoting problems within each line of shell script code.

Sometimes you have to know when to stop refactoring.

The Two Commandments of Blogger

Learned from bitter experience...

1. Don't copy and paste from anywhere else, unless you have no special formatting such as lists or fonts. If you do, you will end up retyping your post anyway because Blogger incorrectly converts the formatting. I have tried this from a number of different sources including Google Documents and somehow the text after lists always ends up with the wrong linespacing.
2. Don't trust the preview. In scenario above, the preview looks fine. When I post, however, the linespacing is messed up and I have to "Save as Draft" to pull the post and re-edit it. This happens in both Firefox and Internet Explorer.

Thanks to my son for his help and suggestions.

The Costumer's Handshake

What is it about code formatting standards? After all, the important thing is that the code compiles, links and passes its tests, right?

No. All psychological and behavioral arguments aside, there is something attractive about well-formatted code. Admittedly, well-formatted code does not guarantee functional correctness, adequate performance and maintainability, but it is clear (at least to me and the folks that I prefer to work with) that it helps readability and therefore maintainability. And it certainly makes a better first impression than a slap-dash bunch of statements thrown together.

What does this have to do with costumers and handshakes?

My wife likes to sew costumes for my daughter, who is deeply involved with the San Francisco Dickens Faire and other such semi-historical pastimes. At one of the Dickens Faires my wife approached a woman and asked her about her costume. This is not an uncommon occurrence at these events, and most people are only too happy to talk about their work. After chatting for a few minutes and establishing a rapport, my wife reached out, looked closer at the garment, and turned the edge of the woman's bodice to examine the stitching and the lining. The woman chuckled and said, "Ah, the costumer's handshake!" and we both immediately knew what she meant.

People who care about the quality of their (and other's) work know to look inside and underneath. Costumers know the handshake. Woodworkers know to open the drawers of a cabinet to check out craftsmanship of the joints inside. They look to see if something is veneer (likely to come off in time), or solid wood, which will last longer. Printers know to hold up a page of a book to the light to check that the imposition of back-to-back pages is in alignment, and to check the quality of the paper.

I don't know if such quick evaluations can help judge the quality of a body of code. There are obvious things to look for such as error handling and formatting consistency, but problems such as duplicated code and bad naming are harder to spot without spending more time getting to know the code.

I haven't worked with many open source projects, but the one I've spent the most time with (Blender - see www.blender.org) certainly doesn't pass a lot of my tests. Does that mean it's not successful? If open-source success is defined as "usage and visibility" then Blender certainly qualifies as a success. A lot of the commercial closed-source code I work with doesn't pass my tests either. Given that success for commercial closed-source is defined as profitability for the company that produces it, I can't comment on that in public.

I do know, however, that (by the craftsmanship standard), much of the code that I work with might not stand up to having its "lining" examined. But that is one of the goals I aim for.

Paranoid Programming

I read and enjoy Joel on Software (and used his CityDesk product for my website). I have to respond, however, to some of his comments in this article.

I strongly agree with most of it. For example, I'm glad to see conventions such as "Don’t put closing braces more than one screen away from the matching opening brace." (And I wish I could enforce that standard at the place I work at!) Also, he writes: "I have to admit that I’m a little bit scared of language features that hide things". I have seen very confused code caused by "features" in C++ like the parentheses operator on iterators. I always prefer to see a real method name such as "Next()" (thanks to Nick Pouschine for putting into words the solution to my unease with that syntax). And I never have, and never will, overload the plus operator for any reason.

I don't, however, see any reason to write off exceptions. I depend very heavily on exceptions in all the C++ and Python code I write. Joel's example of:

dosomething();
cleanup();

just seems wrong. The cleanup code should be on the destructor (or some equivalent in non-object-oriented languages), which should always get called at some point. Note that I'm assuming you've adopted the convention of "every object is correctly owned" so that memory is always freed at the appropriate point in the code, regardless of whether errors occurred or not.

[Apologies in advance for the CAPITALS.] I don't EVER need to "investigate the entire call tree of dosomething()" to see if there’s anything problematic in there. Rather I ALWAYS assume the worst, which is that any line of can cause an error (divide by zero, anyone?). I'm not paranoid, they really are out to get me!

I therefore write all my code to handle errors anywhere. That way, if an error happens in my code or some third-party code that I'm using, it is handled. Once I "accepted" exceptions, then I wrote code that I found more readable. I also was more confident in its error handling capabilities.

For example, instead of having all that nesting in the code from Raymond Chen's article that Joel references, I could put the variables into fields in an object, all the cleanup code into a destructor and then "flatten out" the error checking (I'll try to put an example in here later...). I don't claim that there are fewer lines of code (there may be more), but the error handling code doesn't get in the way of my reading and understanding of the intent of the code. This is especially true when errors have to be propagated back up a long call stack, where there may be many intermediate functions that would have to return the error. With exceptions, I only care about where they are thrown and where they need to be caught, not all the intermediate points in between.

To me, having the "other channel" of exception handling removes noise from the code and makes it more readable. I agree with Raymond that exceptions are tricky and you have to know how to do the right thing (e.g., don't throw pointers to stack-based objects and make sure to understand threading issues). Once, however, I had those principles understood and enforced, I was able to keep the error handling code "out of the way" of the rest of the code. This allows me to focus on the intent of the code as I read it, and not have to mentally "filter out" all the error handling code.

I certainly don't think that I'm smarter than Joel or Raymond. I do know, however, that I prefer using exceptions rather than having every function return error codes, both for code readability and robustness. I know they make me a more productive programmer.

Buffer Overflows Are So Twentieth Century

At one of the companies I worked for in the past, the engineering team was able to agree on a set of standards that met the criteria outlined in my previous post. These standards were short, usually 1-2 page documents. As this was at least Year 3 BW (Before Web), the documents were stored in Lotus Notes.

One of the standards that we quickly agreed on was titled "No unsafe string functions." That is, we banned (except for the very occasional exception case) the use of C-language functions such as strcpy, strcat and sprintf that can cause buffer overflow. We wrote a series of small safe functions that would make sure that buffer overflows would not happen. These functions all took a maximum buffer length as well as the target buffer pointer. We didn't want to convert to std::string because there had been a number of cases where some engineers didn't understand their usage and had caused very serious performance problems.

I wrote a Bourne Shell script to search across the appropriate files for all occurrences of the unsafe functions. I ran this once a week or so. Once in a while, I would find an incorrect usage that had crept in, but usually all the engineers understood the need for such a standard and would adhere to it.

A few years later the company was acquired and a new group of engineers was brought on board. We started over again with the standards process, but something strange happened with this particular standard. This new group could not accept it. I never fully understood why. I re-wrote the standard a number of times and had a long email correspondence with the leader of the group (who I have great respect for). However, those of us on the original team were unable to convince the newer members of the need for this standard.

A year or so later, one of the application engineers that used our product came to me and reported a crash. The product that I worked on was a server product, and crashes were taken seriously. I analyzed the crash and found it was caused by the use of one of those unsafe functions. I fixed the problem. Luckily the crash did not happen at a customer site, so we were able to prevent it before the particular application went live.

I then re-wrote the standard (yet again), and sent a pleading email to the leader of the group asking if they could now accept it.

They could, and did.

And now they understand.

What's a Coding Standard?

There are two kinds of coding standards, although those of you who are less binary than I am may prefer to think of a spectrum of types rather than just two. (You know, of course, that there are two types of people, those that categorize everything into two categories and those that don't.) The first type is a MUST standard. These "MUST" standards have less to do with what the code looks like and more with what the code does. That is, content over form.

An example of this type of standard for C++ would be "All pointer fields in an object must be initialized to NULL or a legal pointer in the object constructor." This is driven by the reality that we should always delete all pointers (that point to memory owned by an object) in that object's destructor. Setting all pointers in the constructor guarantees that, regardless of what happens next and as long as any other code that changes those pointers does so in a proper manner, the destructor can be safely called at any point.

The second type of standard is a SHOULD standard. These standards are more concerned with form, not content. Here things are a more subjective. Here I put standards such as this document containing Linus Torvald's coding standards for Linux (or Google for "linus torvalds coding standards" for some more juicy reading). I agree with all of these except for the 8-space indentions and CamelCase (see, we're already in contention). I'll go over my reasons for disagreement in another post, but this already illustrates my point.

How do we know when something is actually a standard, and not just a document written by someone sometime somewhere? Here I have to defer Nick Pouschine, one of the best people I've ever worked for. His definition of a standard is:

• All code in the project actually follows the standard. Note that this means older code also, since most coders spend their time looking at code, not looking at coding standards. If there is code that does not match the standard, then the coders will not take the standard seriously.
• The standard must be agreed upon by all the coders working on the project. If it is not agreed upon, different coders will do what they want anyway and the standard is not a standard.
• The standard must have at least one reason for being a standard. That reason can be something like the example for constructor usage above, or something like Linus' "A human brain can generally easily keep track of about 7 different things, anything more and it gets confused."

Without these, a standards document is just a document, and not a standard.