Overengineering: Where productivity goes to die

They say premature optimization is the root of all evil. Well, personally, I think it would have to fight pretty hard against overengineering to hold that title.

Let's forget about overengineering for now. Although it's in the title, I will take a minute to talk about another concept, but one which is closely related as we will see: Design. And since this is a programming blog, we're talking about software and code design. Design, of course, is a cornerstone of software development in general. It doesn't actually solve problems per se, problems are solved by algorithms. But any project of non-trivial size and complexity contains several algorithms and data on which those algorithms operate. This material needs organization, so it can be used,maintained and extended most easily, at lower cost. If you poorly organize that material, then your algorithms will most probably be poorly used too.

Let's take a very,very simple example on how design is important, even when dealing with small codebases. Consider this situation: We work on a database for a gym. This contains a table 'Customers'(yeah I know, really original). A customer has proprties like age or weight. Now, let's say we work on a form where there exist a textbox and a button. We want to enter a weight value on the textbox, press the button, and increase the fee of all the clients that weigh more than that value by 5%(racist? Maybe, but it's just example). Then, we print the clients' name(to the console,to a file, to a printer, doesn't really matter).

The code that solves that problem could look something like this:

void okbutton_click()
{
    database.ExecuteSQL("UPDATE Fee=Fee*1.05 FROM Customers WHERE Weight>"+weightTextBox.text);
    custRecordset.Query("SELECT Name FROM Customers WHERE Weight>"+weightTextBox.text);
    for each record in custRecordset
    {
        out.print(record.fields["Name    
    }
}

Yeah yeah I know, ad hoc SQL queries are bad, parameterized SQL,Stored procedures(if that's your thing), etc, etc. Those are also design(and not only that) issues, but on a higher level. This is just a small example.

Now say, that at some point along the line, you design another form, that has a menu with the option 'Make all updates', or something like that. This option updates the database based on some fixed rules. One of them(but not the only one), is the aforementioned increase of fee, only for customers weighing more than 100kg. So how would this look like:

void menuOverweight_click()
{
    ...
    DoUpdateStuff();
        database.ExecuteSQL("UPDATE Fee=Fee*1.05 FROM Customers WHERE Weight>100");    
    DoMoreUpdateStuff();
    ...
}

We start to see a small pattern here, right? We basically update the database in a similar manner in both event handlers. Those 2 lines of code in the button and menu handlers are almost identical. Now, since it's such a small amount of code, perhaps it doesn't matter that much if we duplicate it. But imagine that,in real world, this "update" would take 100 lines of code. Duplicating them in more than one place is obviously less than optimal.

We have now stepped into design territory. We can abstract things, we see the similarities behind the apparent differences. So, a good practice in this case is to make a function "UpdateOverweightFee(weight)", that executes the SQL command which increases the fee for overweight clients. So, instead of executing similar SQL from inside the button and menu handler, we call that function.

This is a good choice. We have now one point where we have 'centralized' a certain functionality. We can use that function from anywhere in the code we need it. And in case we need to correct that "update" code, either because of a previously undetected bug or simply because of some change in the database schema, we only need to correct that function. In the previous state of the code, we would need to correct it in 2 places. So we definately gain something.

So, this is it. We made a correct design choice. We generalized the code so it can serve a need more than just one time. Hm, but why stop here? Surely we can generalize more. Hm, what to abstract, what to abstract. Easy. The increase of the fee. We may at some point want to enforce a different % of increase. So we make the function we created earlier to take that amount as a paramter.

Much better. But...wait a minute. Isn't there anything else why can abstract? Oh, I got it. We currently can supply the "weight" value to the function, and it will update all the customers weighing more. But what if we need, at some other point, for some other reason, to update all the customer that weigh *less* than some value? Couldn't we,oh...maybe supply the appropriate operator too(in the form of a string, like ">" or "<=")? So the call to the function would become something like UpdateCustomersFee(weight,amount,operator)?

Hm. Right. This is much better. Now the function is more abstracted, and serves more purposes. This is definately good. But you know what it would be better? Not just better, but utterly awesome? If we could abstract it even more. Who's to say that,at some time in the distant future, we won't need to update another column of the Customers table based on the weight rule? I can't think of such case at the moment, but it sure could happen. Can't the column name be supplied too as a parameter to our function? Hell yeah it can! So voila, our uber function: UpdateCustomersColumn(columnName,weight,amount,operator).

Oops. Houston, we have a problem. We have gone from having one useful function with a defined purpose, to having a monster that, well, nobody really can tell what its purpose is. UpdateOverweightFee(weight) makes sense. UpdateCustomersColumn(columnName,weight,amount,operator)...doesn't. It's flexible all right, but the problem is that it's so flexible, so customizeable and so general, that we might as well write SQL instead of calling it, just like we did at the start.

And that, my friends, is overengineering. Thinking about design and abstractions is good. But when you're obsessing about them, it becomes a problem. A very serious one.
People that have not experienced it(yet) have no idea how catastrophic it could be. It can literally tie your hands down, making you unable to produce 10 lines of code without your mind wandering off to a dozen of imaginary problems that could be solved if you abstract just a little bit more, if you refine just a little bit more, if you generalize just a little bit more. Or, without feeling guilty because, in one or another way, you've broken or 'bended' OO principles like Open Closed or Liskov substitution.

So, when thinking about design is good, and what it is too much? Personally, I have come to a conclusion: Any given programmer has some experience, and some skill. For simplicity's sake, we rate it on a 0-100 scale. Well, my opinion is this: If you're a 10, code like a 10. If you're a 50, code like a 50. If you're a 90, code like a 90. (If you're a 100, get a life). Don't struggle too much in order to achieve the imaginary optimal design if you're...well, just not that good yet. Learn about correct design, practice about correct design, strive to be better, but when you sit down to code, it's time to get some shit done.

Spend some time thinking about design issues. But don't spend too much. Like I said, if you're a 50, deal with it. So you spent hours and still can't find an optimal design solution. Suck it up and just implement *your* optimal solution, that is the best *you* can think of. Don't feel guilty about it. Nobody was born an amazing developer. Deep down, you know the level of abstraction you're comfortable with. Don't pretend to be smarter than what you really are when coding.

Remember, code is a means to an end. It may be that result is not the only thing that matters, but boy it does matter. Especially if you're a hobbyist, programming is supposed to be fun, not a torture. So, make it fun. Coming up with a smart design solution may be rewarding, but getting e-mails from players that find your game awesome is so much better.

And that's about it. Till next time!