Recently someone on a group on internet asked for some ideas and suggestions on refactoring legacy code. Not your usual refactoring, but the project consisting of refactoring legacy code.  Now, I guess the first thing that you might ask at that point is “what is legacy code anyway”?
According to some TDD purist, legacy code is any code that is not covered by comprehensive set of unit tests. Generally, legacy code is some code in advanced lifecycle phase, burdened by heavy debt, sprinkled with smells like long methods and large classes and often implemented in some older technology, language or language version.
Here is what I replied:

Hi,

I’d say you are in for a ride.
Here are a few thoughts based on my experience.

First of all, do not discard rewriting application from scratch without giving it a really good consideration. In my experience, it is often much easier to develop an application from zero when compared with refactoring legacy version; it is also much more difficult to refactor the legacy code than it might look at the first site. If your team doesn’t have prior experience with refactorng of legacy code, it’s all too easy to underestimate the task. Do an experiment. Choose a component or a class and see how long does it take to refactor it. On one occasion, I performed one such experiment. I was able to reduce a 4000 LOC component to a 2000 LOC component in a period of two weeks (eliminating dead code, duplication etc). 2000 LOC that were left over were still very far from being in any decent shape. We decided to go for developing the project from zero and to discard old code altogether.

On another occasion, I had to deal with some really old java code. Project contained proprietary implementation of XML parser. Not too much open source in the days when they created the application. Honestly, not a kind of code you wish to look at when maintaining something that is a typical enterprise application. Solution to this proved to be relatively easy; while XML used was not 100% well-formed, it was possible to replace it with 3rd party XML parser.

This leads us to my next thought. Start by refactoring on architectural level. Replace whole layers of application with 3rd party solutions when possible. For example, ORM tools and frameworks are only recently hitting the mainstream. It is not very likely that your application is using it. If possible, replace the persistence layer of coded SQL with some ORM tool.

The third suggestion. Know your problem domain but don’t use your legacy code base for the purpose of understating it. Reverse-engineering legacy codebase into domain knowledge is futile task; legacy code is often littered with duplication, dead code, poorly named elements etc. Try to understand the domain by speaking to domain expert/customer. Without domain knowlege you will find yourself in complete darkness. Maybe it is a feature, maybe a bug or a dead code altogether. Take a look at this story:
Refactoring Finds Dead Code

Finally, build up some reliable testing harness. Start by creating functional/integration tests and then add unit tests wherever possible.

You might wish to take a look at this article of mine, it talk about legacy VB apps, but many points are valid for any legacy code:
Moving up the technology stack: VB6 migration reality check

Good luck!

Recently I finished reading Wisdom of crowds. The basic tenant of the book is that groups are capable of excellent decision making. People also seem to be endowed with capacity to coordinate and cooperate. Ability of groups of informed laymen to make correct decisions can surpass even experts or small groups of experts. This, of course, provided some conditions are met: groups have to be diverse and decentralized and some mechanism for them to make collective decisions have to be put in place.
This is well illustrated by the following opening story from the book summed up at Wikipedia in the next sentence:

“The opening anecdote relates Francis Galton’s surprise that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox’s true butchered weight than the estimates of most crowd members, and also closer than any of the separate estimates made by cattle experts).”

A good case for direct democracy? Might as well be. Think only about this fact: today, in most democracies in the world people choose representatives and governors for periods of 4 years or more. When people are elected, they are generally elected in a certain “context” (historical, social, economical etc.). Take a look at Madrid bombings for example. There is no doubt that they had a decisive effect on elections that followed. Had the bombings occurred only 4 days later, the result of election would be radically different according to all polls. Would George W. Bush win the elections had they been taken after 9/11? I am well aware his approval ratings went berserk after the attack, but at that point he was already elected. Enough about politics…
Internet is play-ground par excellence for groups and collaboration on a large scale. Today we take Internet for granted. Still there is a number of interesting phenomena that occur on the Internet and that are closely related to human nature and society in general. Take for an example newsgroups. One person helps another, a complete stranger, and in most cases without any reward involved in the process. Quite different to how the companies in the free market behave and compete. Imagine companies sharing patents or company secrets. Well, I guess this is not so easy to imagine.
Just look at impromptu collaboration that happens on open source projects. They can involve a huge number of geographically and culturally disperse collaborators; still they can be surprisingly successful. All of this without any complex organizational structure, only slight or merit-based hierarchy etc. With Agile, such practices are getting their foot into the commercial enterprises (Self-organizing Teams).

Experts Exchange vs. Stack Overflow

What made me think about the book again is my experience with a site called Experts Exchange. The site lets users ask the questions and then the experts provide the answers. The one that asks the question assigns question points, based on the question difficulty and based on urgency he has in getting the question answered. These points are then awarded to or split between experts that provide the answer. Decision on to whom the points are awarded is up to questioner. In the case of conflict, a moderator can be asked to mediate.
When I started using Experts Exchange, points were earned by providing answers to questions made by others. You also earned some points for becoming a member, to start you up and then more point with time. They used to lock questions once someone posted the answer. The question would be locked until the person that asked the question responded if he accepts the answer or not.
Later on this has changed, so instead posting answers, all experts post comments. This way the question cannot be locked and new question momenum is not lost, nor people have to wait to give answer or post comment. The questioner then awards points to the comment he thinks is the best answer. If he thinks more than one answer deserves points, he can split points and award them to more than one answer.

Another change that occurred is that now you can pay to use Experts Exchange. You can still earn points and access site based on merit. If you pay for Premium Services you have unlimited number of points at your disposal and some other benefits, like private knowledge base. A maximum that can be gained from a single question is 2000 points. Besides points, there is also community rating system in place:

“To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.
•    The Grade of the Solution
•    The Zone Rank of the Expert Providing the Solution
•    The Number of Author and Expert Comments
•    The Number of Experts Contributing
•    The Feedback of the Community”

I think ratings were also added to the site later on. Note that ratings are applied to solution (question + all the answers as a whole).  Ratings will by no means influence how many points are earned/given away/lost. I remember this worked well and I even participated answering questions made by others earning points. Going back recently to Experts Exchange, I got a feeling there has been some change in the way things are done since the early begginings and I don’t thinkg that comercialization is the only factor. I got a feeling there is somehow too much noise that happens before you get a valid answer. Often all you get is noise. Stackoverflow also has points and reputation, but it does not work quite the same. More on that later on, let me first finish telling you my experience with Experts Exchange.

Murky water fishermen and bulldozers

Before next section, a disclaimer is in order: What follows is by no means result of some methodological scientific investigation. The sample my experience is based upon is not significant; it is also quite probable that the nature and expertise level of my questions have changed (hopefully I progresses somewhat in the last 10 years!). Finally, maybe it is just my conspiracy theory loving mind playing tricks with me. Anyhow, here we go.
After I had this rather frustrating experience with unhelpful answers on more than one occasion, I started to note a certain pattern. One type of noise makers would make some general or seemingly inoffensive comment, something like “This is interesting question; I would also like to know how it is solved.” Or “I really don’t know how to do that, but you could try looking at documentation at msdn, here is the URL that talks about the subject…”. Here is a real-life example:

“have a look at the configuration api

http://www.hibernate.org/hib_docs/v3/api/org/hibernate/cfg/Configuration.html

sorry I can’t be more help”

All of this might sound quite inoffensive, in the end its just noise – maybe its not done on purpose. The trick is however, that after some time, if question is not closed by the questioner (and why should one close the question if no acceptable answer was provided?), moderator goes on and after some warning emails closes the question for you. If you do not react, the moderator generally awards the points to those that tried to answer your question. With time, just by posting some hardly useful comments, you start earning points. I call this kind of expert “murky water fishermen”.
Another type of noise was answers that tried to change some basic premise in the question, and to provide alternative way to solve your problem. For example “Why don’t you use Entity Framework instead of LINQ2SQL?” Admittedly, sometimes putting things on their head can be useful, many times the problems are not solved because you keep trying to find the solution along the same lines. Some solutions require that you take a look at the problem from different angle, like the matchstick problem; the solution is only possible if you start thinking in 3D space. This can be especially useful for a newbie, he might not be even aware of the alternative. Most of the time however, you did your homework before coming to Experts Exchange – the basic premise of the problem cannot be changed. I call these, often arrogant coercers – bulldozers. On one occasion, after providing unhelpful alternatives, one of these bulldozers told me “Such thing cannot be done”. After some time, I solved the problem and posted the answer. Bulldozer was quite annoyed, but had to accept the defeat. Moderator gave me back the points. Not that I really wanted the points back, I was just a bit annoyed that this bulldozer had a courage and arrogance to say that my problem was impossible to solve. What surprised me is that moderator also erased the question! I thought that Experts Exchange is also about building the knowledge base (one they often mention), the answer to my question could have been useful to someone else as well, even though the dude that asked and dude that answered the question are one same person.
The first thing that comes to mind is that some mechanism for reducing noise should be put in place. Probably some kind of punishment for creating noise. This is something that Stackoverflow considers. Everyone registered can vote on both the questions and the answers. So, you get more points and better rating even if someone has liked your question, not only the answer. Even more important is the fact that anyone registered can vote on any question or answer. If someone marks your answer as Helpful you get 10 points. If your answer has been accepted, you get only 5 points more. This way, any other member is just as important as questioner. If someone thinks your question/answer was not helpful you lose points. Probably there is less need for moderation; on a large scale any bias is neutralized, there is less possibility to try to hack things like creating ghost users etc. I have not yet registered at Stackoverflow, but the next time I need to ask some question, I will give it a try. And hopefully I will remember to post part 2 of this post, after having some personal experience on how Stackoverflow measures up to Experts Exchange.
Going back to Surowiecki’s book, it seems that Experts Exchange failed one step from unleashing the “wisdom of the crowds”. What lacks is what Surowiecki calls “Aggregation” (Wikipedia: “Some mechanism exists for turning private judgments into a collective decision.”) It is up to the person that asked the question to award the points. There is a rating system in place, but separated from point system. Why not let the community be the judge on the answer value and helpfulness and award points? Also, why not let the answer receive the points more than once? I’d argue that even answer to a simple question that helped people a hundred of times is more valuable than answer to more complex question that helped someone on one occasion. Finally, why not let the questions be awarded points and why not have points taken for bad answers/questions? Using ratings system separated from point system, where contributions are heavily point-driven seems to dilute the value of community feedback.

Wisdom or Maddness?

Surowiecki’s book is an interesting read. It is even close to being practical and I would definitely recommended it if you are into social aspects of web 2.0. The book however, takes pretty much one side to the phenomenon of the crowds. This is not necessarily biased; as a matter of fact even the title of the book alludes to “Extraordinary Popular Delusions and the Madness of Crowds” book by Charles Mackay. Indeed, the madness and hysteria are the sides of crowds we generally have heard of. But by no means have we investigated them enough. One who did investigate it is Wilhelm Reich. Fortunately, The Mass Psychology of Fascism is on my desk and thanks to Internet one of my favorite movies “W.R.: Mysteries of the Organism” just a piratebay away.

Edit: In the meantime I registered at the stackoverflow.com and started participating. While you can decide to offer some of your own points as a reward, this is seldom necessary. The community driven point system seems to be fearer and overall experience is favorable so far.

While we are slowly getting into the proofreading phase, the book has already been listed for pre-order at Amazon discounted 37%. A good catch for early birds  The URL is http://www.amazon.com/gp/product/047043452X.

A few years ago I published an article for Visual System Journal on VB6 migration “Refactoring – elixir of youth for legacy VB code”.
I used “Engine-Collection-Class” pattern based application as a study case and applied different refactorings to upgrade VB6 code to .NET.
VSJ has just published an article follow-up where I deal with migration in context of .Net 3.5 and Visual Basic 2008. If you are interested in VB6 migration and upgrade process, check out my “Moving up the technology stack: VB6 migration reality check” article online at:
http://www.vsj.co.uk/articles/display.asp?id=756
and let me know what you think.

One of the tenants of .Net is language neutrality. It means you can pick the language you know and like the best and you will be able to write more or less the same program as you would in any other .Net language. This has pretty much worked as promised and today with Visual Studio .Net 2008 you get C#, C++ and VB.Net support out of the box. Some languages like J# have been made obsolete but many new, and most interestingly those based on Dynamic Language Runtime are in the works, or have been recently released.

However, once you get over your syntax preferences (as in VB vs. C style) and those die-hard habits, there isn’t that much difference in what you can do with each of these languages. For sure, VB .Net has some dynamic capabilities, and this can lead to some interesting features, like I have demonstrated in VB’s Statically Typed Wrapper for Dynamic Code article, but frankly these are not revolutionary. There must be a better reason for being able to use the same language on one platform than syntactic sugar-coating.

In VB 2008, however, you get a native XML support in VB in style of E4X programming extension for ECMAScript (or to put is simply JavaScript). Processing XML in VB using LINQ, XML literals, with ASP style templates and accessing it through XML Axis properties makes VB the best choice for working with XML on .Net platform. If you tried XSLT and didn’t like it (don’t worry, you are not the only one: MovingAwayFromXslt), then you will be thrilled with VB’s new capabilities, that can easily turn VB into real XML pipeline. There are drawbacks of course, using XML literals can often mean embedding your schema and transformations inside the VB code, but how serious these really are? How often can you change the schema or transform and not recompile anyway and is that really important? And yes, this increases VB syntactic load, but hey when was VB conservative in this aspect?

This means that choosing between VB and C# finally might matter in some cases! And that is actually a good thing. It means that “right tool for the right job” might finally mean something in VB vs. C# debate. (Yes, I know these differences are in the end only syntactic, but this times it does matter.) While both C# and VB are general-purpose programming languages, there is no reason why they shouldn’t be more specialized in some aspects. When you put it that way, this seems to be in the accord with Domain-specific languages ideas, that are getting into the spotlight recently. So, if you need to process some XML, no reason not to do it in VB (even if you prefer C# in general). Multi-language paradigm is a great asset of .Net, only if put into the practice correctly!

Writing a book is a LOT of work. When I finished writing Refactoring in VB, I didn’t think I will embark on similar adventure any time soon. Well, what to you know…

I already started working on my next book. Should be in the bookstores sometime in the spring of the next year. I plan to follow the same approach as in Refactoring in VB, meaning it will mix typical narrative section with reference style definitions. Content wise, this time I will also cover refactoring in ASP and maybe a few things more. I will keep you posted about the progress

InfoQ has published an article adapted from Chapter 1 of my book Professional Refactoring in Visual Basic. Check out the article for a interesting discussion.

LINQ to SQL is simple ORM framework from Microsoft that ships with .Net framework 3.5. Today, it seems that Microsoft is trying to divert all attention to alternative Entity Framework, but this one is not without controversy. This maybe the reason why Microsoft didn’t address one very serious limitation of LINQ to SQL in SP1 of .Net framework 3.5: LINQ to SQL works only with Microsoft SQL Server (2000 onwards I think). Or could it be something else?
Whatever the reason, this limitation is serious and will put many towards choosing free alternatives like NHibernate. Another limitation of LINQ to SQL is that it is no 100% POCO. It almost is, but the framework forces you to use System.Data.Linq.EntitySet programming one-to-many relationship (and EntityRef on the other side of relationship). This can have some awkward consequences, most obvious being existing code; should you decide to use LINQ to SQL to persist some existing classes, you will have to change original source code to accommodate this. Other ORMs do not pose this limitation, so neither LINQ to SQL should.
I am hoping that these obvious limitations are only consequence of products immaturity and that Microsoft will fix it’s first-born ORM in the future.

It’s great to see that support for refactoring in VB is constantly growing. Here is another tool on the list. JetBrains have announced 4.0 Beta of ReSharper Visual Studio Add-in. This time, ReSharper comes with support for Visual Basic. While it is still lagging a bit behind C#, since it supports refactoring of Visual Basic 8.0 code (comes with Visual Studio 2005), while C# 3.0 is fully supported including LINQ, it still supports some very useful refactorings in VB likeExtract Superclass, Pull Members Up and Push Members Down. According to my records, these are supported for the first time in VB.

These things tend to happen during those lovely quiet afternoons. I was enjoying reading about Mars colonization and genetically engineered bacteria terraforming when I got the call from our CEO. You probably already know the story, our VIC (Very Important Client, sometimes confused with GOD) has a problem. Even worst, the problem is identified in their production environment. I am to put on my superhero uniform and to go to client’s installation and face the “eye of the storm”. My mission is two-fold. My appearance is supposed to pacify the client and I might even do something about the problem, I have some whole 2 or 3 hours to come up with the solution.

Fortunately, once I meet with our guy working in-situ and our liaison (a junior manager has been assigned on client side to help us deal with the problem) the hysteria has subsided, and a few interesting details come into the light. The problem is by no means new, it has been reported more than a year ago, but for some reason nothing was done about it. It is a performance problem, so that is why it might not have been considered critical in the begging. What actually happened recently is that their CEO was annoyed by the problem and that made it hit the fan.

So, we start reviewing the app. My team-mate tells me that problem is produced only during login flow. We investigate bit further and by looking performance of components in Component Manager we are able to identify single offending component. In development environment, this component is taking some 40 seconds to respond, each time with exactly the same delay.

I ask them to monitor the component in production environment and to let me know. I will not receive the result until tomorrow, so I go back to the office. I am just too curious to wait, and I decide to take a look at component’s code. It is a COM+ component programmed in VB6 quite a few years ago. Component serves as a sort of Http proxy for stored procedures on MS SQL 2000, something of an in-house SOAP implementation that exposes stored procedures over http. Stored procedure name and parameters are received through Http request and reply rendered in form of Xml and sent back as a response.

A few lines almost immediately catch my eye:

strExecID = Request.Form("exec")

strParams = Request.Form("params")

‘ ...

Set Conn1 = CreateObject("ADODB.Connection")

Conn1.CursorLocation = adUseClient

Conn1.open strConnString

Set Rs1 = Conn1.Execute("EXEC " & strExec & " " & strParams)

‘ ... process Rs1 and generate Xml

Following problems are immediately evident:

1. Cursor Location

By setting cursor location to adUseClient, application will first pull down all the data over the wire before processing any of it. This should be changed to adUseServer.

2. Parameters

Parameters are passed as comma delimited string that is concatenated to “EXEC ” command + Stored Procedure name string that is used to execute query using the Connection object. This syntax is inferior to using ADODB Command and Parameters object. The problem in this case is that it would be probably too complicated to ask client component to supply all parameter metadata like type, size etc necessary when adding parameters to command. They decided to use position based parameter string executed on connection object instead. There are different solutions for this problem:

* Component could use some configuration data for this purpose

* ADODB Command is capable of obtaining parameter information from database. While this is not very efficient if it is done before each execution, some cache mechanism could be implemented in order to cut number of round-trips.

3. EXEC syntax

When executing stored procedures, standard ODBC syntax with CALL statement is the preferred way. This should reduce the number of compilations on the server. In order to check this, I decide to perform a small experiment.

The experiment

I place two controls on the form in VB and write the following code:

Private Sub UseCall_Click()

    Dim Count As Integer

    Dim Cn As ADODB.Connection

    Set Cn = CreateObject("ADODB.Connection")

    
    Cn.Open ("Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;" & _

    "Initial Catalog=Northwind;Data Source=TESLATEAM")

    Dim Rs As ADODB.Recordset

    Do Until Count > 100

        Set Rs = Cn.Execute("{Call [Ten Most Expensive Products]}")

        Do Until Rs.EOF

            Debug.Print Rs.Fields(1)

            Rs.MoveNext

        Loop

        Count = Count + 1

    Loop

End Sub

Private Sub UseExec_Click()

    Dim Count As Integer

    Dim Cn As ADODB.Connection

    Set Cn = CreateObject("ADODB.Connection")

    Cn.Open ("Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;" & _

    "Initial Catalog=Northwind;Data Source=TESLATEAM")

    Dim Rs As ADODB.Recordset

    Do Until Count > 10

        Set Rs = Cn.Execute("EXEC [Ten Most Expensive Products]")

        Do Until Rs.EOF

            Debug.Print Rs.Fields(1)

            Rs.MoveNext

        Loop

        Count = Count + 1

    Loop

End Sub

As you can see, the only difference between two event handlers is that in first case, {CALL } and in second EXEC syntax is used.

I open Performance Monitor and add SQL Compilations/sec counter. It is found under SQL Server:SQL Statistics performance object.

First, I press UseCall button. The counter is showing barely single SQL Compilation.

Using “CALL” syntax to execute Stored Procedure

However, when I use Exec syntax, there are repeatedly more than 10 Sql compilations per second. In production environment, this might incur significant load on the server.

Using “EXEC” syntax to execute Stored Procedure

This is enough to convince me that I am at the right track, and I proceed fixing the 3 problems I outlined above. Soon I have new version and I am ready to take the component to testing site we maintain at client’s premises.

Sobering up

I receive the monitoring results from production environment waiting in my inbox the next day. Things can really move fast when you have the right lever. The results are similar to one in testing environment, the component is taking a very long time to respond, with times raging from 700ms to over 50s. No doubt we have found our performance bottleneck.

I am off to the client with fixed code. I want to debug the component and to observe how the new code is being executed. I start executing the login test case and soon the IDE stops in debug mode reporting an error. The default Error Trapping options are not changed and IDE will break on errors in class module.

Damn, I have made the error somewhere and this might not be such an easy ride as I expected. I was already preparing for hero’s welcome back in the office. I guess that medal ceremony will have to wait for now.

Code is relatively simple and I should be able to find the bug without too much effort. I run the test case a few times but no error is produced until the command is executed. Finally, I have identified the error and it is not inside the component. The component is receiving non-existing stored procedure name for execution!

I decide to see how original version will do. The result is the same, the IDE breaks on the same error produced by non-existing stored procedure. This leaves me quite puzzled. Finally, I decide to change Error Trapping option to “Break on Unhandled Errors”. Voila! Application executes just fine, showing normal behavior, including the 40 second delay.

Following some dark alleys of error handling code, I finally fins out that client component handles the error produced in the component I just “fixed” and returns empty reply back the execution stack. As it happens, the component is not used at all and calling it is complete waist of time. And not just any waist of time, but very critical delay producing serious performance problem.

I soon implement the final solution eliminating the call to the component altogether. Application continues to work as it was supposed to be so I let the bygones by bygones. I explain the problem and the solution to the junior manager that is working with us and he is nice or content enough not to ask “But how this happened…”.

Finally, I am convinced once more that there is quite a lot of truth in Hoare’s proverbial sentence “Premature optimization is root of all evil”, cited by Diego in foreword to my book on refactoring in VB.