(clipped)  “… this is the perfect time to refactor this puppy and make it more portable, reusable, reliable, easier to configure etc. Now that’s fine and dandy, but then I started feeling a tad overwhelmed regarding to where to start. Should it be a separate library? How should it be configured? Should it use IoC? DI?

So my [admittedly subjective] question is this — given a relatively small, but quite useful class like the one below, what is a good approach to refactoring it? What questions do you ask and how do you decide what to implement or not implement? Where do you draw the line regarding configuration flexibility?

[Note: Please don't bash this code too much okay? It was written a long time ago and has functioned perfectly well baked into an in house application.]”

Before we get into the code sample, let’s address some of the key questions…

“What is a good approach to refactoring it?  What questions do you ask…?”

The first question I would ask is:

1.  Do I need to refactor this code at all?

In this case, the author commented at the end of his question that the code “was written a long time ago and has functioned perfectly well baked into an in house application”.  This comment suggests there may not be a need to refactor at all!  If the code has survived and hasn’t been modified in a long time, it probably doesn’t make sense to change it now.  However, let’s assume we have good reasons to start (e.g. need to start making a changes)…

2.  What is my objective in refactoring the code?

The term “refactoring” is becoming quite commonplace, and it is often used interchangeably with any type of code modification.  However, in Refactoring: Improving the Design of Existing Code, Fowler defined it as

“Refactoring (noun):  a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior.”

With this definition, Fowler highlights that refactoring is not any general modification to the code, but specifically a change with the intention to “make it easier to understand”.

In this episode of Hanselminutes, “Uncle Bob” Martin makes the comment:

“You are constantly mercilessly refactoring systems,
not necessarily because they need to be refactored
but because they need to remain flexible. So it’s
rather like moving the gears on a bicycle or driving
your car for no other reason than to keep it well
lubricated, and in order to make those kinds of
changes you need tests, it all boils down to tests in
the end.”

(Before we move on, I want to highlight another important fact.  You cannot be sure you haven’t changed behavior unless the code is covered by good unit tests.  So, without good unit tests, you are not really refactoring.)

With this context in mind, we can determine how to make the code more flexible, easier to understand, and cheaper to modify.

3.  What should I do?  What should I refactor?

The reason I was attracted to this particular code sample to highlight is that it is actually a fairly decent piece of code.  I can see, as the author points out, that it could have been good enough to support an existing application without many problems.  I also think this sample is reflective of the type of code I generally see in the wild:  functional, but not expressive.

Additionally, even for a small code sample, it contains quite a bit of code duplication. I feel this amount of duplication is typical (unfortunately) of the average application.

The biggest step you can make to improve your code is learning (1) how to remove duplication, and (2) how to make your code more expressive.

The slides from the presentation are below:

This was one of my first presentations in this type of forum, but I think it went relatively well.  It’s surprising how much effort goes into making a successful presentation, and I have a newfound respect to those who do this regularly!

If you have any feedback on the presentation, please comment here or let me know on twitter.  I would appreciate any feedback to help me improve.

Relevant Links

• DevEvening Melbourne (Richmond)
• Oleg Sych’s Resources on T4
• T4MVC Project on CodePlex
• T4CSS
• Clarius Visual T4 Editor

There have been some interesting discussions recently about the importance of having a “common language” between developers and business domain experts.  DDD refers to this concept as its “Ubiquitous Language”:

“The concept is simple, that developers and the business should share a common language, that both understand to mean the same things, and more importantly, that is set in business terminology, not technical terminology.”

The intention is that once we share a common vocabulary, we can improve our requirements and design.  The class design (the business entities and their relationships) hopefully better reflects the true interactions of the business we are trying serve.

The Problem / An Example

The difficultly in identifying the “real names” typically lies somewhere between developers and the business domain experts.  However, on some very large projects, I seemed to have encountered another variation of this problem.

Let’s consider a large project, divided into multiple teams.  Each team is responsible to design and implement separate functionality, but each team has significant overlap with the other.

For example, imagine Really Big Company’s executive  management has decided to build an “improved” customized version of their current email system to communicate better with customers.  So, in assemblies our project teams, we decide to divide the project into two primary teams: Team A will build the “inbox” functionality, and Team B will build the contact management module.

Since Team A and Team B follow good DDD principles, they being working on creating their common language (shared names).  Team A discovers that the domain experts for the Inbox system refer to their mail recipients as “contacts”, and, ironically, Team B’s domain experts refer to their contacts as “customers”.

At some point, they realize that “customers” are really just one “type” of contact.  With this shared understanding, the developers create a design where [Customer] will be a subclass/child of [Contact].  The domain experts readily agree the improved design.

So now, developers will distinguish contacts from customers.  And the next time they hear domain experts refer to a “customer”, they think they can assume a[Customer].  The business agreed on it, right?!

So, the real question — did the business user really mean [Customer]?  Or did they really mean [Contact]?

Conclusion

It is unlikely that the vocabulary of the business users will change, especially when the business users are not part of the full time project staff.  Remember, the developer staff isn’t the only interaction that these domain experts will have!  So, even when we have “agreement”, it is unlikely (and probably undesirable) to change the language of the business.

So then what?

After all this, we may be tempted to throw out our ideas on common language, shared names, and ubiquitous language.  However, let’s stop to see what we have accomplished.

In our theoretical meeting, when the domain expert mentioned “customer”, the technical team can now ask “Do you mean [Customer] or [Contact]?”.  And, with any luck, the business now understands what we mean, and can give a simple answer.  “Ah, yes, you’re right.  I do mean [Contact]”.

Common language isn’t a cure-all.  But it sure helps.

I’m curious to hear your experiences as well.

In my last post, I described how sacrificing quality can adversely impact the morale of your development team.  In that post, I cited a particular quotation from Tom DeMarco in Peopleware:

“We all tend to tie our self-esteem strongly to the quality of the product we produce… Any step that you take that may jeopardize the quality of the product is like to set the emotions of your staff directly against you…

…the decision to pressure people into delivering a product that doesn’t measure up to their own quality standards is almost always a mistake.”

The rest of that post described the psychological affects that compromising quality may have on a development team.

This post will highlight another problem that compromising quality can have on your development team – making it more difficult to attract (and keep) the best developers.

Part II

In addition to the quote above, Tom DeMarco continued in Peopleware:

“We managers tend to think of quality as just another attribute of the product, something that may be supplied in varying degrees…

…The builders’ view of quality, on the other hand, is very different.  Since their self-esteem is strongly tied to the product, they tend to impose quality standards of their own.  The minimum that will satisfy them is more or less the best quality they have achieved in the past.”

In particular, notice the last sentence:  “The minimum that will satisfy them is more or less the best quality they have achieved in the past.”  This characteristic is one of the best traits we developers have – the desire to continuously improve.  Kaizen, the process of continuous improvement, is increasingly attracting attention as a key factor in making huge long term gains, both for companies and individuals. It is also a key principle behind agile techniques such as Extreme Programming and Scrum.

In our industry, continuous improvement is not just a desire, but also a survival technique.  Things change so quickly, if we do not improve continuously, we will not only fall behind, but become obsolete in a few short years.

With these thoughts in mind, now consider what happens when a developer is told to accept a lesser quality product.  On the rare occasion, the developer might only be disappointed or annoyed.  If it becomes a rule, rather than an exception, any good developer will realize this approach will lead them quickly down a dead-end path.  And the more passionate your developer is about their career, the faster they will reach this conclusion.

Again, taking shortcuts on quality is likely to affect you more in the long run.  Where you might be able to deliver things sooner in the short term, you are more likely to lose the developers that can deliver faster in the long run.  Don’t risk losing your best people by not allowing them to do their best.

“Extract Method is one of the most common refactorings I do.  I look at a method that is too long or look at code that needs a comment to understand its purpose.  I then turn that fragment of code into its own method.

I prefer short, well-named methods for several reasons.  First, it increases the chances that other methods can use a method when the method is finely grained.  Second, it allows the higher-level methods to read more like a series of comments.  Overriding also is easier when the methods are finely grained.

It does take a little getting used to if you are used to seeing larger methods.  And small methods really work only when you have good names, so you need to pay attention to naming.  People sometimes ask me what length I look for in a method.  To me length is not the issue.  The key is the semantic distance between the method name and the method body.  If extracting improves clarity, do it, even
if the name is longer than the code you have extracted.”

Few would argue with the benefits of this refactoring.

But, what happens if we perform Extract Method to the extreme?  What happens if we follow Uncle Bob‘s advice and extract till we drop?

‘For years authors and consultants (like me) have been telling us that functions should do one thing. They should do it well. They should do it only.

The question is: What the hell does “one thing” mean?

After all, one man’s “one thing” might be someone else’s “two things”.’

Uncle Bob’s post provides an example of extracting until nothing else can be done, and then he ends with this comment:

“Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.”

As he predicts, many people do think he’s going too far.  Here’s a couple excerpts from the post’s comments:

“Following the flow of the data through the fully extracted version becomes difficult, since the developer will need to jump around constantly throughout the body of the class.

If the goal is to make development and maintenance easier and fully extracting the class makes it more difficult for a developer to follow the flow of the data is it better to fully extract just for the sake of following a rule?

My point is that patterns in code are easier to see when things are not broken down into such small chunks. At the fully decomposed state it isn’t obvious that an Adapter on the Matcher would simply fit into place. By decomposing the methods so fine you lose context, so much so it isn’t evident how the method relates to the accomplishing the goal of the class.”

and, another:

‘A function by definition, returns 1 result from 1 input. If there’s no reuse, there is no “should”. Decomposition is for reuse, not just to decompose. Depending on the language/compiler there may be additional decision weights.

What I see from the example is you’ve gone and polluted your namespace with increasingly complex,longer,more obscure, function name mangling which could have been achieved (quick and readable) with whitespace and comments. To mirror a previous poster, I rather see a javadoc with proper commenting than to trace what’s going on for such a simplistic case. I’m afraid to ask what you plan to do when the class is more complex and symbolExpression(..) isn’t descriptive enough!’

These arguments make a good point.  However, these arguments also apply to object-oriented code in general.  Reading and navigating object-oriented code can often be more difficult than its procedural counterparts.  However, we hope to overcome these disadvantages by creating a structure that is more readable and reusable overall.

In Uncle Bob’s example, the newer, more granular methods provide a more complete and accurate view of the behaviors and capabilities of the SymbolReplacer class.  In isolation, it might appear as overkill and “polluted namespaces”.  However, if you were maintaining a large codebase and needed to understand how to use (or reuse) SymbolReplacer, I believe Uncle Bob’s approach would make your task much easier.  You don’t need to read through javadoc (as one commenter prefers).  Instead, the method names are more clear, the size is smaller and easier to override, and the class itself almost becomes readable English.  In my opinion, these advantages outweigh the loss of navigability.

But, perhaps, as Martin Fowler mentions “it does take a little getting used to”.  Uncle Bob said almost the same thing: “Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.”

With the wisdom of those two, I think we owe it to ourselves to set aside our skepticism and give it a real try.  We can come back later, compare results, and make a decision then.  I have found that those who are willing to try their advice, in the end, never go back.  Perhaps, you will will find that your code gets cleaner and opportunities for reuse start showing themselves in surprising ways.

Should quality be compromised?  When?

…the metaphor of technical debt seems to open the issue of quality as being a gray area, something that can be “traded away”.  The decision to compromise quality should not be made lightly under any circumstances.

I realize this statement doesn’t exactly blow your mind.  Of course, quality is important.  We all believe so.

So, let’s discuss a seemingly innocent scenario we may face as a development team:

In this iteration, we need to release our software slightly earlier than anticipated.  We won’t have the time to do the usual refactoring cycle after development.

Doesn’t seem like such a big deal.  And, on paper, it’s not.

However, what message did we just send to our developers?

In Peopleware (a must read), Tom DeMarco has this to say:

“We all tend to tie our self-esteem strongly to the quality of the product we produce… Any step that you take that may jeopardize the quality of the product is like to set the emotions of your staff directly against you…

…the decision to pressure people into delivering a product that doesn’t measure up to their own quality standards is almost always a mistake.”

Many managers and team leaders may think such a reaction to a relatively small decision is inappropriate.  They would be right.  However, as humans, we seem to be programmed this way.

To understand better, consider the Broken Window Theory, which provides further examples and research outside of software development:

“Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it’s unoccupied, perhaps become squatters or light fires inside.

Or consider a sidewalk. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of trash from take-out restaurants there or breaking into cars.”

As humans, we have subconscious feelings about quality of things around us, and we treat those things accordingly.  If we have a beautiful new car, flawless and free of scratches, we keep it clean and polish it regularly.  The first ding hurts the most.  Over time, we get more scratches, they hurt less, we worry less, clean the car less, and care less.

Looking back to our example, the decision to fast-track the release by a few days was probably not worth the harm to our team’s morale and self esteem.  The impact is largely subconscious, but very real, and does the most damage.

As creators and builders, we developers are driven by the desire to produce something which makes us proud.  When robbed of this feeling, for whatever reason, it hurts more than we may even want to recognize.

The term “technical debt” was coined by Ward Cunningham to describe the obligation that a software organization incurs when it chooses a design or construction approach that’s expedient in the short term but that increases complexity and is more costly in the long term.

As many others have already already written on this subject, I will instead focus on this question from his summary:

What do you think? Do you like the technical debt metaphor? Do you think it’s a useful way to communicate the implications of technical/business decision making to non-technical project stakeholders?

The “Good”

Firstly, yes – I believe the technical debt metaphor is a useful way to communicate such implications to non-technical stakeholders.  I’m sure most developers can relate to some version of a conversation going something like this:

The Big Boss:   Thanks, that was an excellent demo!  Let’s make sure to get this into tomorrow’s release!

You:  Umm… I still have some things to finish up.  I’m planning to have it ready for testing sometime next week.  I am on track to meet the scheduled deadline.

The Big Boss:  Finish up?  Everything is working great!  What do you have left to do?

You:  Well, there’s just a few changes I need to make… strengthening the exception handling, making a few improvements to the design, cleaning the code up a bit…

The Big Boss:  Ah, don’t worry!  The demo looked fine to me and the customers don’t care about all that!  Besides, Friday is the end of the quarter and it will be great to get this feature out now!

You:  (sigh…)

Of course, as developers we try to explain to The Big Boss that we still need to do the last iteration of refactoring.  We want to deliver code with a level of quality that we expect of ourselves.  However, often The Big Boss cannot relate.  Or, at least, he doesn’t think it’s as important as we do.  In this situation, the technical debt metaphor can be an helpful way to describe such issues using terminology that he is more likely to appreciate.

The “Bad”

Although the metaphor can be useful, it also has some important shortcomings.  These shortcomings are especially significant in dealing with non-technical stakeholders.

Concern # 1 – The term “debt” is too soft

Overly Acceptable

Financial debt is abundant and widespread, often accepted as a fact of life.  I agree that some level of technical debt is also unavoidable in doing real software development.  However, especially with a non-technical manager, the decision may be made too readily to accept technical debt without understanding the full implications.

“Interest rates” are usually low

The metaphor of “interest payments” is used to illustrate the implications of taking a more “quick and dirty” approach.  In these days of low interest loans, the connotations of this analogy may not be strong enough to describe the real impact of accepting lesser quality workmanship.

Concern # 2 – The “Interest Rate” Varies

When describing technical debt, the individual issue incurring the “debt” should be highlighted in terms of severity (it’s interest rate).  Take two possible examples of technical debt a team may choose to incur:

1. Refactoring to improve naming of methods/variables of a new class is not performed.
2. Unit tests are not created on this same new classes.

The first example of debt may certainly impact readability/maintainability, particularly in the long run.  However, this is probably not severe and would probably be fairly to change in the next release or two.  It is this type of debt where the use of this metaphor seems most appropriate.

The second example of debt is more severe and will be harder to “service”.  The initial code will be released without sufficient testing from the beginning.  Unnecessary dependencies may have been introduced that are difficult to extract later.  Future development may be severely hindered without the “safety net” of unit tests to refactor…  Whatever the impact, the point is that each instance of technical debt is different and must be considered independently.

Summary – Should quality be compromised?  When?

In any case, my biggest concern is the metaphor of technical debt seems to open the issue of quality as being a gray area, something that can be “traded away”.  The decision to compromise quality should not be made lightly under any circumstances.

Update:  My next post further describes the issues of compromising quality

On the surface, this tenet seems fairly obvious.  Of course, we need to trust our tests!  However, as he describes some situations where this rule is often violated, I realize I would be guilty on many occasions.

For example, I always struggle with the idea to include data access layer tests in my unit tests.  Yes, I know that testing the database is really an integration test and “purists” would argue these should not be included.  However, I feel the only way to test the data access code in a meaningful way is to hit the database.  In other words, the code in the data access layer and the related code in the database are a single logical unit.

In this podcast, Roy helps me to understand the difference, which I have summarized here:

Clarification # 1 – Data Access Layer Unit Tests are not trustworthy

With a trustworthy test, you must be able trust the results 100% of the time.  If the test fails, I know the code is broken and must be fixed.  This means I shouldn’t have to ask things like “Was the database down?”, “Was the connection string OK?”, “Was the stored procedure modified?”.  By asking these questions, it shows that I don’t trust the results and I have a poorly designed “unit test”.

Clarification # 2 – Data Access Layer tests must be integration tests

In the podcast, Roy also states that the data access layer is generally a very thin layer on top of the database, invoking some execution logic within the database itself.  For example, the data access layer will issue a command/query such as a SELECT, INSERT, UPDATE, DELETE or similar set of statements using a stored procedure.  There generally is not much else that sits inside the data access code.  So, Roy also believes that this type of code must be tested as a single unit.

However, Roy also points out that these must be also be integration tests.  I had made the assumption that since they must be tested as a unit, then they must be unit tests!  However, by doing so, I have created untrustworthy tests.  Now, it makes sense to me that the data access layer will likely not include any unit tests at all!  They will only be created as integration tests.

Clarification # 3 – Separation of Unit Tests from Integration Tests

It probably seems like an obvious observation to make, but I feel its important to distinguish:  Unit Tests and Integration Tests are often both developed using a “Unit Test Framework”.  We’re using the same tools to create both types of tests.

However, the problem occurs when we blend the two types of tests together.  Instead, we should create different projects/assemblies, so they can be run easily and independently from one other.  It should be obvious and easy for someone to get all the code out of source control, build the solution, and know which tests should pass from the beginning.  If you mix integration tests into the same project, the other developer will have to determine which tests “should” pass and which “should” fail.  This makes things more difficult than necessary for other developers and may likely mean that your tests will ignored and no longer maintained.

Pedram Rezaei wrote an excellent article on lazy evaluation in C#, which helped me better understand lazy evaluation support in C#.  However, a seemingly minor comment from his post also struck me:

As you can see both Where and Select methods also return an instance of a type implementing IEnumerable<string> meaning that they are not executed until an enumerator is created… and its MoveNext method is called…

For some reason, I never realized returning IEnumerable<> meant the closure would not be evaluated until each MoveNext invocation.  Even though I knew that LINQ queries were generally “lazily evaluated”, I had not realized that this was the reason.

In learning this, I should expand on this statement from my last post:

** ToList( ) is only required if you want to maintain the return type as List<Employee>.  Otherwise, it is probably more appropriate to change to IEnumerable<Employee> here.

The referenced code block:

public List<employee> Managers(List<employee> emps)
{
return emps.Where(e => e.IsManager).ToList();
}

Although the statement and code above is still acceptable, it will behave in a way that wasn’t obvious to me at the time.  By converting the IEnumerable<> to a List<>, I had changed the implementation from being ‘lazily evaluated’ to being ‘eagerly evaluated’.  This isn’t necessarily a bad thing, but it is important to be aware of the difference (see this article by Sam Allen for more information).

One of the terms I wanted to understand better was “closures”.  In the demonstration, I was familiar with the usages (like anonymous delegates in C# or blocks from Smalltalk), but I wanted to learn more.  My investigation led me to a relatively old post on the topic from Martin Fowler.  At the bottom of his article is another reference to the C# implementation of closures, based on C# 2.0.

As these articles were written in 2004, C# has since added more support for closures in C# 3.0 with lamba expressions, and again with LINQ and extension methods in 3.5.  I found it interesting to rewrite the same examples in both C# 3.0 and 3.5:

Example 1 – List of Employees who are Managers

Fowler’s Original Example in Ruby

def managers(emps)
  return emps.select {|e| e.isManager}
end

C# 2.0 Example from Walnes

public List<Employee> Managers(List<Employee> emps)
{
    return emps.FindAll(
        delegate(Employee e)
        {
            return e.IsManager;
        }
    );
}

C# 3.0

We can now replace the anonymous delegate with a smaller lamba expression:

public List<Employee> Managers(List<Employee> emps)
{
    return emps.FindAll(e => e.IsManager);
}

LINQ and 3.5

We can now use the extension methods on IEnumerable<T> from which List<T> derives.  In this case, the Where method is appropriate.

** ToList( ) is only required if you want to maintain the return type as List<Employee>.  Otherwise, it is probably more appropriate to change to IEnumerable<Employee> here.  [Update: see this post for more differences regarding these return types.]

public List<Employee> Managers(List<Employee> emps)
{
    return emps.Where(e => e.IsManager).ToList();
}

Now, in 3.5, the C# code is nearly identical in structure and clarity as the orginal ruby example.

Example 2 – referring to a local variable in the closure block

Fowler’s Original Example in Ruby

def highPaid(emps)
  threshold = 150
  return emps.select {|e| e.salary > threshold}
end

C# 2.0 Example from Walnes

public List<Employee> HighPaid(List<Employee> emps)
{
    int threshold = 150;
    return emps.FindAll(delegate(Employee e)
    {
        return e.Salary > threshold;
    });
}

C# 3.0

Again, we can now replace the anonymous delegate with a smaller lamba expression:

public List<Employee> HighPaid(List<Employee> emps)
{
    int threshold = 150;
    return emps.FindAll(e => e.Salary > threshold);
}

LINQ and 3.5

Again, we can now use the extension methods on IEnumerable<T> from which List<T> derives.  In this case, the Where method is appropriate.

** Again, ToList( ) is only required if you want to maintain the return type as List<Employee>.  Otherwise, it is probably more appropriate to change to IEnumerable<Employee> here.  [Update: see this post for more differences regarding these return types.]

public List<Employee> HighPaid(List<Employee> emps)
{
    int threshold = 150;
    return (List<Employee>) emps.Where(e => e.Salary > threshold).ToList();
}

Now, in 3.5, the C# code is nearly identical in structure and clarity as the orginal ruby example.

Example 3 – returning a closure from a method

Fowler’s Original Example in Ruby

def paidMore(amount)
  return Proc.new {|e| e.salary > amount}
end

C# 2.0 Example from Walnes

public Predicate<Employee> PaidMore(int amount)
{
    return delegate(Employee e)
    {
        return e.Salary > amount;
    };
}

C# 3.0

Again, we can now replace the anonymous delegate with a smaller lamba expression:

public Predicate<Employee> PaidMore(int amount)
{
    return e => e.Salary > amount;
}

There would be no change to the above in 3.5 (same as C# 3.0) here.

Example 4 – Create a function using the closure & assign to a variable

Fowler’s Original Example in Ruby

highPaid = paidMore(150)
john = Employee.new
john.salary = 200
print highPaid.call(john)

C# 2.0 Example from Walnes

Predicate<employee> highPaid = PaidMore(150);
Employee john = new Employee();
john.Salary = 200;
Console.WriteLine(highPaid(john));

There would be no change to the above in 3.0 or 3.5 (same as C# 2.0) here.

Download source here to see complete C# examples.