DevEvening Presentation – T4

As part of our first DevEvening, I did a presentation on T4, the little-known code generation technology built-in to Visual Studio.  It was a pleasure meeting everyone at our first meeting, and I hope it was the first of many more to come.  I also want to give a big thank you to Alex Mackey for making it all happen – We are lucky to have you in Melbourne!  Thanks also to Clarius for their generous donation of two professional licenses of their Visual T4 Editor — congratulations Tarn and David!

The slides from the presentation are below:

This was one of my first presentations in this type of forum, but I think it went relatively well.  It’s surprising how much effort goes into making a successful presentation, and I have a newfound respect to those who do this regularly!

If you have any feedback on the presentation, please comment here or let me know on twitter.  I would appreciate any feedback to help me improve.

Relevant Links

Is Ubiquitous Language Possible?

The Goal

There have been some interesting discussions recently about the importance of having a “common language” between developers and business domain experts.  DDD refers to this concept as its “Ubiquitous Language”:

“The concept is simple, that developers and the business should share a common language, that both understand to mean the same things, and more importantly, that is set in business terminology, not technical terminology.”

The intention is that once we share a common vocabulary, we can improve our requirements and design.  The class design (the business entities and their relationships) hopefully better reflects the true interactions of the business we are trying serve.

The Problem / An Example

The difficultly in identifying the “real names” typically lies somewhere between developers and the business domain experts.  However, on some very large projects, I seemed to have encountered another variation of this problem.

Let’s consider a large project, divided into multiple teams.  Each team is responsible to design and implement separate functionality, but each team has significant overlap with the other.

For example, imagine Really Big Company’s executive  management has decided to build an “improved” customized version of their current email system to communicate better with customers.  So, in assemblies our project teams, we decide to divide the project into two primary teams: Team A will build the “inbox” functionality, and Team B will build the contact management module.

Since Team A and Team B follow good DDD principles, they being working on creating their common language (shared names).  Team A discovers that the domain experts for the Inbox system refer to their mail recipients as “contacts”, and, ironically, Team B’s domain experts refer to their contacts as “customers”.

At some point, they realize that “customers” are really just one “type” of contact.  With this shared understanding, the developers create a design where [Customer] will be a subclass/child of [Contact].  The domain experts readily agree the improved design.

So now, developers will distinguish contacts from customers.  And the next time they hear domain experts refer to a “customer”, they think they can assume a[Customer].  The business agreed on it, right?!

So, the real question — did the business user really mean [Customer]?  Or did they really mean [Contact]?

Conclusion

It is unlikely that the vocabulary of the business users will change, especially when the business users are not part of the full time project staff.  Remember, the developer staff isn’t the only interaction that these domain experts will have!  So, even when we have “agreement”, it is unlikely (and probably undesirable) to change the language of the business.

So then what?

After all this, we may be tempted to throw out our ideas on common language, shared names, and ubiquitous language.  However, let’s stop to see what we have accomplished.

In our theoretical meeting, when the domain expert mentioned “customer”, the technical team can now ask “Do you mean [Customer] or [Contact]?”.  And, with any luck, the business now understands what we mean, and can give a simple answer.  “Ah, yes, you’re right.  I do mean [Contact]”.

Common language isn’t a cure-all.  But it sure helps.

I’m curious to hear your experiences as well.

Sacrificing Quality – Part II – Kaizen

Recap of Part I

In my last post, I described how sacrificing quality can adversely impact the morale of your development team.  In that post, I cited a particular quotation from Tom DeMarco in Peopleware:

“We all tend to tie our self-esteem strongly to the quality of the product we produce… Any step that you take that may jeopardize the quality of the product is like to set the emotions of your staff directly against you…

…the decision to pressure people into delivering a product that doesn’t measure up to their own quality standards is almost always a mistake.”

The rest of that post described the psychological affects that compromising quality may have on a development team.

This post will highlight another problem that compromising quality can have on your development team – making it more difficult to attract (and keep) the best developers.

Part II

In addition to the quote above, Tom DeMarco continued in Peopleware:

“We managers tend to think of quality as just another attribute of the product, something that may be supplied in varying degrees…

…The builders’ view of quality, on the other hand, is very different.  Since their self-esteem is strongly tied to the product, they tend to impose quality standards of their own.  The minimum that will satisfy them is more or less the best quality they have achieved in the past.”

In particular, notice the last sentence:  “The minimum that will satisfy them is more or less the best quality they have achieved in the past.”  This characteristic is one of the best traits we developers have – the desire to continuously improveKaizen, the process of continuous improvement, is increasingly attracting attention as a key factor in making huge long term gains, both for companies and individuals. It is also a key principle behind agile techniques such as Extreme Programming and Scrum.

In our industry, continuous improvement is not just a desire, but also a survival technique.  Things change so quickly, if we do not improve continuously, we will not only fall behind, but become obsolete in a few short years.

With these thoughts in mind, now consider what happens when a developer is told to accept a lesser quality product.  On the rare occasion, the developer might only be disappointed or annoyed.  If it becomes a rule, rather than an exception, any good developer will realize this approach will lead them quickly down a dead-end path.  And the more passionate your developer is about their career, the faster they will reach this conclusion.

Again, taking shortcuts on quality is likely to affect you more in the long run.  Where you might be able to deliver things sooner in the short term, you are more likely to lose the developers that can deliver faster in the long run.  Don’t risk losing your best people by not allowing them to do their best.

Extract Method – How Much is Too Much?

Extract Method is one of the most basic and common refactorings. In Refactoring: Improving the Design of Existing Code, Martin Fowler gives the following motivation for using Extract Method:

“Extract Method is one of the most common refactorings I do.  I look at a method that is too long or look at code that needs a comment to understand its purpose.  I then turn that fragment of code into its own method.

I prefer short, well-named methods for several reasons.  First, it increases the chances that other methods can use a method when the method is finely grained.  Second, it allows the higher-level methods to read more like a series of comments.  Overriding also is easier when the methods are finely grained.

It does take a little getting used to if you are used to seeing larger methods.  And small methods really work only when you have good names, so you need to pay attention to naming.  People sometimes ask me what length I look for in a method.  To me length is not the issue.  The key is the semantic distance between the method name and the method body.  If extracting improves clarity, do it, even
if the name is longer than the code you have extracted.”

Few would argue with the benefits of this refactoring.

But, what happens if we perform Extract Method to the extreme?  What happens if we follow Uncle Bob‘s advice and extract till we drop?

‘For years authors and consultants (like me) have been telling us that functions should do one thing. They should do it well. They should do it only.

The question is: What the hell does “one thing” mean?

After all, one man’s “one thing” might be someone else’s “two things”.’

Uncle Bob’s post provides an example of extracting until nothing else can be done, and then he ends with this comment:

“Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.”

As he predicts, many people do think he’s going too far.  Here’s a couple excerpts from the post’s comments:

“Following the flow of the data through the fully extracted version becomes difficult, since the developer will need to jump around constantly throughout the body of the class.

If the goal is to make development and maintenance easier and fully extracting the class makes it more difficult for a developer to follow the flow of the data is it better to fully extract just for the sake of following a rule?

My point is that patterns in code are easier to see when things are not broken down into such small chunks. At the fully decomposed state it isn’t obvious that an Adapter on the Matcher would simply fit into place. By decomposing the methods so fine you lose context, so much so it isn’t evident how the method relates to the accomplishing the goal of the class.”

and, another:

‘A function by definition, returns 1 result from 1 input. If there’s no reuse, there is no “should”. Decomposition is for reuse, not just to decompose. Depending on the language/compiler there may be additional decision weights.

What I see from the example is you’ve gone and polluted your namespace with increasingly complex,longer,more obscure, function name mangling which could have been achieved (quick and readable) with whitespace and comments. To mirror a previous poster, I rather see a javadoc with proper commenting than to trace what’s going on for such a simplistic case. I’m afraid to ask what you plan to do when the class is more complex and symbolExpression(..) isn’t descriptive enough!’

These arguments make a good point.  However, these arguments also apply to object-oriented code in general.  Reading and navigating object-oriented code can often be more difficult than its procedural counterparts.  However, we hope to overcome these disadvantages by creating a structure that is more readable and reusable overall.

In Uncle Bob’s example, the newer, more granular methods provide a more complete and accurate view of the behaviors and capabilities of the SymbolReplacer class.  In isolation, it might appear as overkill and “polluted namespaces”.  However, if you were maintaining a large codebase and needed to understand how to use (or reuse) SymbolReplacer, I believe Uncle Bob’s approach would make your task much easier.  You don’t need to read through javadoc (as one commenter prefers).  Instead, the method names are more clear, the size is smaller and easier to override, and the class itself almost becomes readable English.  In my opinion, these advantages outweigh the loss of navigability.

But, perhaps, as Martin Fowler mentions “it does take a little getting used to”.  Uncle Bob said almost the same thing: “Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.”

With the wisdom of those two, I think we owe it to ourselves to set aside our skepticism and give it a real try.  We can come back later, compare results, and make a decision then.  I have found that those who are willing to try their advice, in the end, never go back.  Perhaps, you will will find that your code gets cleaner and opportunities for reuse start showing themselves in surprising ways.

Sacrificing Quality Costs More Than You Think

My last post highlighted my concerns with the use of the “technical debt” metaphor, ending with this question and summary:

Should quality be compromised?  When?

…the metaphor of technical debt seems to open the issue of quality as being a gray area, something that can be “traded away”.  The decision to compromise quality should not be made lightly under any circumstances.

I realize this statement doesn’t exactly blow your mind.  Of course, quality is important.  We all believe so.

So, let’s discuss a seemingly innocent scenario we may face as a development team:

In this iteration, we need to release our software slightly earlier than anticipated.  We won’t have the time to do the usual refactoring cycle after development.

Doesn’t seem like such a big deal.  And, on paper, it’s not.

However, what message did we just send to our developers?

In Peopleware (a must read), Tom DeMarco has this to say:

“We all tend to tie our self-esteem strongly to the quality of the product we produce… Any step that you take that may jeopardize the quality of the product is like to set the emotions of your staff directly against you…

…the decision to pressure people into delivering a product that doesn’t measure up to their own quality standards is almost always a mistake.”

Many managers and team leaders may think such a reaction to a relatively small decision is inappropriate.  They would be right.  However, as humans, we seem to be programmed this way.

To understand better, consider the Broken Window Theory, which provides further examples and research outside of software development:

“Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it’s unoccupied, perhaps become squatters or light fires inside.

Or consider a sidewalk. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of trash from take-out restaurants there or breaking into cars.”

As humans, we have subconscious feelings about quality of things around us, and we treat those things accordingly.  If we have a beautiful new car, flawless and free of scratches, we keep it clean and polish it regularly.  The first ding hurts the most.  Over time, we get more scratches, they hurt less, we worry less, clean the car less, and care less.

Looking back to our example, the decision to fast-track the release by a few days was probably not worth the harm to our team’s morale and self esteem.  The impact is largely subconscious, but very real, and does the most damage.

As creators and builders, we developers are driven by the desire to produce something which makes us proud.  When robbed of this feeling, for whatever reason, it hurts more than we may even want to recognize.

Shortcomings of the Technical Debt Metaphor

The metaphor of “technical debt” is gaining in popularity to describe the implications of taking shortcuts in developing software.  Steve McConnell has an excellent post on technical debt, where he begins his discussion:

The term “technical debt” was coined by Ward Cunningham to describe the obligation that a software organization incurs when it chooses a design or construction approach that’s expedient in the short term but that increases complexity and is more costly in the long term.

As many others have already already written on this subject, I will instead focus on this question from his summary:

What do you think? Do you like the technical debt metaphor? Do you think it’s a useful way to communicate the implications of technical/business decision making to non-technical project stakeholders?

The “Good”

Firstly, yes – I believe the technical debt metaphor is a useful way to communicate such implications to non-technical stakeholders.  I’m sure most developers can relate to some version of a conversation going something like this:

The Big Boss:   Thanks, that was an excellent demo!  Let’s make sure to get this into tomorrow’s release!

You:  Umm… I still have some things to finish up.  I’m planning to have it ready for testing sometime next week.  I am on track to meet the scheduled deadline.

The Big Boss:  Finish up?  Everything is working great!  What do you have left to do?

You:  Well, there’s just a few changes I need to make… strengthening the exception handling, making a few improvements to the design, cleaning the code up a bit…

The Big Boss:  Ah, don’t worry!  The demo looked fine to me and the customers don’t care about all that!  Besides, Friday is the end of the quarter and it will be great to get this feature out now!

You:  (sigh…)

Of course, as developers we try to explain to The Big Boss that we still need to do the last iteration of refactoring.  We want to deliver code with a level of quality that we expect of ourselves.  However, often The Big Boss cannot relate.  Or, at least, he doesn’t think it’s as important as we do.  In this situation, the technical debt metaphor can be an helpful way to describe such issues using terminology that he is more likely to appreciate.

The “Bad”

Although the metaphor can be useful, it also has some important shortcomings.  These shortcomings are especially significant in dealing with non-technical stakeholders.

Concern # 1 – The term “debt” is too soft

Overly Acceptable

Financial debt is abundant and widespread, often accepted as a fact of life.  I agree that some level of technical debt is also unavoidable in doing real software development.  However, especially with a non-technical manager, the decision may be made too readily to accept technical debt without understanding the full implications.

“Interest rates” are usually low

The metaphor of “interest payments” is used to illustrate the implications of taking a more “quick and dirty” approach.  In these days of low interest loans, the connotations of this analogy may not be strong enough to describe the real impact of accepting lesser quality workmanship.

Concern # 2 – The “Interest Rate” Varies

When describing technical debt, the individual issue incurring the “debt” should be highlighted in terms of severity (it’s interest rate).  Take two possible examples of technical debt a team may choose to incur:

  1. Refactoring to improve naming of methods/variables of a new class is not performed.
  2. Unit tests are not created on this same new classes.

The first example of debt may certainly impact readability/maintainability, particularly in the long run.  However, this is probably not severe and would probably be fairly to change in the next release or two.  It is this type of debt where the use of this metaphor seems most appropriate.

The second example of debt is more severe and will be harder to “service”.  The initial code will be released without sufficient testing from the beginning.  Unnecessary dependencies may have been introduced that are difficult to extract later.  Future development may be severely hindered without the “safety net” of unit tests to refactor…  Whatever the impact, the point is that each instance of technical debt is different and must be considered independently.

Summary – Should quality be compromised?  When?

In any case, my biggest concern is the metaphor of technical debt seems to open the issue of quality as being a gray area, something that can be “traded away”.  The decision to compromise quality should not be made lightly under any circumstances.

Update:  My next post further describes the issues of compromising quality

Trustworthy Unit Tests – Testing the Data Access Layer

In the latest Hanselminutes podcast, Roy Osherove describes some of his best practices in unit testing techniques.  In the discussion, Roy describes one of the key tenets from his book which says that a unit test must be “trustworthy”.

On the surface, this tenet seems fairly obvious.  Of course, we need to trust our tests!  However, as he describes some situations where this rule is often violated, I realize I would be guilty on many occasions.

For example, I always struggle with the idea to include data access layer tests in my unit tests.  Yes, I know that testing the database is really an integration test and “purists” would argue these should not be included.  However, I feel the only way to test the data access code in a meaningful way is to hit the database.  In other words, the code in the data access layer and the related code in the database are a single logical unit.

In this podcast, Roy helps me to understand the difference, which I have summarized here:

Clarification # 1 – Data Access Layer Unit Tests are not trustworthy

With a trustworthy test, you must be able trust the results 100% of the time.  If the test fails, I know the code is broken and must be fixed.  This means I shouldn’t have to ask things like “Was the database down?”, “Was the connection string OK?”, “Was the stored procedure modified?”.  By asking these questions, it shows that I don’t trust the results and I have a poorly designed “unit test”.

Clarification # 2 – Data Access Layer tests must be integration tests

In the podcast, Roy also states that the data access layer is generally a very thin layer on top of the database, invoking some execution logic within the database itself.  For example, the data access layer will issue a command/query such as a SELECT, INSERT, UPDATE, DELETE or similar set of statements using a stored procedure.  There generally is not much else that sits inside the data access code.  So, Roy also believes that this type of code must be tested as a single unit.

However, Roy also points out that these must be also be integration tests.  I had made the assumption that since they must be tested as a unit, then they must be unit tests!  However, by doing so, I have created untrustworthy tests.  Now, it makes sense to me that the data access layer will likely not include any unit tests at all!  They will only be created as integration tests.

Clarification # 3 – Separation of Unit Tests from Integration Tests

It probably seems like an obvious observation to make, but I feel its important to distinguish:  Unit Tests and Integration Tests are often both developed using a “Unit Test Framework”.  We’re using the same tools to create both types of tests.

However, the problem occurs when we blend the two types of tests together.  Instead, we should create different projects/assemblies, so they can be run easily and independently from one other.  It should be obvious and easy for someone to get all the code out of source control, build the solution, and know which tests should pass from the beginning.  If you mix integration tests into the same project, the other developer will have to determine which tests “should” pass and which “should” fail.  This makes things more difficult than necessary for other developers and may likely mean that your tests will ignored and no longer maintained.