Code Retreat in Wroclaw

Next weekend there’s a Code Retreat in Wroclaw: http://coderetreat.wroclaw.pl. I’ll be there so If you want to cut some code together retreat with us =)

expected exception in tests

I have a feeling too many people get it wrong so let me stress:

@Test(expected = SomeException.class) // IS BAD FOR YOU

I do enjoy jUnit a lot, thanks guys for sharing! Yet, I don’t like @Test(expected) feature of jUnit. In my TDD classes I simply teach to avoid using it. I admit the feature can potentially make few tests cleaner. However, it tends to bring more trouble than value. Newbies too often misuse it which leads to poor quality tests. Therefore:

Rule 1: don’t use @Test(expected)

My colleague (cheers Bartek!) pursues Master at the Uni. They teach him testing. One day a lecturer was presenting some test code:

@Test(expected=Exception.class)
public void testSomething() {
  //line 1: throws NullPointerException
  //lines 2-30: some crappy test code
}

The test is useless. Due to some bug in the setup the test method breaks in line 1. Therefore most of the test code is not even executed. The expected exception does great job in hiding this issue; the test is green! Therefore:

Rule 2: If you violate Rule #1 then at least make the exception very specific. Example:

@Test(expected=BreadOutOfStockException.class)
public void shouldBuyBread() {}

Even if you use specific exception then the test does not document exactly where the exception is thrown. Sometimes it may lead to subtle bugs. Therefore:

Rule 3: Don’t violate Rule #1 when the test method has multiple lines.

Instead you can simply write:

@Test
public void shouldBuyBread() throws Exception {
  //given
  me.goToBakery(bakery);
  bakery.setBreadAmount(0);

  try {
    //when
    me.buyBread();
    //then
    fail();
  } catch (BreadOutOfStockException e) {}
}

Above is my preferred way of documenting ‘exceptional’ behavior. There you go, it is The Ultimate Test Template for ‘exceptional’ behavior.

JUnit is getting better. New goodies for exception handling in tests arrive. @Rule ExpectedException better documents what action triggers the exception. Also, it nicely maps to //given //when //then style. The downside is that you need some extra code if you want to interrogate the exception, for example if you want assert if exception message contains a substring (api as of 07-2010). Nevertheless:

Rule 4: Favor @Rule ExpectedException over @Test(expected) when test method has multiple lines.

Rule 5: Keep an eye on new versions of your testing libraries.

If you are already at RI in TDD than consider my Rule #1 as a safety net. You’ll never know when a new junior team member joins. He may easily misuse the patterns he sees in the codebase. Javadoc for JUnit does not say how to safely use @Test(expected) (at the moment).

If you are learning testing don’t get too upset with my dogmatic rules. If only you write any tests then I am smiling already. You smile too, write tests and be happy. Soon you’ll get your RI rank in TDD.

Rule 6: Write tests & smile!

//given //when //then forever

@Test
public void shouldDoSomethingCool() throws Exception {
    //given
	 
    //when
 
    //then

}

I like to call it the Ultimate Test Template. I’m so fond of those 3 little comments but surprisingly, I didn’t buy it at first. Micheal (it’s you, tapestry-maven-iPhone fan boy =), friend from ThoughtWorks showed me it a couple of years ago. Actually, he didn’t show it to me – I just overheard him coaching a young dev about it. That day I thought I didn’t need any hip comments because my tests were great anyway. It was foolish.

Don’t be a fool like me and start writing //given //when //then today. Life is too short for messing around – you want to get level 85 in software craftsmanship soon, right? Here’s the deal: use the template for 1 iteration and if you don’t like the results then I will give you your money back. Seriously, no matter what you think about it – buy it! BTW. If you need to document some ‘exceptional’ behavior in your test somewhere there is the template for tests with exceptions.

Lately, I’ve been selling //given //when //then quite relentlessly. I even try to sell it via Mockito api. (The link also shows how to install the template in Eclipse so don’t miss it!)

I tried to lobby for the Church of given-when-then in Krakow, Warsaw & Kiev. I heard rumors that Wroclaw develops a growing number of brothers and sisters in faith =)

I think I forgot to thank Dan North for given-when-then and Liz Keogh for the idea of BDD aliases in Mockito. There you go!

TDD unlimited

So I read a short rant that questions TDD (or unit testing – it’s hard to figure). Here’s my message:

If you want to learn your trade don’t get too excited about gurus that outsmarted the Universe and found cracks in TDD. There are way more IT celebrities that successfully practice TDD. They blog about, they teach and you know what? Their code kicks ass big time. So… Write loads of tests, try TDD, taste BDD. Learn it yourself and at some point you won’t be needing blogs to decide what side to take (Alliance or the Horde? =).

Now, if you already know your trade please try to influence others to write more tests and learn TDD. How much inspirational is this or that! Anyway, it’s just my humble request to the world. Of course, you can write whatever you want, dear influential blogger!

Mockito in Zurich

I will have a pleasure presenting Mockito at Jazoon in Zurich. My session is on Thursday – make sure you don’t miss it. During the session I’m going to show few slides and do live TDD with Mockito & other mocking frameworks. I’m going to use Infinitest. Guys, it’s huge. Brett Schuchert wrote about it, Bartek told me to try it several times. Finally I did and I must say I’m impressed. Infinitest is like transcending to a real TDD.

is debugger a modern translator?

Couple of days ago we interviewed a young adept of the craft. We let him code but he got stressed out a bit and wrote very complicated code. At some point he stopped understanding the code he wrote. So he started using debugger to figure out the meaning of the code. This is when debugger became sort of a modern electronic translator.

The thing is we probably don’t want to use debugger as a translator. We prefer to understand the code we write. Instead of debugging I usually recommend writing a small test that covers the logic in question. You can try this exercise next time you are about to launch the debugger.

Don’t get me wrong – debugger is a handy tool in every developer’s kit. It’s just the less you use it the better you are in producing high quality code. Actually, it’s probably other way around – when you get fluency with TDD/simple design you will soon discover how rarely you use debugger.

excuses for not doing dev testing

Yesterday I heart loads of excuses why the tests have not been written. I immedietaly remembered Misko Hevery‘s session at amazing geecon conference (be sure to attend next year!). Misko told a story about influencing developers to write more tests, testable code and TDD. I liked his slide where he collected all the excuses he heart from devs explaining why they haven’t written tests. Basically, Misko’s conclusion was that the only valid reason of not writing tests/TDD was lack of skill. TDD or ability to write tests is a skill like any other and it is no shame if you haven’t acquired it yet!

However, some of the excuses I heart yesterday were beyond any common sense:

• this is not really worth testing
• this is a small application
• we don’t have time
• (my favorite one) tests would make it harder to refactor

That was all bullshit and the only reason the tests were not written was that it was hard to write them. Devs didn’t know how to write tests for the DAO layer (how to ensure db is in correct state, how to maintain test database, etc.). Devs didn’t know how to write integration tests because of lack of experience with tools like Selenium. So they didn’t write any tests, not even plain jUnits.

Half a year ago, the same development team wrote loads of tests achieving high quality code base & high test coverage. Those days they wrote a django application. Django is a modern framework with built-in means of testing. It significantly contributed to the fact that developers wrote tests without cheating themselves with it’s-not-worth-testing kind of excuses (complete list above).

The interesting thing about the situation I encounter yesterday was this: junior developer wanted to write tests for some DAO code that did not have any tests (actually, there were zero tests at all…). So I asked senior developer responsible for architecture to help out setting up a TestCase that can prepare connection to test database, etc. Instead of helping, the senior guy started his mantra of excuses trying to convince junior developer not to write tests. Fortunately, I was there…

be careful with fundamentalism

One of my colleagues told me the other day:

“Szczepan, last year, when I started working in the X team someone warned me not to speak too loud about unit testing”

Apparently, there were feisty TDDers in the X team and they witch-hunted devs who not necessarily had written test-first. I guess it’s easy to become a fundamentalist of any methodology or technique. Personally, I don’t mind dogmatic coaching… so long it works. In the X team it clearly didn’t because devs were not feeling comfortable to even talk about testing.

At devoxx I met Jakub and he told me how he had introduced TDD to his team. Apparently, he used his tech lead mandate and just demanded test-fist. After a week or two of resentment, the developers started finding benefits of the new technique. They removed needles from the voodoo dolls, thanked Jakub for almost an eye-opening experience and continued to test first. Nice work! Hopefully I remember his story right – our conversation might have been after few beers.

subclass-and-override vs partial mocking vs refactoring

Attention all noble mockers and evil partial mockers. Actually… both styles are evil :) Spy, don’t mock… or better: do whatever you like just keep writing beautiful, clean and non-brittle tests.

Let’s get to the point: partial mocking smelled funny to me for a long time. I thought I didn’t need because frankly, I haven’t found a situation where I could use it.

Until few days ago when I found a place for it. Did I just come to terms with partial mocks? Maybe. Interestingly, partial mock scenario seems to be related with working with code that we don’t have full control of…

I’ve been hacking a new feature for mockito, an experiment which suppose to enhance the feedback from a failing test. On the way, I encountered a spot where partial mocking appeared handy. I decided to code the problem in few different ways and convince myself that partial mocking is a true blessing.

Here is an implementation of a JunitRunner. I trimmed it a little bit so that only interesting stuff stayed. The runner should print warnings only when the test fails. The super.run(notifier) is wrapped in a separate method so that I can replace this method from the test:

How would the test look like? I’m going to subclass-and-override to replace runTestBody() behavior. This will give me opportunity to test the run() method.

It’s ugly but shhh… let’s blame the jUnit API.

The runTestBody() method is quite naughty but I’ve got a powerful weapon at hand: partial mocking. What’s that? Typically mocking consist of using a replacement of the entire object. Partial stands for replacing only the part of real implementation, for example a single method.

Here is the test, using hypothetical Mockito partial mocking syntax. Actually, shouldn’t I call it partial stubbing?

Both tests are quite similar. I test MockitoJUnitRunner class by replacing the implementation of runTestBody(notifier). First example uses test specific implementation of the class under test. Second test uses a kind of partial mocking. This sort of comparison was very nicely done in this blog post by Phil Haack. I guess I came to the similar conclusion and I believe that:

• subclass-and-override is not worse than partial mocking
• subclass-and-override might give cleaner code just like in my example. It all depends on the case at hand, though. I’m sure there are scenarios where partial mocking looks & reads neater.

Hold on a sec! What about refactoring? Some say that instead of partial mocking we should design the code better. Let’s see.

The code might look like this. There is a specific interface JunitTestBody that I can inject from the test. And yes, I know I’m quite bad at naming types.

Now, I can inject a proper mock or an anonymous implementation of entire JunitTestBody interface. I’m not concerned about the injection style because I don’t feel it matters that much here. I’m passing JunitTestBody as an argument.

Let’s draw some conclusions. In this particular scenario choosing refactoring over partial mocking doesn’t really push me towards the better design. I don’t want to get into details but junit API constrains me a bit here so I cannot refactor that freely. Obviously, you can figure out a better refactoring – I’m just yet another java developer. On the other hand, the partial mocking scenario is a very rare finding for me. I believe there might be something wrong in my code if I had to partial mock too often. After all, look at the tests above – can you say they are beautiful? I can’t.

So,

• I cannot say subclass-and-override < partial mocking
• not always refactoring > subclass-and-override/partial mocking
• partial mocking might be helpful but I’d rather not overuse it.
• partial mocking scenario seems to lurk in situations where I cannot refactor that freely.

Eventually, I chose subclass-and-override for my implementation. It’s simple, reads nicer and feels less mocky & more natural.

should I worry about the unexpected?

The level of validation in typical state testing with assertions is different from typical interaction testing with mocks. Allow me to have a closer look at this subtle difference. My conclusion dares to question the important part of mocking philosophy: worrying about the unexpected.

Let me explain by starting from the beginning: a typical interaction based test with mocks (using some pseudo-mocking syntax):

#1

Here is what the test tells me:

– When you read the article
– then the reader.read() should be called
– and NO other method on the reader should be called.

Now, let’s have a look at typical state based test with assertions:

#2

Which means:

– When you read the article
– then the article should be read.

Have you noticed the subtle difference between #1 and #2?

In state testing, an assertion is focused on a single property and doesn’t validate the surrounding state. When the behavior under test changes some additional, unexpected property – it will not be detected by the test.

In interaction testing, a mock object validates all interactions. When the behavior under test calls some additional, unexpected method on a mock – it will be detected and UnexpectedInteractionError is thrown.

I wonder if interaction testing should be extra defensive and worry about the unexpected?

Many of you say ‘yes’ but then how come you don’t do state based testing along this pattern? I’ll show you how:

#3

Which means:

– When you read the article
– then the article should be read
– and no other state on the article should change.

Note that the assertion is made on the entire Article object. It effectively detects any unexpected state changes (e.g: if read() method changes some extra property on the article then the test fails). This way a typical state based test becomes extra defensive just like typical test with mocks.

The thing is state based tests are rarely written like that. So the obvious question is how come finding the unexpected is more important in interaction testing than in state testing?

Let’s consider pros & cons. Surely detecting the unexpected seems to add credibility and quality to the test. Sometimes however, it just gets in the way, especially when doing TDD. To explain it clearer let’s get back to the example #3: the state based test with detecting the unexpected enabled. Say I’d like to test-drive a new feature:

#4

I run the tests to find out that newly added test method fails. It’s time to implement the feature:

I run the test again and the new test passes now but hold on… the previous test method fails! Note that the existing functionality is clearly not broken.

What happened? The previous test detected the unexpected change on the article – setting the date. How can I fix it?

1. I can merge both test methods into one which is probably a good idea in this silly example. However, many times I really want to have small, separate test methods that are focused around behavior. One-assert-per-test people do it all the time.

2. Finally, I can stop worrying about the unexpected and focus on testing what is really important:

public shouldSetReadDateWhenReading() {
article.read();
assertEquals(today(), article.getReadDate());
}

public shouldReadArticle() {
article.read();
assertTrue(article.isRead());
}

Ok, I know the example is silly. But it is only to explain why worrying about unexpected may NOT be such a good friend of TDD or small&focused test methods.

Let’s get back to mocking.

Most mocking frameworks detect the unexpected by default. When new features are test-driven as new test methods, sometimes existing tests start failing due to unexpected interaction errors. What happens next?

1. Junior developers copy-paste expectations from one test to another making the test methods overspecified and less maintainable.

2. Veteran developers modify existing tests and change the expectations to ignore some/all interactions. Most mocking frameworks enables developers to ignore expectations selectively or object-wise. Nevertheless, it is still a bit of a hassle – why should I change existing tests when the existing functionality is clearly not broken? (like in example #4 – functionality not broken but the test fails). The other thing is that explicitly ignoring interactions is also a bit like overspecification. After all, to ignore something I prefer just not to write ANYTHING. In state based tests if I don’t care about something I don’t write an assertion for that. It’s simple and so natural.

To recap: worrying about the unexpected sometimes leaves me with overspecified tests or less comfortable TDD. Now, do I want to trade it for the quality? I’m talking about the quality introduced by extra defensive test?

The thing is I didn’t find a proof that the quality improved when every test worried about the unexpected. That’s why I prefer to write extra defensive tests only when it’s relevant. That’s why I really like state based testing. That’s why I prefer spying to mocking. Finally, that’s why I don’t write verifyNoMoreInteractions() in every Mockito test.

What do you think? Have you ever encountered related problems when test-driving with mocks? Do you find the quality improved when interaction testing worries about the unexpected? Or perhaps should state testing start worrying about the unexpected?