Unit Test Quandry

Hello again David

I must say you do have valid points and it's obvious you know what you talk about, but I must say I still disagree.

If I write 5 lines of code, compile, run ALL my tests (in less than 5 seconds) see that they're all green, then I KNOW that I haven't introduced a bug.

If I do the same but only test those tests that I assume will be affected by my changes, and they all turn up green, then I ASSUME that I haven't introduced any new errors.

I typically would not check in a mere 5 lines of code (as I'm probably in the middle of something anyway) so if the whole suite is only run at checkin, then I won't know until lunchtime if my assumptions are correct.

say that I check in, go to lunch and come back and am greeted with an email from the automatic build/test process (yes we also use CC.net) saying that some tests failed.

I now most likely will have to spend a good 5 minutes tracking down which of my changes introduced the error. Furthermore I'm feeling stupid that code I know I've tested actually failed.

I'm sure you know of TDD, but let me just quote a snippet from a website that gives instructions on it. This is from: http://www.agiledata.org/essays/tdd.html
---
A significant advantage of TDD is that it enables you to take small steps when writing software. This is a practice that I have promoted for years because it is far more productive than attempting to code in large steps. For example, assume you add some new functional code, compile, and test it. Chances are pretty good that your tests will be broken by defects that exist in the new code. It is much easier to find, and then fix, those defects if you've written two new lines of code than two thousand. The implication is that the faster your compiler and regression test suite, the more attractive it is to proceed in smaller and smaller steps. I generally prefer to add a few new lines of functional code, typically less than ten, before I recompile and rerun my tests.
---

So if I am to test everything always (which in MY opinion is not a waste of time, but actually saves you time in the end), having unit tests run fast is important.

i agree with you: mocking the DB is not a good idea. but mocking the RESULTS that the DB gives is an excellent idea, as that makes the tests go that much faster. Also: most of my tests are NOT on basic CRUD things, as there are so many other interesting things to test in the business domain. Ideally I would like to not have to write CRUD at all and focus 100% on the domain problem at hand (to me CRUD is just something you need to do to get the problem solved.. no customer I've met has ever said "we need a program that can load into memory and persist to DB invoice objects")

I hope this clarifies why speed is important to me, and I also hope that you or anyone else who is bound to have much more experience than me with CSLA can give some practical advice on HOW to achieve a superfast test-suite, and not just debate over WHY you would need it.

for the record, we have had projects with 1100 unit tests running in less than 10 seconds (no DB-interaction in the tests mind you, they were in a different suite)

also, just on the end here I'd like to say that in my opinion if you run 10000 unit tests that basically test each part of the system, you're still doing UNIT testing, as you test each thing in isolation. Integration / application testing assumes that each test goes through more than just a unit

Andreas,

Your points are valid and I can see from the size of the projects you're working with you know what you're talking about.  But here's the thing.

andreakn:

If I write 5 lines of code, compile, run ALL my tests (in less than 5 seconds) see that they're all green, then I KNOW that I haven't introduced a bug.

If I write 5 lines of code, compile, run the 10 tests that completely test the behaviour of my unit in less than 5 seconds, see they're all green, then I KNOW that I haven't introduced a bug.

The difference is that I've only run 10 tests, not the entire suite, but I still KNOW that the unit I'm working on is 100% accurate.  Those 10 tests give me complete 100% coverage of the functionality of my unit.  Any code that uses my unit will still work exactly as required.  I don't need to test the rest of the code to prove that, I already KNOW - it's not an assumption.

My point is that you shouldn't need to run all the tests to KNOW that the unit works 100% correctly.  You only need to run the exact number of tests required.

If you need to run any more than that, then the unit tests for that unit are not complete.  And that is where the problem is.

So I agree with the sentiment of the article you quote and I would agree that on the whole the way we work is to write small "units" of software with the associated "unit" test cases.

And I'm sure you'd agree that TDD (as a concept), along with the associated testing tools like NUnit and VS 2003/2005, provides a much easier way to do this level of testing compared to the way we did it 5-10 years ago.

hi david and andreas,

thanks for keeping this thread alive. 

just like andreas, i also prefer running all the unit test project to be really sure that i don't break anything.  i don't have to browse for the corresponding testfixtures for the modified classes and run it.  i prefer running the whole unit test project since i usually have changes that span a number of code sheets and it will be tedious to select just the ones that are supposed to be affected by the changes.

i'm just new to tdd and unit testing and would love to know how to really work around this tight coupling of csla with the datasource in order to make it really testable with bearable speed.

currently, i'm working around with another creation method from the collection classes to create an empty one to be used by the unit tests.  the unit tests therefore are adding items to it to mimic the Fetch creation method.  i thought i could include somewhat a switch inside dataportal_xx methods where one will not hit the DB but i also don't know how to do it.

jokiz

I think that one of the real benefits from this CSLA forum is that you can get some great discussions on topics that are not directly CSLA related - although obviously everyone shares a common interest in the CSLA framework.  This thread falls into that category.

If you're making multiple changes across several source files, then running the entire test suite is probably the easiest way to make sure you've not broken something - agreed.  That's because you've changed multiple units - so you need to run multiple unit tests.

With regard to the testing of CSLA objects, have you seen the latest blog entry about Mock Objects from Fredrik Normen.  He presents an interesting way to create objects, passing in mock objects via a special constructor method.  Have a look for some possible ideas.

Hey there,

I've been reading this whole thread. Obviously there are two camps on this discussion. The basic fact is that you can't test a single business object in CSLA without running code from another class. This is called object coupling. If you want to have a maintainable system, you need to go for object cohesion, not coupling. (for a  good explanation of the difference, see http://www.toa.com/pub/oobasics/oobasics.htm#ococ).

There are of course alternatives to this. The thing I came up with is the following. I create a new interface called IDataPortal. This interface will declare all the public methods of the Csla.DataPortal (I'm not sure about the eventhandlers though, but we'll come to that later).

public interface IDataPortal

{

object Create(object criteria);

T Create<T>();

T Create<T>(object criteria);

...

}

I then create a new class that implements this interface and maps them one on one with the Csla.DataPortal:

public class MyDataPortal : IDataPortal

{

public object Create(object criteria)

{

return DataPortal.Create(criteria);

}

...

}

On my business object, the code changes slightly:

public class MyBusinessObject : BusinessBase<MyBusinessObject > {

private static IDataPortal _dataPortal;

public static IDataPortal LocalDataPortal

{

get

{

if (null == _dataPortal) _dataPortal = new MyDataPortal(); //Instantiate default dataportal

return _dataPortal;

}

set { _dataPortal = value; }

}

...

 

When I create a new business object, I then call:

LocalDataPortal.Create<Sheet>();

In your test you would write:

MyBusinessObject.LocalDataPortal = MockDataPortal;

Expect.Once.On(MockDataPortal).Method("Create").Will(Return.Value(whateverYouWantItToReturn);

MyBusinessObject myNewObject = MyBusinessObject.NewObject();

 

I'm not really happy with the fact that the members of MyBusinessObject have to be static, as this will mean that every test that doesn't initilaize the LocalDataPortal property will end up with the MockObject of the previous test, or even the default that hits the database.. But hey, this makes it testable. You can even have tests hit the database by setting the property to null. If some members need to call Csla.ApplicationContext.User, this is ok, because the Csla.ApplicationContext.User property's type is an IPrincipal, which can also be mocked.

I'm not sure how deep the implications are, as I don't know the Csla framework that well yet. But this gives me the reassurance that a solution can be found.

UmpSens:

Obviously there are two camps on this discussion. The basic fact is that you can't test a single business object in CSLA without running code from another class.

Hmm, I don't think that properly describes the camps.  I think its more of  a 'mocking costs more time than the benefits realized.'  After all, the Csla code has its own unit tests and, assuming they pass, why take them time to mock its behavior away?  What did you prove if your BO interacts correctly with mock?  Well nothing really.. you gain no knowlege that the BO will operate correctly with the actual production code you want to release.

Then on the other side you have mock advocates.

I'm not trying to prove my object interacts correctly with the Csla framework. I'm trying to bypass the framework. When I test a business object that uses tha csla framework, I assume two things:

• The Csla framework does what it's supposed to do without bugs (some might consider this bold, however if you want to test the Csla framework, this is not the right place to do it)
• My business object is buggy until all it's tests pass

When I instantiate a business object with the default Csla implementation guidelines, I actually run at least three methods in my business object:

• the static factory method to create the object (calling the Csla.DataPortal.Create)
• the business object's constructor, which will probably initialize the object and children objects
• The static DataPortal_Create method in my business object

So I'm actually testing the implementation of three methods in one single call. If it runs well, all is ok. However, if one of them fails, I don't know which one did. What I want to do is test each method separately.

So to test a single method of my business object, I am writing my contract first, setting up the expected calls it will do to the framework or any external code. That way, I know the calls were made, and the method did what was expected. I can apply that to all the methods of my class individually.

I think a potential problem with the approach you are taking is that you are changing the behaviour of your BO (by re-implementing the Data Portal a different way) to make it testable.  That doesn't seem quite right to me.

I understand what you're trying to do and why.  And I understand the argument from the pro-Mock Objects camp as to why you might want to do this as well. 

But altering your design just to make something testable, is that really the way to go?  Is that really what TDD is promoting?

Surely, you should stick with the design and find a way to test your BO that does not involve changing the design.  It might be harder to do it this way, but it's more correct.

What's to stop a developer using your LocalDataPortal property for something other than unit testing as part of your application?

 

I'd like to introduce a slightly different question if I may... the concept of unit testing a CSLA business layer has had me in a quandry for a couple of years.

Based on how David describes his tests, it seems to me there's two levels of unit testing the BL.

Level1 - (what david describes) You are basically testing just the CRUD operations of the business object, with the goal of catching bugs caused by changes to the database schema or the data access code of the business object.

Level2 - You are testing the business logic of each BO.  It's not just that I can retrieve a previously saved BO, this level is also testing that if I change PropertyX back and forth between these 10 values the IsValid changes appropriately.  That if I read invalid data from the database the BO's IsValid is false after loading.  And so on and so on.

The quandry for me is that it seems like to achieve Level2 you'd have to spend a ridiculous amount of time writing and maintaining all the tests.  I've pretty muched dismissed it as too burdensome.

So the question becomes whether or not Level1 testing is really worthwhile.  How often do you make and catch the type of mistake that it would detect?

Dan, I think you make a good point and one that I would argue comes back to something I said in an earlier post some time ago - it depends on your definition of a "unit".

So, IMO, I think the two levels you describe are actually part of what I choose to call the "unit tests" for a BO.  My 100% functional unit (i.e. my BO) has to behave exactly the way I expect and it has to persist itself.  But that's my definition of a "unit" in this scenario.  Other people may have a different opinion.

And I agree that the creation and maintenance of all the test code to perform all the required tests is a large task, but I'm sure we'd all like it as an ideal to aim for.  However, we (you and me both!) choose not to write the level 2 tests, because we want to spend our time writing the application and not writing test cases.  I'm not saying that we don't have any level 2 style tests at all (as we have some), but we don't have the ideal 100% coverage for each BO.

So that's why we developed an abstract test base class which allows us to test the CRUD part of the "unit test".  This didn't take very long and gives us basic coverage for any BO. 

Does it test the basic CRUD functionality of a BO?  Yes
Did it take long to setup?  No
Does it test the business rule functionality of a BO?  No.
Would that take a long time to setup?  Yes.

So, it's a compromise we're prepared to take.  Yes, we may find that some bugs "get through the cracks" because we didn't write all the tests for all the BOs.  But against that we're getting more of the application development completed.

It's a classic compromise trade-off between: test coverage vs. project delivery.

UmpSens:

I'm not trying to prove my object interacts correctly with the Csla framework. I'm trying to bypass the framework. When I test a business object that uses tha csla framework, I assume two things:

• The Csla framework does what it's supposed to do without bugs (some might consider this bold, however if you want to test the Csla framework, this is not the right place to do it)
• My business object is buggy until all it's tests pass

Exactly.  Part of the behavior of the object is how it interacts with the dataportal.  If you don't know it interacts correctly (because you've mocked away that functionality),  you don't know for sure that all of our objects behavior is correct.  How can you say you have confidence that your BO is correct if the getting / modification of data isn't being tested? 

UmpSens:

When I instantiate a business object with the default Csla implementation guidelines, I actually run at least three methods in my business object:

• the static factory method to create the object (calling the Csla.DataPortal.Create)
• the business object's constructor, which will probably initialize the object and children objects
• The static DataPortal_Create method in my business object

So I'm actually testing the implementation of three methods in one single call. If it runs well, all is ok. However, if one of them fails, I don't know which one did. What I want to do is test each method separately.

So to test a single method of my business object, I am writing my contract first, setting up the expected calls it will do to the framework or any external code. That way, I know the calls were made, and the method did what was expected. I can apply that to all the methods of my class individually.

It seems to me its easier to set through the code than to create mock's for the Csla functionality.  It also seems less error prone, since any code may have bugs, including the mocks themselves.  I also test the expected behavior, but by thoroghly inspecting the object once its returned.  Usually its easy to see why I didn't get the expected results, but if not, I can simply trace the code which is at some point going to be the production code.  That is what I think is key; testing your actual production code.  Using mocks, you're not doing that.

Just my sixteen cents (after inflation).
Andy

Dan,

I would say not to skip out on level two, but instead decide what to test in level two, just like in level one.

Should you test that FirstName makes the object invalid if it does not contain a value?  Probably not.  That's a simple rule, just one line of code and using the Csla framework.  But I think you should test more complex rules.

Its a trade off yes; take more time up front to ensure quality.  However, the cost of finding a bug down the road is much, much higher than the cost of testing and finding the bug up front.  This is always true, and its something many people forget.  It will cost you more (a lot more) to find the bug when its running on your customers computers than it will on your local desktop. 

The cost of creating the initial tests can be high; maintaining them is hopefully easier, as long as you stick to TDD.  That is, finding a bug requires you create a test to prove the test exists, then fixing it.  Changing functionality requires you to modify the test to the new expected behavior, then coding to 'fix' the BO.

Also remember that people seem to not write tests after the fact, which means your test library is small or non-existant.  If there's already a test setup and cleanup routine, it becomes easier to continue writing tests than if there were none to begin with.

Andy

Andy and David,

Yes we all agree on the trade-off / compromise point of view.  I guess now what I'm wondering is about the bang-for-the-buck of whatever you have defined as a "unit".

I like the idea of being able to easily and quickly check basic CRUD operations to verify the database schema, views, stored procedures, BO data access code, DAL etc. are all still in alignment.

My question is beyond that what's been particularly useful or not useful.  Do you find that only 2% of the bugs are data access related?  Is it still worth it since the CRUD tests are so easy to write?  Do you find with perhaps a few additional business logic tests per BO to test to the complex validation you catch another 60% of bugs?  And so on.

Also, David forgive me if it was covered earlier in the thread as I didn't read every word going all the way back, but could you give a little detail on what exactly you mean about having an abstract test base (or whatever the exact wording you used)?

I guess the bang-for-the-buck is what it's all about at the end of the day.  I think most of the "boilerplate" style code doesn't require test cases, because it can either be code generated or sensibly cut-and-pasted.  That stuff is easy to code and easy for someone else to pick up and understand.

It's the unusual and non-standard behaviours that warrant the effort of writing unit tests.  That's where you get the bang-for-the-buck payback.  That's where you need some test cases to provide the "guidance" to the next developer that comes along who needs to change/extend the "complex" behaviour.  You definitely want test cases in place then to make sure that the existing behaviour is not broken.

For what it's worth I think the "bug" we've fallen over most frequently is BOs not saving when you call the Save() method.  And every time this comes back to the property setter in the BO not making the object dirty.  A simple oversight in terms of the setter code, but it has the unexpected effect of not putting your data into the DB.

And there must be an easy way to create a test harness for this (using reflection?), so you don't have to code it up for every different BO. But we haven't gone down that route yet.

With regards to the abstract test base class, see the comments I made on post 1152 where I explained the principle of what was happening.  In essence though, a test class/framework can be just like any other class/framework - you can use inheritence, generics, whatever, to get the job done in the most practical way possible.

So, the principle behind the abstract test base class was a class that could be inherited from that provided the basic framework for doing the CRUD testing (it's analoguous in some ways to the BusinessBase base class).  It has abstract methods that each derived class must implement to provide the BO specific stuff - like the setting of properties.

So a new test class for a new BO means you inherit from the abstract base and implement the methods that require BO specific behaviour.  And hey presto - you have simple CRUD testing.