Ovid doesn't always use TDD, even for production-style programming. (That's non-exploratory programming.) He lamented that:
One problem I have with the testing world is that many "best practices" are backed up with anecdotes....
This will continue to be true until it's possible to measure the cost of creating and maintaining a piece of software throughout its lifecycle. If TDD costs me 25% of my initial development time but saves multiple day-long debugging sessions over five years, I'll stick with TDD. (I can give you plenty of anecdotes where the lack of tests cost lots of debugging time, including one amusing-only-in-retrospect semi-predicate problem where I reverted a checkin from another developer who reverted my checkin until we both realized that we had to account for all three termination conditions, and he was fixing one bug and I another.)
Of course, those studies still won't be valid until there's a way to get repeatable and condition isolated results out of the studies, but that requires turning programming into a mechanical practice devoid of all creativity and repeatable ad infinitum and unmodified in laboratory conditions.
I can only give you the best advice I have. In the past decade, I've become a much better programmer in part due to learning how to use TDD effectively. (I've also had a decade more practice, though.)
Frankly, I've no problem with you using Anecdote Driven Development. I have every problem with testing advocates tell me I should be relying on their anecdotes.
This will continue to be true until it's possible to measure the cost of creating and maintaining a piece of software throughout its lifecycle.
What you're actually saying here is "it's very, very hard to accurately measure this cost, so
So for the time being, you be "Test First", I'll be "Test Comfortably" (since I'm a polyglot mix of testing strategies), and we'll just agree that it's not right to allow our anecdotes to drive other people's behavior, eh?
(meanwhile, I'll keep reading those studies because while they may not be perfect, there's interesting information there).
Re:To TDD or Not To TDD
chromatic on 2009-03-10T19:08:29
Studies in a variety of fields require acknowledging that things can't be done perfectly, but we can do enough to get a sense of how things actually work to make sane recommendations.
You might as well quote a study that says "Berlitz German students translate the libretto of The Magic Flute twice as fast with 80% fewer errors than students of other language learning schools." ETOOMANYUNCONTROLLEDVARIABLES.
Re:To TDD or Not To TDD
Ovid on 2009-03-11T08:19:44
I can't tell what you mean due to your sarcasm. I know you're not saying "the benefits of TDD are too hard to study, so we won't even try" because that would be an idiotic statement to make and you're not an idiot. So what are you trying to say? Are you saying that the efficacy of TDD is beyond reason and we must rely on anecdote? Again, I know you're not saying this, but you're not saying much else, either
:) In short, if you're going to make cause and effect assertions, I want to hear something backing that up other than anecdotes.
Re:To TDD or Not To TDD
chromatic on 2009-03-11T08:38:11
In short, if you're going to make cause and effect assertions, I want to hear something backing that up other than anecdotes.
I believe you're in the wrong line of work then. How would you even design such a study?
If you compare two programmers, one using TDD and one not, you have to account for productivity, knowledge, experience, and creative differences between them.
If you compare two teams, one using TDD and one not, you have to account for the same, multiplied by at least the number of programmers on each team, multiplied by some factor to account for the management of each team, multiplied by some factor to account for personality and personnel issues.
If you compare a single programmer's or team's work on the same project with and without TDD, you have to account for the second system syndrome and all of the knowledge and experience gained from working on the project before.
If you compare work on different projects, you have to account for the fact that they're different projects.
If you want the study to represent the real world in any interesting fashion, you have to account for ongoing the maintenance costs of a real application, not a green field project discarded at the end of a semester.
I can point you to teams who can answer the questions "Has our defect rate gone down over time?", "Has our velocity improved over time?", and "Has our customer's satisfaction increased over time?", but I don't know how to measure that objectively, and especially not in a fashion that allows unbiased and empirically valid comparisons between teams. I believe that's impossible, or at least so difficult that it's prohibitively expensive along multiple axes of time and cost.
I can't prove that TDD or XP or pair programming or frequent iterations are the best way to develop all software everywhere. I'll never claim that, and I share your distrust of anyone who does. I can only give you my best advice, based on my experiences and my observations of other teams. These are the most effective ways I've seen to develop and maintain software in general. I can give more specific advice in specific situations -- but I'm not going to let epistemological limitations deter me from giving people the best advice I have.
Re:To TDD or Not To TDD
Ovid on 2009-03-11T08:52:26
So I can now write a new blog post entitled "chromatic admits there's no evidence for testing's effectiveness"
:) Re:To TDD or Not To TDD
chromatic on 2009-03-11T10:53:15
If you want people to think you have the reading comprehension of the average programming.reddit.com commenter these days, feel free. (I know you're teasing -- your epistemology isn't broken -- but subtlety is deader on the Internet than in modern literature.)
Re:To TDD or Not To TDD
chromatic on 2009-03-11T20:24:36
For everyone else following at home, my response was teasing too. Ovid's in no danger of misunderstanding what I wrote.
Re:To TDD or Not To TDD
Dominus on 2009-03-11T18:00:01
If you compare two programmers, one using TDD and one not, you have to account for productivity, knowledge, experience, and creative differences between them.
Sociologists deal with that sort of thing all the time. Their methods aren't perfect, but they do manage to get somewhere.
The secret is that you don't compare two programmers; you compare two thousand.
Re:To TDD or Not To TDD
doom on 2009-03-11T18:28:42
I believe you're in the wrong line of work then. How would you even design such a study?
What I would do, is I would consult some social scientists. We are in the wrong line of work to be ruling out the possibility of doing an intelligent study of programming methodologies.
