Benchmarks suck, but so do people who believe them

brian_d_foy on 2006-05-26T00:02:07

Chris alerted me to a wacko going around Wikipedia misinterpreting a single slide from my Uniforum benchmarking talk (PDF). This is the same wacko that wanted everyone on Perlmonks to boycott O'Reilly, apparently. When enough people in Perlmonks told him what he could do to himself, he moved over to trashing Perl (and here too after being slapped around in the Perl article) on Wikipedia. Chris keeps cleaning up the mess. I think this guy has infected a lot of other places, but Chris says he keeps cleaning it up, so I don't go looking for it.

In one slide from the talk, I say "Benchmark.pm comes with Perl..." and in the next, I say "..and it sux". I used the deliberate mispelling and a smaller font-size analogy to kwalitee to point out that I don't mean that the module is bad, but that a lot of people mis-interpret benchmarks because they trust the numbers too much and they let the computers do the thinking for them. Later I show an example from an earlier Stonehenge class where we constructed a benchmark that didn't show the Schwarzian Transform as good as it really is. Simply quoting me saying something "sux" because I said it as a throw away joke in a talk doesn't really say anything about anything, and certainly is far from quoteworthy in an encyclopedia.

I had written about bad benchmarks earlier in "Wasting time thinking about wasted time" on Perlmonks and also in the chapter-in-progress in Mastering Perl. Those explanations aren't as sexy as using "sux" though, and the reasonableness of the much longer discussions aren't as useful for misrepresentation and shock value. It's much harder to characterize me as anti-Perl when you read the whole thing, especially when I'm pointing out the flaws in things I've done myself.

So, to perfectly illustrate my point that you can't trust people with statistics, this wacko is using my comment that people mis-interpret and mis-represent benchmarks to support some vague notion that Perl is just bad. It doesn't have any relevance to anything to do with Perl as a language, really, and certainly doesn't support the anti-Perl stuff.

The curious thing, however, is that Wikipedia even tolerates this. Several other people keep reverting this guy's edits, and he keeps putting them back. The "play nice" attitude of Wikipedia ignores thousands of years of history that show we can't do that. Just because something is on the internet doesn't mean people are going to change. The LART and the clue stick were invented for a reason.

It's all part and parcel of attaching my name to something though. I get to be the target of militants and fanatics who have nothing better to do then think about something they hate.

This doesn't mean I'm going to stop doing anything. Once wacko is hardly worth losing sleep over.


Benchmarks

Wassercrats on 2006-05-26T02:09:48

So, I guess you think it's wacky for me to be against a publisher who post-copyrights technical books, which people tend to want to be up-to-date, and a publisher of a book that "Rather than simply describing the vulnerabilities and their exploits theoretically or showing you how to use pre-existing tools to exploit the vulnerabilities...provides the nuts & bolts you need to learn how to program your own exploit code." At least you linked to my post, but I'm wondering exactly what you disagree with about it.

As for Chris cleaning up my mess, he's not. He's been removing good content, and the matter is in arbitration. We'll see how it turns out. Chris is a blatant Wikipedia vandal, even with the Perl article, and an administrator had to warn him twice.

You shouldn't have said Benchmark.pm "sux" if it doesn't, but if someone would have just added more detail from the slides to the Wikipedia article, that would have been fine with me. It's hard to be sure whether other slides are meant to support the sux statement or define it, so I just quoted the sux statement and some other slide without assuming anything. Instead of adding details, people just removed the quote, citing bogus reasons like it's not relevant and telling me not to post my point of view, as though it wasn't an exact quote from YOU. I believe there's a Wikipedia guideline about not removing content when you can fix it. They eventually removed the entire benchmark table I created.

Chris alerted you to me? If you're really not Wikipedia's Scarpia, it would help if you say so on the talk page of the Perl article, or my talk page, or the mediation page. Scarpia knows about the accusations and isn't denying it. He's even added information on Wikipedia pages regarding your personal life.

Copyright Dates in Books

chromatic on 2006-05-26T06:17:50

I guess you think it's wacky for me to be against a publisher who post-copyrights technical books...

As explained before, this is what book publishers do with the permission of the Library of Congress so as to receive copyright the entire length of the copyright period, and not just part of the last year.

(Of course, O'Reilly books enter the public domain long before the copyright period expires.)

If the copyright year of 2006 for a book that goes to the printer in December 2005 really causes you that much distress, look at the latest date in the Printing History section on the copyright page.

Re:Copyright Dates in Books

Wassercrats on 2006-05-26T09:40:26

There was a greater difference between the date I bought the book and the printed copyright date than that. Lets try to not be misleading with our examples. I don't know when the book first went on sale, but I bought the book in November and the copyright was listed as the next year. See my original post about it here, where I also quote copyright law.

If there's some aspect of the law that makes printing a later copyright legal, and if it's done on all major books when applicable, then I'd withdraw my complaint, but at this point I'm not interested in looking into it more deeply than I have. I haven't even reread the related Perl monks threads.

Re:Copyright Dates in Books

speters on 2006-05-26T11:42:11

It is common practice for publishers of technical books to postdate the copyright date. For example, the Hibernate Quickly sitting on my desk, has a copyright of 2006, but was released August 2005. The date is simply the date the copyright is registered, which you are not required to do. Copyright protection actually begins when you start writing your book, article, program, etc.

Re:Copyright Dates in Books

n1vux on 2006-05-26T15:43:43

It's not just technical books.

But August seems a bit early by the standards I got used to when I was in the used book trade part time.

Rule of thumb in publishing used to be a book released in the last 3 months of the year carried next year as copyright date (he says working from fuzzy memory). This is not unlike Detroit model years.

 

Re:Copyright Dates in Books

chromatic on 2006-05-26T17:06:34

... at this point I'm not interested in looking into it more deeply than I have.

There's a word for people who refuse to change their opinions and continue to complain anyway after being corrected.

Re:Copyright Dates in Books

Wassercrats on 2006-05-26T23:54:51

To me, the hacking book is enough to justify a boycott. As for the copyright issue, I believe I'm the only one who quoted copyright law, and I haven't seen enough information on how common it is to post date books to let O'Reilly off the hook for what seems like deceptive business practices and a violation of copyright law. But at the same time, I'm not going to say you're wrong if you think it's legal and common.

Re:Copyright Dates in Books

pudge on 2006-06-07T23:16:00

To me, you're an idiot no one cares about.

Re:Copyright Dates in Books

hex on 2006-06-09T23:48:05

+1, Insightful

Re:Copyright Dates in Books

petdance on 2006-06-08T00:16:08

To me, the hacking book is enough to justify a boycott.

Don't let us stop you! Carry on! Stick it to the man!

Re:Benchmarks

Aristotle on 2006-05-26T06:19:49

You shouldn’t have said Benchmark.pm “sux” if it doesn’t,

Apparently you don’t understand the concept of irony.

Re:Benchmarks

sigzero on 2006-05-26T12:26:23

"It's hard to be sure whether other slides are meant to support the sux statement or define it, so I just quoted the sux statement and some other slide without assuming anything."

Ha!

Re:Benchmarks

runrig on 2006-05-26T16:54:25

You shouldn't have said Benchmark.pm "sux" if it doesn't...

It's obvious (to me, anyway) from the slide that he's implying that saying "Benchmark sux" means that it actually sucks in the same way that saying CPANTS "measures kwalitee" means that it actually measures quality. So just saying that "Benchmark sux" is taking the slide out of context.

Re:Benchmarks

Wassercrats on 2006-05-27T00:00:18

I saw that kwalitee analogy in the slide, but...well, if I could find a link to get to the top post I would look at the slide again, but this forum software sucks. I though the kwalitee thing was just explaining that quality is to kwalitee as sucks is to sux.

“How to write your own exploit code”

Aristotle on 2006-05-27T21:47:32

Mmmm, I missed that:

“Rather than simply describing the vulnerabilities and their exploits theoretically or showing you how to use pre-existing tools to exploit the vulnerabilities…provides the nuts & bolts you need to learn how to program your own exploit code.”

I have explained to a number of people how to conduct SQL injection attacks. Will you boycott me now?

Re:“How to write your own exploit code&#8221

Wassercrats on 2006-05-28T05:37:09

It's not as bad if you're selective about who your teaching, and it depends on why you're teaching them, but yes, I'll boycott you too. Not sure what you sell though.

Re:“How to write your own exploit code&#

petdance on 2006-06-08T00:19:01

I would also like to be boycotted.

I have written an article, in The Perl Journal or maybe The Perl Review, one of the two, about how to do SQL injection attacks AND hack poorly written CGI scripts, on the way to explaining why taint checking is a good thing.

What do I need to do to be boycotted? Is my bringing it to your attention here sufficient? Is there some form I need to fill out?

Please reply at your earliest convenience.

Re:Benchmarks

ask on 2006-05-30T09:58:28

It's hard to be sure whether other slides are meant to support the sux statement or define it, so I just quoted the sux statement and some other slide without assuming anything.

Why in the world would you quote something you clearly don't understand?

"I saw it on the internet so it must be true!!"

  - ask

Re:Benchmarks

Wassercrats on 2006-05-30T12:37:02

Because it was written by a notable author and I was quoting it, not interpreting it in a way I was unsure of. It was written as though "sux" was the bottom line, and it would have been ok to quote it even if I quoted nothing else. I provided a link to the source. Other Wikipedians could have given the module the benefit of the doubt and added more details if they thought it would clarify things. Quoting "sux" was appropriate whether there's total suckiness or not, if that's what a notable author and editor of the Perl Review said.

Speaking of suckiness, ever since I said I'm willing to help test Slashdot's New Discussion System, I'm seeing this entire threaded discussion on one page. If it were in order of date, that would be fine (even preferable) because I'd just look at the bottom for new posts, but since it's threaded, I need an index of posts, which isn't there anymore.

Re:Benchmarks

pudge on 2006-06-07T23:17:16

Speaking of suckiness, there's ... you.

Your opinion is ignored, since I'm the one writing the code, and I hate you.