Numbered test file abuse

schwern on 2009-12-29T01:43:23

I hate numbered test files. Not that it isn't useful to force the ordering of some tests, but because its not then necessary to force the ordering of EVERY test. At the worst case you're back to BASIC. Observe the test suite from SQL::Statement.

00error.t
01prepare.t
02executeDirect.t
03executeDBD.t
04names.t
05create.t
06group.t
07case.t
08join.t
09ops.t
10limit.t
11functions.t
12eval.t
13call.t
14allcols.t
15naturaljoins.t
16morejoins.t
17quoting.t
18bigjoin.t
19idents.t
20pod.t
21pod_coverage.t

Now, how much of that is really trying to order the tests and how much of it is just the order it happened to be written? Does the limit test really have to go after the functions test? Why is there a quoting test in the middle of three join tests? I see three that make any kind of sense, 00error.t (since most of the tests use RaiseError), 20pod.t and 21pod_coverage.t, which maybe go last, though its honestly not important that they do.

What's the harm? It cripples command line completion. And you've got the old BASIC problem of renumbering. I want to add a new test, where does it go? Do I have to puzzle out the implied dependencies? Do I stick it at the end? But then the POD tests aren't last any more. Do I renumber everything? Do I use a duplicate number? Do I say the hell with it and cram it into an existing test file?

Not worth it.

Realistically most test dependencies really want to express two things: Run this first and run this last. For that you have 00foo.t and zz-bar.t. 00compile.t, 00setup.t, zz-teardown.t, zz-pod.t, etc... Anything else, just write it without the number. Or if you really do have a fixed order use Test::Manifest.

If you do have tests that would do better running in order, then instead of smashing them together into one arbitrary numbering system, group them. That is, stick them into a common subdirectory and then apply the first/last numbering again. A clear candidate in SQL-Statement would be t/join to contain naturaljoins.t, morejoins.t and bigjoin.t. This has the advantage of easily letting you run all one group's tests in one shot. "prove -lr t/join".

I don't mean to pick on SQL::Statement, its just what I'm patching right now. Lots of distributions obsessively number their test files to no real advantage.

---

Test File Names

Ovid on 2009-12-29T07:53:52

What I usually recommend people to do, particularly if the test suite gets large, is to name tests after their packages. For SQL::Statement, you would get something like this:

t/sql/dialects/ansi.t
t/sql/dialects/anydata.t
t/sql/dialects/csv.t
t/sql/eva l.t
t/sql/parser.t
t/sql/statement.t
t/sql/statement/function.t
t/sql/statem ent/functions.t
t/sql/statement/getinfo.t
t/sql/statement/operation.t
t/sql/s tatement/placeholder.t
t/sql/statement/ram.t
t/sql/statement/term.t
t/sql/sta tement/termfactory.t
t/sql/statement/util.t

If you want something more detailed because your packages are too large you can break down those final .t tests into finer-grained tests.

t/sql/statement/function-equal.t
t/sql/statement/function-noequal.t
t/sql/stat ement/function-lower.t

By switching to this system, it makes it ridiculously easy to figure out where every test goes (not true for many large test suites) and it also makes it easy to write editor tools to jump between tests and packages.

And cross-post this to blogs.perl.org! It's gotten to be fairly stable and more people are switching :)

Er.

masak on 2009-12-29T12:02:20

Wouldn't that make a test file starting with an alphabetic character run after the 99-prefixed files?

Re:Er.

schwern on 2009-12-29T13:35:13

Oh yeah, I knew there was a reason I was using zz as a prefix.

I group the numbers...

DiamondInTheRough on 2010-01-09T05:42:17

I don't necessarily use consecutive numbers, instead, I use three-digit numbers, with the first digit indicating a "category".

This is from one distribution of mine.

001_compile.t
100_machine.t
102_misc.t
103_wix_component.t
104_wix_fragment.t
111_environment.t
112_files_entry.t
113_files_component.t
114_files_directoryref.t
115_startmenu.t
116_registry.t
117_directorytree.t
118_icons.t
119_createfolder.t
120_feature.t
121_files.t
122_removefolder.t
500_new.t
800_perlcritic.t
801_pod.t
802_pod_coverage.t
803_minimumversion.t
804_manifest.t
805_meta.t
806_portability.t
807_version.t
899_prereq.t
901_perl_589.t
902_perl_5100.t
903_perl_5101.t
904_portable.t


In this case, the 0xx tests are for compilation only, the 1xx tests do small portions of the distribution, the 5xx is an "overall" test, 8xx are author tests, and 9xx are long-running (we're talking hours) release tests that are guarded with an environment variable.

To each his own, however.

Re:I group the numbers...

schwern on 2010-01-09T21:10:15

This is why God invented directories.