Completely unscientific, of course, but here are numbers you might find interesting.
We have a set of tests which exercise our Catalyst, Moose and DBIx::Class code quite heavily. These would normally take 52 minutes to run.
Naturally, we don't want them to take that long, so we run them under Test::Aggregate. They take 11 minutes to run. The only significant difference that Test::Aggregate makes is that it loads all of your code at once, eliminating the overhead of restarting the Perl interpreter and reloading those modules for every test.
52 minutes to 11 minutes. That's a lot of overhead (and, as mentioned, completely unscientific and probably useless data).
Can you isolate how much is perl and how much is the modules?
-- dagolden