Once upon a time there was >500k lines of Perl code and >2.4k non-CPAN packages in a huge SVN repository. Sounds like fun? Yes, especially for a new-comer :-)
Now? Even more lines and even more packages in even huger SVN repository ;-) but also a tag and daily build system that uses CPAN::Mini::Inject to insert tarballs to local CPAN::Mini mirror to be installable via CPAN shell. In addition it's possible to browse them using CPAN::Mini::Webserver.
How? http://github.com/jozef/HTTP-DAV-Browse/ that allows to walk through SVN WebDAV (check examples/hdb-build-tarballs for complete script), http://github.com/jozef/Build-Daily/ to allow daily/svn_revision based versions and http://github.com/AndyA/CPAN--Mini--Inject/ that has now option to index *.pm files of tarballs (--discover-packages).
Future? Feeds for everyone about failed builds, smoketesting, automated Debian packages builds or ??? Let's see where the fantasy will take us...
For reporting the smoke tests you might want to look at Smolder (http://github.com/mpeters/smolder/tree/master). You basically just hand it TAP archive and it will generate a nice looking report and send emails and an Atom feed. Reports are per-project. This might get kind of tricky with so many modules, but if you just organized the modules by project (if that makes sense) and then each test submission was a different module then it would probably work (especially if you use some tags to help organize things).
You can check out the Parrot Project's Smolder reports at http://smolder.plusthree.com/app/public_projects/smoke_reports/8
