Does anybody know of software that could be used to build a mirroring reverse proxy? I am looking for something to cache and mirror files for distributions and other archives of mostly static files.
To clients, it looks like a copy of the archive. If a file does not exist, or is too old, it is fetched from the master servers and saved to disk. The files can be populated from CDs or mirrored through rsync to get complete copies. Ideally, it would load balance from multiple mirror servers.
It is possible to set something up with Apache2, mod_proxy, and mod_disk_cache but that does not allow load balancing between servers and stores files in an opaque cache.
Would Perlbal be usable for this? Could it be extended? I think a mod_perl module could be written.
Re:Apache+squid
iburrell on 2007-06-04T01:22:28
Can Squid store the files on disk in a structure that mirrors the master servers? Part of what I am looking for is a way to preload the cache with files from the CD or DVD. I also want to be able to sync parts of the mirror with rsync or wget.Re:Apache+squid
Juerd on 2007-06-04T08:18:26
Why would you worry about the storage structure? As long as everything appears to be the same through http, does the physical layout on disk matter?
No, Squid won't use a similar structure, but you can preload anyway, by just requesting the relevant URIs.
If you're not loading from a normal internet site, though, you may have to set up a tiny http server for this purpose yourself. Or perhaps another proxy:) Re:Apache+squid
Aristotle on 2007-06-04T10:30:32
Not really. Squid’s directory structure doesn’t look like what it has cached, but it’s pretty easy to hack some Perl to bulk-feed a directory structure plus given base URI into the Squid cache dirs.
