One of the tricks that people use to scale websites is memcached: "a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load". Memory is cheap so you might as well cache things in memory on lots of servers in your datacenter. On the memcached mailing list under a thead entitled "Largest production memcached install?" some interesting figures popped out: Bloglines has memcached on a couple hundred machines and Facebook has roughly 200 dedicated memcached servers in its production environment... all 16GB 4-core AMD64 boxes. That's around 3 terabytes of memory cache!
