Reading this article about myspace.com and their technology lead me to some interesting tidbits.
"Chau developed the initial version of the MySpace Web site in Perl, running on the Apache Web server, with a MySQL database back end. That didn't make it past the test phase, however, because other Intermix developers had more experience with ColdFusion, the Web application environment originally developed by Allaire and now owned by Adobe. So, the production Web site went live on ColdFusion, running on Windows, and Microsoft SQL Server as the database."I think that explains why they had so many performance problems early on. This is one case where "go with what your teams knows" may have been bad advice. They eventually ditched it for C# and saw a big imrpovement.
Whenever a particular database was hit with a disproportionate load, for whatever reason, the cluster of disk storage devices in the SAN dedicated to that database would be overloaded. "We would have disks that could handle significantly more I/O, only they were attached to the wrong database," Benedetto says.They solved this by going to a storage technology that pooled their resources instead of partitioning them. I think this supports my theory that partitioning is usually a bad idea and you should share resources as much as possible. Partioning used to be a big sell for expensive EJB tools and IBM hardware, but the end result is that some of your hardware is under-utilized while parts of your application are starving for resources.
The cache is also a better place to store transitory data that doesn't need to be recorded in a database, such as temporary files created to track a particular user's session on the Web siteââ¬âa lesson that Benedetto admits he had to learn the hard way. "I'm a database and storage guy, so my answer tended to be, let's put everything in the database," he says, but putting inappropriate items such as session tracking data in the database only bogged down the Web site.Storing sessions in your lossy cache storage is a mistake, in my opinion. If your session suddenly dissapears for no reason when you're browsing myspace.com, this is why -- they put it in the same unreliable storage that they use for caching. But then he goes on to say that he really doesn't care if your data gets lost:
In other words, on MySpace the occasional glitch might mean the Web site loses track of someone's latest profile update, but it doesn't mean the site has lost track of that person's money. "That's one of the keys to the Web site's performance, knowing that we can accept some loss of data," Benedetto says. So, MySpace has configured SQL Server to extend the time between the "checkpoints" operations it uses to permanently record updates to disk storageââ¬âeven at the risk of losing anywhere between 2 minutes and 2 hours of dataââ¬âbecause this tweak makes the database run faster.Classic.
