An automated way to beat the digg effect
As much as I am proud to say Libervis.com *finally* got it's first digg frontpaging I am ashamed to say that the server didn't took all that traffic well. Both Nuxified.org and Libervis.com are on the same server and both sites crashed, apparently due to too many mysql queries.
The only reason these sites are up right now is because someone took the story off the digg homepage prematurely, which obviously lessened the amount of traffic that was coming in and the sql server recovered.
Well, we can't let this sort of thing happen again. Be it from digg or some other site, such awesome traffic peaks are generally a good thing for our growth and if we just permanently remain vulnerable to them instead of benefiting from them that just wont do.
So on to the topic.. one idea I have about beating the digg effect is to do the following:
- Check whether traffic is coming from digg.com or not.
- Check the amount of traffic coming from digg.com and if it is higher than (insert the highest amount we can reasonably bear) then do the following (basically an if statement).
.nyud.net:8080to the host name of the URI which is receiving excess traffic (but not necessarily other url's) in which case the page is automatically mirrored and accessed in a Coral Content Distribution Network.
Actually the first step may not even be necessary. We can just check for traffic amounts, if that's possible, and redirect the suspected URI over CDN.
Also, once the traffic levels fall below the excess level the URI should be returned back to normal.
It may not be an ideal solution since the traffic going to coral isn't registered by our own server so we miss the actual traffic details, and the CDN isn't updating frequently enough to account for comments posted to a story, but if it would work it may be a surefire way to protect a site from crashing.
Alternative solution may be to automatically generate a static page when excess traffic is detected. Maybe I'm complicating things too much with the above and maybe that would be a better solution...
Anyway, would any of the above be possible in PHP, possibly aided by the .htaccess file? Also, if you've got other suggestions I'm all eyes and ears!