Digg.com is an ideal target for creating instant gratification content. You can come out of nowhere and have tens of thousands of new users in 24 hours. An inevitable woe is the Digg Effect — a sign of success and often the key to server failure.
How do you make 10,000 comparisons of Digg users to the rest of the community?
How do you display 600,000 links to relevant Digg stories in thousands of users’ browsers?
How do you tell 40,000 users where they are located on a map of 67,000 users?
How do you display 15,000,000 Diggs on that map?
How do you do it all on one shared hosted server?
Birth of a Digg Tinkerer
In December, I ran an experiment that required collecting a lot of data on thousands of users. Other parts of the experiment included measuring traffic and Google Adsense revenue during the Digg Effect. I created a little gadget, called “DiggStatus“, where Digg users could type in their user names to see how their Digg usage statistics stack up against the rest of the community. The experiment was much more successful than I had anticipated.
Weekend Projects
Every once in a while, it’s nice to get away from clients, meetings, deadlines, revisions, and product launches. Build something you want to build. Make your own deadlines. Change your mind — no deadlines. Finish it when and if you feel like it.
During the last few months, I have created a small collection of projects geared toward Digg users. Each is intended to be in itself a one-hit wonder. Get to the front page, get viewed by thousands, then recede into oblivion.
The Inevitable
Creating content for Digg is all or nothing. Either it hits the front page and you get blasted with thousands of visitors in a short period of time, or it passes through Digg’s upcoming stories section without a hint of interest.
Creating tools and gadgetry for the Digg front page puts a developer with limited resources in a sticky situation. You have to last through the Digg Effect, or nobody will see it.
Looking into a Broken Mirror
Digg users are accustomed to using mirror sites to view content after the target web server crumbles under the load. Unfortunately, mirrors are only useful for static content. Mirrors can’t learn your application’s recommendation algorithm. Mirrors don’t know the coordinates of users on the map you just made.
As a tool creator, it is imperative to withstand the Digg Effect.
Every K Counts
Look at the amount of data your web server will have to send for every visitor.
If your web site has a layout that utilizes images, jump ship. Set up your to-be-Dugg page outside of your layout. Add color with HTML. Structure your page with very light-weight HTML. A logo at the top of the page can be all it takes to push your web server over the edge. With links back to your site, only interested traffic will be directed to your normal site.
If you require images, find somewhere else to put it, so the bandwidth can be diffused between multiple locations.
Keep Quiet
With thousands of Digg users slamming your server in a very short period of time, even the simplest database call can be destructive to your server. If you use a framework for your site, make sure you have caching set up so you don’t ping the database for the same data thousands of times just to display the page.
In the tool itself, you need to optimize, optimize, optimize! Database queries add up faster than you may think.
Put Your Money Where Your Mouth Is
This week, I released two tools that utilize a map I generated last weekend of the Digg community.
The first tool allows visitors to search for a user name to find where that user is located on the map. This one required 3 database queries per request — one to find the location of the user, one to count the user’s friends, and one to count the user’s fans. While it was on Digg’s front page, about 40,000 users were requested. The database was therefore queried about 100,000 times. All of this and I didn’t receive any mention of downtime from users. If I had used one more database call, and taken the total up to 130,000 queries, the server might not have been able to handle the load.
The second tool I released this week displays Diggs (votes) on the map in real-time (with a small delay) and placed at the Digging users’ location on the map. In 24 hours, the tool has displayed 15,000,000 of those votes. Unfortunately, there were a few hours that users had intermittent access to my site. Contributing to the overload, the first tool was still getting quite a few hits from Digg, it was featured on the BBspot.com home page as a Daily Link, and there were quite a few StumbleUpon users.
However, despite a little bit of downtime, this one did quite well, thanks to some heavy optimization. It may appear like there are tons of queries going on, with all those icons popping up on thousands of screens around the world. The search functionality in the other tool required 3 queries per request. This tool only required one query per 100 Diggs (though, the tool may at times display over 1,000 Diggs per minute). The results were returned to the client in a compact comma-delimited list, where the Flash front-end would then break the results apart.
Moral of the Story
If you create content for Digg, take a close look at the your page and/or tool’s weight on the server. You need to save every byte you can. You should only contact the database when absolutely necessary.
The Digg Effect should be a sign of victory not a sign of defeat.