"My name is Andy Leonard. I tweet."
"Hi Andy, we love you."
It's true, I am a bona fide Twitter-holic. If you use the service you may have noticed disruptions lately. Frank La Vigne (recently married - congratulations Frank!) blogged about it. The Twitter developers are blogging about it. And they have raised the ire of Mr. Scoble, who posted his feelings on the matter at FriendFeed - a competing service (if "competing" applies here...).
From the Twitter Dev blog:
The events that hit our system the hardest are generally when "popular" users - that is, users with large numbers of followers and people they're following - perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that's behavior we have to limit on our side.
The Twitter Dev comments have been interpreted by some as complaining about user load. This is not good for Twitter and they should take immediate steps to manage the buzz before this goes any farther.
I jokingly tell clients sometimes the problem with their application or database performance is a combination of their data and their clients. I only do this in person and after reaching a comfortable comfort level with the client, and even then it's sarcasm that I carefully throw out. Why? It's the technological equivalent of saying the word "bomb" while waiting in the security line at the airport - within earshot of the good people of the TSA.
In other words, it's a really dumb thing to say.
Let's look at why: First, Twitter is demonstrating that they made a mistake in architecture. I don't know what that mistake is, but it's obvious to everyone using or attempting to use the platform that an error has occurred. It's not the HTTP 500 error I saw last night or the on-again-off-again link to older tweets. It goes beyond that back to the design.
Part of the problem can be stated thus: "Twitter did not know we were going to grow so fast." Another way to say that is "Twitter doesn't scale."
As a database professional in the data warehousing field, I feel Twitter's pain.
We are often pressured to "just make it go!" - deliver something now and fix it later. And oh the temptation is strong, the logic sounds sound, the song so sweet... "you can circle around later and fix it" they say. When you hear these words and are so tempted you can be certain of one and only one thing: the person saying this to you is lying. Malice may or may not be present - they may simply be repeating what they heard or they may be utterly ignorant, but they speak not the truth.
Second - although I could be wrong about this - I would wager good money that Twitter doesn't need a DBA. I've seen / heard / experienced this before. They have a talented team of application developers who have built large applications in the past and never once paid a database professional. Look at the money they've saved! </sarcasm>
Why does this happen? Experienced database professionals slow down project development. We get in the way. We muck about with stress tests and bulky architectures and referential integrity and schemas and the like. Who needs us? We're an unnecessary expense.
Or are we?
From the quote above, the bottleneck is occuring in the database. It's those pesky queries. And the users causing them to be executed. It's <insert-your-favorite-excuse-here> - everyone and everything, except the designers and architects who built a non-scaling solution.
My lovely bride Christy has this saying: "Good judgment comes from experience, and experience comes from bad judgment." My hope is Twitter will discontinue offering excuses and blaming users, and instead fix the (apparently database-scaling-related) problem.
If this comes across as harsh, that is not my intention. I have been there and done that myself. These are learning experiences and growth opportunities. Hopefully Twitter will enjoy the fruits of this learning and growth in the future.
For now, I've created a FriendFeed account. Let's see how they scale...