We noticed at Yahoo! that we were started to get lots of timeouts from Facebook's APIs that we use on the Yahoo! homepage.

I've not heard anything official from Facebook yet, so this next part is all speculation.

I suspect that as a user updates their privacy settings, as forced to the first time they hit facebook.com now, Facebook flags that users account is dirty (something changed) and purges the cache of the users stream data.
Now that the cache is empty for a particular user, when a friend of that user views Facebook, the backend servers must go back to the master data source and confirm permissions on what is visible before it is displayed.

That is fine when you have a few users updating their permissions, if however you force ALL your users to do this, you site rapidly becomes overwhelmed, as you effectively removed your cache. This effect is known as "The Thundering Herd".

As the load increases, due to all extra requests for data that would normally be handled by caches, you start to get race conditions as requests return and try to populate the cache. Also as the load increases, requests start to take longer as connections are kept open while backends are queried. Because connections are being kept open, the server stops handling requests, as there are no more connection slots and you start to get errors as clients can no longer connect.

How do you prevent this. Don't push changes that require you whole userbase to update their records at once. Stage the release, starting with 1% until you can guage how much load it will generate on your servers. Then you can safely ramp up your change safely.

Here is an uploaded copy of my draft presentation:

This talk will be an introduction to hardware hacking, what it means, how to get started, tools you need, interesting blogs and where to find inspiration.

The talk will cover using the Arduino plus some other interesting topics.

For those people that are interested, we will be also try and setup a London Hardware Hackers group, to meet, share ideas, hardware, techniques and generally drink beer.

So we had loads of people drop by our stand. Some of the just along for the schwag, but most did ask about Yahoo!s Open API offerings.

We seemed to have a substantial amount of people asking about our Geo APIs such as Placemaker, FireEagle and GeoCoding tools. Partly driven by Gary Gale's series of Geo talks, luckily we had most, if not all, of Yahoo!s London Geo team manning the stand as well.

Our other APIs did also received significant love. Although I must admit that between gestating babies and developing for the Yahoo! homepage, I have not really put the necessary time towards playing with YQL to be able to talk about it as authoritatively as a such. Right now I could talk the legs off Facebook's APIs and FQL, but they don't pay me

So today still no baby, 9 days overdue and still counting. So I'm back at FoWA again today, as Monday the fun begins whether we want it to or not!

Last week we turned on the opt in to the Yahoo! Homepage http://uk.yahoo.com/trynew

My personal project was writing the Facebook App, which does seem to have received a fair amount of press coverage.

However for the last 18 months we've been working on a complete rewrite of the homepage.

This has included releasing four countries (France, United Kingdom, United States and India) all using the same codebase and the same developers.

Personally I've been involved in setting up testing frameworks, continuous integration and code quality.

I'll try and publish further updates later. Specifically, if possible, information on the Facebook App.

This is a brief reminder of the series and a list of all the articles.

These articles are the originals and are being used as a base for a YDN article to be published shortly.

1. Introduction to CodeSniffer
2. CodeSniffer Output
3. Writing an example CodeSniffer Standard
4. How does CodeSniffer work?
5. Writing an example CodeSniffer Sniff