Earlier this year there was a massive change in technology.  This change affects billions of people today in subtle ways, but lays the groundwork for very profound developments.  Google has fundamentally altered the way it deals with it’s search index.

The details behind their move to “Caffeine” from “MapReduce” will be written in papers, spoken about at conferences and will stand as a significant milestone in the realm of big data.  However, most of the billions affected by the change though will hear nothing the change or of the technical details, nor would they care if they did.

What it means for the rest of us is instant information.  I know it already feels like we live in an instant information world, and to some degree we do.  RSS feeds, apps and widgets have provided a direct link to new information between end users and content creators.  What we haven’t had is instant discovery.

Using the absolutely massive computing resources available to Google and some of the best large-scale software ever produced, it still took Google from 8 hours to  two weeks (depending on the layer they were updating) to digest the planet’s web pages and produce a new index for us to search.  They decided long ago that this simply wasn’t going to be good enough.  An 8 hour to two week turnaround on the planet’s web presence sounds very impressive, and it is truly impressive, but it’s not instant.  Needing to wait as little as 8 hours before new web pages can be discovered sounds perfectly reasonable, but it’s not instant.  That’s what Caffeine is about.  It fundamentally changes how Google addresses its data in a manner that allows them to change and update small parts instantaneously rather than need to re-crunch an entire index.  Google can, and eventually will provide instant discovery for the world’s public web resources.

Like most major infrastructure changes, the effects will not be immediately visible.  The ultimate effect of the US Interstate Highways was not obvious in the 1950s, but looking back today, it’s clear that they fundamentally shaped the US into what it is today.  Caffeine will be the same.  It will have a small impact at first, allowing Google to provide slightly fresher search results (50% fresher by Google’s measure) and will no doubt result in a few other new features that they’ll roll out over the coming months.  But as time goes on, instant discovery will entrench itself into the way we work and play.  It will do so subtly, but soon we will remember the time before instant discovery the way we remember the time before e-mail on our phones or the time when we had no Internet altogether.

http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html
http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/