The Google Everflux Phenomena

I tend to spend a lot of time in various search engine forums and
newsgroups answering questions related to search engines. One of
the most common questions that seems to come up is along the
lines of:

"My site was in Google yesterday and ranking well and today it's
gone! What happened?"

or

"I made some changes to my site and Google picked them up, but
now, it shows the old page again. Why would Google do this?"

and a number of other variations on the above two questions.

With the importance of Google these days, it's no wonder that
situations like the above would have webmasters quite worried.
After all, with all the reports of sites being penalized or
banned, having your site completely disappear could be a bit
troublesome.

Luckily, there is a very easy explanation for the above phenomena
and it's been lovingly referred to as "Everflux". What exactly is
everflux?

Well, "everflux" stems from Google's attempt to create the
freshest possible index and by fresh I mean up-to-date. To
understand this, let's look first at Google's normal update
cycle.

Generally, somewhere around the beginning of the month (all
though this can vary widely such as in the past couple of months)
Google's primary spider (actually there are many more than one
primary spider, but for simplicity I'm going with the singular)
heads out and begins to index the sites in it's database. This
process generally takes anywhere from 5 to 10 days. During this
time, the spider indexes any new pages and re-indexes pages
already in it's index.

After this spidering process occurs, there is generally about a
two to three week delay before the results from this spidering
are publicly available. During this period, which has
affectionately been termed the "Google Dance" the results
returned from Google tend to fluctuate a bit. This "dance" can
last anywhere from 2 or 3 days up to about 1 week.

This is the normal cycle for Google and it does quite well except
for sites where the content changes frequently such as news sites
etc. This is because, with the current system, there can be
anywhere from a 2 or 3 week minimum delay for changes to a
webpage or site to be reflected in the primary database and up to
6 or even 7 weeks depending on when the changes were made to a
site. If changes were made in time for the monthly spidering,
those changes would be reflected in a couple of weeks, but if the
changes were made after the monthly spidering, then the site
would have to wait for the following months spidering to be
picked up and it would end up taking much longer.

Even a two or three week delay is too long when dealing with
breaking news and other current events. The solution? Google's
"Freshbot".

Google's "Freshbot" as it has been termed is a secondary spider
that is constantly crawling the web. It crawls sites Google has
found to be either news sites or other important sites that
change on a constant basis. It also tends to find sites that have
either recently changed or are brand new.

This secondary spider adds it's findings not to the main database
but to a temporary database. This temporary database is
incorporated into the results returned from the primary (main)
database which allows Google to continue its normal update cycle
but also return very fresh and up-to-date content.

The confusion comes from the fact that this temporary database
that is used by the Freshbot is, in effect, rewritten on a daily
basis with the results from the latest round of spidering. This
means that a page that was in the temporary database on one day
may be completely missing the next.

This can cause a lot of confusion as a new site could be found
one day by the Freshbot and added to the temporary database only
to be overwritten and disappear the following. The same goes for
changes to a page that are found by the Freshbot and then revert
to the old version within a day or two. This is simply the
natural "flux" caused by this temporary database.

The good news is that these sites that are found and then
disappear will almost always reappear permanently once the
primary spider crawls them and they are added to the main index.

So, if this has happened, is happening, or does happen to you at
some point, never fear, it is simply the Google "Everflux"
phenomena at work.

More about google and SEO