Managing Bandwidth (23rd April 2008)

People would search for car names and screenshots of mods on Project Cerbera would show up. They would click these, despite not really wanting to install a GTA mod.

Inbound traffic is dominated by image searches. Here’s what I got from AWstats today:

Top 10 Search Referrers, 2004–2008
SitePage Views
2008Google (Images)479,203
Google46,617
Yahoo!2,323
Windows Live1,608
MSN Search1,149
Unknown search engines614
AOL482
Google (cache)433
alice.it421
Ask78
2006Google (Images)703,812
Google30,715
MSN Search3,873
Yahoo!3,415
Unknown search engines926
AltaVista508
Baidu342
AOL269
AOL (de)187
Netscape124
2005Google (Images)420,138
Google28,168
Yahoo!1,656
MSN Search960
AOL657
Unknown search engines483
AltaVista482
AOL (de)265
Netscape180
Baidu170
2004Google (Images)75,589
Google4,368
Yahoo!345
AOL185
Unknown search engines159
Ask100
AOL (de)77
Virgilio67
MSN Search63
Netscape56

Today I got a 509 Bandwidth Exceeded message when trying to access the site. Checked my inbox and there was the e-mail confirming this. Chatted with Dave Bateman, my ever-friendly host. Decided to remove the site from Google Images.

Simple to do. Just added this to the bottom of my robots.txt file:

# Google Image Searches (2008-04-23)
User-agent: Googlebot-Image
Disallow: /

After a month or two we’ll see what effect this has.

Update (Monday 23rd June 2008)

When viewing my stats in June 2008, nothing had changed. So I moved that entry from the bottom of my file to be the 2nd entry; the first Disallow rule. The exact code:

# Google Image Searches (2008-04-23, moved up from bottom of file on 2008-06-23)
User-agent: Googlebot-Image
Disallow: /

Time to sit and wait again.

Update (Friday 21st November 2008)

It worked! Bandwidth use per month dropped from 27GB in June 2008 to 12GB in July 2008. It’s been falling by about 1GB per month since then, reaching 7.5GB in October 2008.

Coincidence? The referrer stats say otherwise:

Top 3 Referrers, 2008
SitePage Views
May 2008 Google (Images)186,808
(No Referer)27,725
Google23,141
June 2008 Google (Images)150,979
(No Referer)31,505
Google19,985
July 2008 Google (Images)21,817
Google16,972
(No Referer)16,033
October 2008 Google13,633
(No Referer)11,682
Google (Images)9,057

Now my site is using a much more reasonable amount of my host’s resources. Phew!