If you look at the source for this web page you will see the following:

<a href="http://youlookfab.com">this link to Angie's site</a>

Very simple, but this little fragment is the basis of the revolution that is the world wide web. Without it there would be no web sites, no web surfing, and no Google search.

There are essentially two benefits that Angie gets when I create a link like this to (youlookfab.com) YLF. First, the obvious one. People will click on the link and end up on her page. So I am sending her additional direct traffic.

The second benefit is more subtle, but also more powerful. By creating this link to Angie’s page I am telling Google that YLF is important. As a result Google may place YLF closer to the top of their search results, and then send Angie additional search traffic. This second bump in traffic has the potential to have a far greater impact. There are only a few people today on TheBlogEasy, but there are millions of people doing Google searches.

In order to explain the implications of this I will now make some gross oversimplifications about the way Google search works.

Google Search and Pagerank

Google has a little piece of software called the crawler. This crawler goes from page to page, following all the links and building a giant database that captures all the links on the web. Another piece of Google software then uses this giant database to answer your search queries when you type them in on the Google homepage. But how does Google decide which search result goes on top? That’s where the links come in!

Google looks at the number of links coming in to a page and uses this as a reflection of the page’s importance. This makes sense – popular pages will have many incoming links. Unpopular pages that no-one cares about? They won’t have any incoming links at all. So by linking to YLF on this page, I have sent a little signal to Google that YLF has some importance. Google uses the term “pagerank” to describe this importance measurement that is based on incoming links.

Paid Links: An Industry is Born

People are smart, and when they realized that Google was sending sites a lot of traffic they starting to think about ways that they could influence the Google search engine. They needed to convince it to put their site closer to the top of the search results so they could get more traffic. One obvious way to do this is to convince other sites to link to you so that the crawler would find these links, their pagerank would increase and Google would send them more search traffic.

This worked so well that people started to pay each other for these links.

And therein lies a problem. Google’s algorithm is relying on the fact that the links are natural in order to use them as an indicator of a site’s importance. When people start buying links, they are buying importance. Even if their site is horrible to look at and contains unreliable information, with enough money they can trick Google’s search engine into thinking that the page is important.

Then Google starts sending people to crappy sites and they lose confidence in the search engine. Not good.

Policing Natural Links and Punishing those who Distort Them

Needless to say, Google is very concerned about anything that distorts the natural order of things. This includes paid links and also link networks, where everyone in a community links to each other in order to get more pagerank. So a few years ago they started to lay out guidelines about what webmasters should and shouldn’t do with links. And they started penalizing sites that they judged to be breaking these rules.

Google isn’t transparent about their penalties, but there are widely believed to be three main ones. The -30 penalty forces your site to the 30th place in the search results even if it would normally rank much higher than that. The page 99 penalty (or -950 penalty) relegates certain pages on your site to the very end of the results (page 99) for certain keywords. The third and most serious penalty is exclusion from the index.

One good example here is the company Text Link Ads. They provide a marketplace for people to buy and sell paid links. That is, they make money by distorting the natural order of things that Google cares about so much. If you search for “text link ads” in Google you will not find their website. It has been banished.

Interestingly, the topmost result when I searched for “text link ads” a few minutes ago was an article reviewing the Test Link Ads service. So they are managing to get good placement despite having been banished.

The Nofollow Exception

It is possible to create paid links that don’t anger Google. All you need to do is add the “nofollow” attribute to any unnatural link. Modifying the link I gave above, this would look as follows:

<a rel="nofollow" href="http://youlookfab.com">this link to Angie's site</a>

The rel=”nofollow” bit tells the Google crawler to ignore this link, so it won’t be counted when calculating the pagerank of YLF. Of course, this doesn’t solve the problem for people who buy and sell links because they want links that do impact the pagerank.

What this Means for You

I believe that if you care about getting traffic from Google, you need to care about their guidelines. For this reason we do not pay for natural links to YLF, nor do we accept money to put natural links on YLF . When we do link to an advertiser in a post thanking our sponsors or in a review article where the advertiser provided free merchandise, we use the “nofollow” attribute. We are also upfront with advertisers about this practice.

Note that while buying paid links is one example of Search Engine Optimization (SEO) — the set of techniques one can user to make ones site rank well in the search engines — I am not implying that all SEO is bad. There are many good SEO practices that make Google a better search engine by making it more knowledgeable about your site’s content (e.g. using a sitemap). What is bad is anything that distorts the natural link order.

Could you get away with buying and selling links? Probably. Is it worth it? If you are highly dependent on Google traffic, probably not. You might also agree with Google that the web is better off with people linking based on quality and relevance of content, rather than pure dollars. In fact, if paid links started to dominate the web, then Google would have a much harder time providing good search results. And then there would be no SEO-related  reason to pay for links in the first place.

Tragedy of the Commons

This brings us to the tragedy of the commons. From Wikipedia:

The tragedy of the commons refers to a dilemma described in an influential article by that name written by Garrett Hardin and first published in the journal Science in 1968. The article describes a situation in which multiple individuals, acting independently, and solely and rationally consulting their own self-interest, will ultimately deplete a shared limited resource even when it is clear that it is not in anyone’s long-term interest for this to happen.

Will this be true of links on the web? Will commercial interests ultimately ruin the relevance that one can infer from a hyperlink?

One way to find out how your site ranks is to do a search on Google and see where you end up. This is time consuming, however, and there are online tools that make things a lot easier. The SEOBook rank checker is one such tool. Just type in the keyword and your site URL (in my case, the URL is “www.youlookfab.com”) and click “Check Rank”. The result comes back pretty quickly.

Here are some examples this morning (rankings in parentheses):

• smart casual style (1)
• structured clothing (1)
• what to wear to a black tie event (7)
• pretty tops (9)
• smart casual (10)
• how to get rid of deodorant stains on darks (10)
• dress over leggings (10)
• clothes to make you look slim (28)
• body type (44)

There are some mysteries about Google keyword ranks. For example, the keyword “what to wear” doesn’t rank at all for YLF. It only appears in the paid results on the right hand side of the page (because we pay to put it there – more on that in a future post).

There are many other tools like the one I used above. Some of them will also check other search engines, like RankChecker, which gives results for Google, Yahoo and MSN (Live). Note that most of these sites are covered in advertising for products and services that promise to improve your rankings in the search engines. Be very wary of these promises – there are a lot of good search engine optimization (SEO) professionals out there, but there are also a lot of snake oil salesmen.

Well, something worked. Our traffic returned to pre-plunge levels a few days ago. Courtesy of Google Webmaster Tools, here is a chart of our traffic from Google over the period.

Other default characteristics you can specify:

• Width and height
• Background color
• Headlines only versus Headlines and articles
• Whether images are displayed

Note these are only defaults. Your user can change these characteristics to suit their blog. Bear this in mind when creating your widget and try the different options to see whether they look OK.

Once you have created your widget it is immediately available on Widgetbox. See the YLF widgets here.

Likes:

• Quick and easy. It takes a few minutes to create a widget, and the result is easy for your readers to deploy.
• Monitoring tools. There is a nice little tool to monitor the performance of your widgets.

Dislikes:

• Limited creative. Widget customization options are a bit limited. I would like to make my widgets more graphic and “flat” (less gradient and less glass effect), but this isn’t possible. There is no way to change the text headlines to a different color, so you are stuck with their choice of blue. It might also be nice to have a background image.
• Ugly thumbnails. The thumbnail representation of my widgets on the Widgetbox site look pretty ugly, and they also reflect an older version of the widget in some cases. I think this might hurt my ability to get widget users, but I don’t want to design to their thumbnail.
• Neutered backlink. You don’t get a real backlink from the widget. People clicking on the widget will come to your site, but the Google crawler won’t recognize that link when calculating the ranking for your site. So if a blogger deploys your widget on their site instead of putting you in their blogroll, you lose the positive impact that the link would have had on your search engine rankings.

One thing to watch like a hawk with any widget is the overhead it adds to your user’s page load time after they have installed your widget. The widgetbox impact looks reasonable to the naked eye, but the widget does make quite a lengthy exchange with their server. Reload this page and watch the messages that appear at the bottom of your browser (e.g. “Read widgetserver.com” and “Transferring data from flash.quantserve.com…”). If I had a lot of other scripts on this site I might not be happy with this additional overhead.

Still, it is a very quick and easy way to create some nice looking, functional widgets. I think we will use these on YLF until we do something more sophisticated of our own.

Since the plunge I have…

1. Installed the All in One SEO Pack plugin. Google’s content analysis tool (part of Google Webmaster Tools, or GWT) told me that I have duplicate title tags. This plugin ensures that each page has its own unique title tag.
2. Removed the meta description from the header. GWT content analysis also told me that I have duplicate meta descriptions. Some research on the web revealed the consensus that no meta description is better than duplicating the same meta description across all the pages.
3. Resubmitted the sitemap. I also installed the latest version of the Google XML sitemaps plugin.

One thing I noticed in GWT was that our incoming external links had dropped very low. the reason? Our home page was not the list. I checked that our site was indexed and it was. Recently the external link stats returned to normal and our home page is back in the list.

We are also getting quite a few web crawl errors that don’t make sense. Perhaps our server was down while the Google crawler was doing its thing, but I don’t see that in the server logs.