There are a number of ways a staging site can cause you problems with SEO.

For one thing, the search engines might find and index your staging site.  Worse, it might actually rank better than your main site – especially if you are road testing SEO-focused technical upgrades on the staging site first.  Worse still, the search engines might decide to drop the main site from the index showing preference to the staging site.

This is especially a problem for an ecommerce site.  Even more so if (like they should be) the admin systems for your staging and main websites are distinct – if a user places an order on the staging site, this might not get fulfilled, which ultimately damages your online reputation in the long term, even if you repair the technical issues to prevent it happening in the future.

You might try to prevent the staging site being indexed using a robots exclusion file (robots.txt).  This is quite common, but it carries its own risks.  Many times I have seen robots exclusion files from staging sites uploaded to main websites in the past.  The result is what stopped a staging site being indexed suddenly stops a main site from being indexed.  And until the sites are dropping from their normal ranking, it can be quite difficult to notice – do you go and check your robots.txt file every day to make sure it hasn’t changed?

To overcome these risks, there are two things you should do.

Firstly, you should implement the canonical tag as standard in your code.  This has to be done correctly, as a poor implementation is also risky.  However, the benefit of doing this right is that even if search engines do begin to index your staging site, the URLs they list should be those on the main website, so searchers should no inadvertently end up on the staging site in the first place.

The canonical tag should always use the domain of the main website, and the extended URI of the page which is being viewed.

For example, on http://staging.domain.com/page10.htm, the canonical tag should be

<link rel=”canonical” href=”http://www.domain.com/page10.htm” />

The second thing that you should do is to password protect your staging site.  This has two benefits.  The primary benefit is that the site cannot be indexed by search engines, and so should not appear in search results (remember to send the correct “Unauthorised” 401 header response, so the search engines understand it is protected content). The secondary benefit is that competitors cannot get an early heads up on new features of functionality that are being tested on the staging site.

As an aside, have you been able to find and explore your competitors’ staging sites at all?  If they have left them open to public interrogation by not doing the above, it gives you a chance to keep up with what they may be planning.  Thanks to Sam Crocker for that tip at last year’s ProSEO!

By implementing the staging site in the manner described above, you are able to maintain the same robots exclusion (robots.txt) file on both your main website and your staging site (written from the point of view of your main website), preventing ranking catastrophes should it be inadvertently uploaded to the main site from staging at any point.

Bear in mind though, that if you are using .htaccess on Apache to password protect your staging site, then there is a risk you might upload the .htaccess file that causes the password requirement – this is much easier to spot than an inadvertent robots.txt upload though, as all users will be asked for a password.  Make sure you have a backup of your normal root .htaccess file if your site uses one, so you can restore it if you need to.

You might (rightly) point out that if you carry out the password protection, the canonical tag is unnecessary.  This is true, but in my experience few developers will password protect their staging servers.  However, whether they do or not – if the password protection is ever absent (for example, if an .htaccess file is deleted by mistake), the canonical tag provides back-up protection.

More than this though, so long as it is done correctly, in my opinion, use of the canonical tag should just form part of your best practice, anyhow.

As part of her presentation, she discussed how she has raised ad revenues by including third-party advertising on Yahoo’s log-in page.

My impression at the time was that if Yahoo thinks that distracting its users with adverts while they log-in is a good idea, then they are doomed.

Now Yahoo have sacked Carol Bartz – is that an indication that the penny may have dropped?

Infographic by WordStream Internet Marketing

On 15^th April, 1989, following a breakdown in command by South Yorkshire police, and a catastrophic chain of planning mistakes and poor stadium design, 96 Liverpool fans from around the UK were crushed to death by sheer weight of numbers, in a disaster similar to those at the Roskilde music festival in June, 2000 (http://en.wikipedia.org/wiki/Roskilde_Festival) and the Love Parade in Germany in July 2010 (http://en.wikipedia.org/wiki/Love_Parade_disaster).

Why bring this up now, 21 years after the event? Well, feeling still runs high among Liverpool fans that there is no sense of justice for the families of the dead.  Despite a general acceptance among most people in the UK that the disaster was the result of a collection of mistakes by those in charge of the safety of fans on the day (http://www.bbc.co.uk/liverpool/content/articles/2006/12/07/local_history_hillsborough_feature.shtml), there is still resentment that some quarters of the press, through sensationalist journalism at the time, reported that fans were responsible for the disaster – claims subsequently retracted by most of the offending press.

The official enquiry into the disaster exonerated fans from blame, and highlighted several failure in command by the police, and issues with the design of stadiums and the general treatment of football fans by organising authorities.  It is because of this report that all UK top league stadia are all-seater now, for example.

Therefore, to read reporters in section of the press in the United States describe the Hillsborough disaster as a “riot” is not only disrespectful and ignorant, it is also distressing. Barely a family on Merseyside didn’t lose a brother or sister or friend at Hillsborough.  Describing Hillsborough as a riot would be like describing Hurricane Katrina as a riot – not only patently untrue, but indescribably upsetting to the families of the victims.

Therefore, I would suggest that Alex Beam of the Boston Globe does a little research into the subject matter of his articles before writing such ill-informed nonsense in the future. If he has the sense to do so, maybe he will consider retracting some of the nonsense he has written at http://www.boston.com/sports/soccer/articles/2010/12/07/alex_beam_hardball_in_liverpool/ and perhaps even consider a donation to the families support group.

To read more about the history of the Hillsborough Disaster, search Google, or try these handy links:

http://en.wikipedia.org/wiki/Hillsborough_disaster

http://www.bbc.co.uk/liverpool/content/articles/2006/12/07/local_history_hillsborough_feature.shtml

http://www.youtube.com/watch?v=W5PCrbRl3DI

http://news.bbc.co.uk/onthisday/hi/dates/stories/april/15/newsid_2491000/2491195.stm

http://www.mirrorfootball.co.uk/archive/The-Hillsborough-disaster-The-darkest-day-in-British-football-history-article392289.html

http://www.epltalk.com/the-truth-about-the-hillsborough-disaster/491

http://www.telegraph.co.uk/sport/picturegalleries/5145204/Hillsborough-disaster-Two-decades-of-hurt.html

http://www.liverpoolfc.tv/history/hillsborough

UPDATE: A few hours after the Boston Globe Article was brought to my attention, Globe editors appended the following to their article:

“Correction: Because of a reporting error, Alex Beam’s column on Tuesday in the “g” section mischaracterized the 1989 Hillsborough stadium disaster involving Liverpool soccer fans as a “riot.” The official investigation into the disaster, which cost 96 lives, placed the blame primarily on poor crowd control and inadequate stadium design.”

However, Latent Dirichlet allocation (LDA) is a fairly advanced statistical concept, using complex probability maths.  If you have a mathematically inclined brain, and you’re brave, you could check out the Wikipedia page entry for LDA here. Several people in the SEO community seem to confuse LDA with a concept many of us have known about in SEO for a few years – latent semantic indexing (LSI).

Latent semantic indexing is also a statistical concept using probability maths, but it is less complex than LDA. If you’re still feeling brave, the Wikipedia page describing the mathematical concept is here.

LDA and LSI both describe mathematical models that are designed to be used for information retrieval – i.e. returning search results.

In a nutshell, LSI examines the words used in a document and looks for their relationships with other words. Google issued a patent in 2005 that discuss looking at three types of relationship – lateral (where a word means the same or very similar to something else, e.g. car, automobile and auto), “kind of” (where a word is a kind of something else, e.g. car and vehicle), and “part of” (where a words is part of a larger concept, e.g. engine and car).

LSI allows a search engine to determine the kinds of words that a web page might be relevant for, even if they are not actually used on the web page itself.  Bu writing content that is packed full of words that have relationships with each other, you are strengthening the document for all of those words.

However, LSI has one major weakness – ambiguity.  How could a seach engine determine if you are talking about Microsoft office, or the office in which you work.  Especially if you excel in what you do, providing a fresh outlook in your job as a publisher of bibles (you want to give more people access to the word of God, see).  I can’t think of any OneNote or Powerpoint puns, but you get the idea.

LDA, on the other hand is a significant extension of LSI.  Words are grouped into topics.  They can exist in more than one topic, in fact most do.  LDA tackles ambiguity by comparing a document to two topics and determining which topic is closer to the document, across all combinations of topics which seem broadly relevant.  In doing so, LDA helps an information retrieval system (such as a search engine) to determine which documents are most relevant to which topics.

The boffins at SEOMoz (apologies for the Bigmouthmedia-style start to that sentence) have built an excellent tool to determine an LDA score for text documents against search terms, and have tests many hundreds of sets of Google search results against what might be expected by their LDA score.

Now clearly, the results did not exactly match the LDA-based predictions, but they came so very close that SEOMoz – famous for their belief that link profile far outweighed most other ranking factors in Google – now suggest that the text on your web pages has a much stronger influence than they used to believe.

Looking beyond that, clearly there are a large number of factors that Google uses to rank websites – it’s all a rich tapestry, so they say. However, LDA is possibly the most spam resistant way of determining what a web page is about.  LSI synonym spamming won’t necessarily help – especially if the synonyms are ambiguous, or worse change the context of the meaning of the original word. Google bombing links won’t help LDA scoring.  Admittedly, it doesn’t prevent some of the black-hat mainstays like hidden content and cloaking, but then there are human-applied ranking penalties and bans for that kind of behaviour.

I’m not suggesting LDA can’t be spammed (actually, clearly it can by some elements of keyword density spamming), but it does allow documents that aren’t spammed to compete well in topics with those that are.

So, do Google use LDA algorithms?  There don’t seem to be any patents specifically mentioning it yet, but the patents aren’t always published quickly.  Additionally, the patents (if they exist) will almost certainly be clouded in a little fog to prevent SEOs from backward engineering Google.  However, the correlation seems very strong, and of course LDA is a specific mathematical information retrieval model. My belief is that it is likely that they do apply it.

The final, albeit tenuous evidence is the way that Google talk about their search results, and about how you should write your content.

Matt Cutts is always saying things like “if you want to rank well, write great content.”  To many this seems like a cop out, but for an LDA-based information retrieval system, this is exactly what you should be doing.

Additionally, at the search event earlier this week, while Google Instant was being extolled to the masses of gathered journalists, Ben Gomes – one of the chief engineers at Google, and in many ways, the public face of Google Instant – referred over and over again to searching for topics, and refining your search topic.  Not keywords.  Not search terms. Topics.

Naysayers claim that search engine optimisation is irrelevant now, or that the long-tail is dead.  But I couldn’t disagree more.

First of all, search engine optimisation is not dead.  Google Instant does not change how Google crawls, indexes or ranks a website – the primary reasons for undertaking search engine optimisation in the first place.  Sites still need to compete against each other for popular keyword searches in order that they can acquire traffic from Google.
In fact, as it is now much easier and quicker for a searcher to refine a search and edit what they searched for, it is now more important than ever to appear in the first few positions for important keyword phrases than it ever has been.  The search engine optimisation industry should now be more vital than ever!

What will change is that users are able to refine and specify their search much more easily, and so users are much more likely to continue to refine their search until one of the first few results on a page is relevant to what they are searching for.  Far from being dead, the long-tail search is going to come into its own!

Google claims that Google Instant is going to save the world 11 billion hours a year that people will spend not searching.  I think it is more likely that users will learn to search in a different way.  They will save time, but they will search more and search around topics to find the information they want. People will be less willing to accept the first information they find about a subject, and will retrieve and digest information more than before.

Ecommerce sites will need to become more competitive.  Those with niche markets will be able to serve those niche markets, but it will become more difficult than ever to break into already busy markets, such as MP3 sales, groceries, perfumes, and gadgets.

Usability test at Google, including eye tracking, determined that user focused on the search box first until their suggested search was close to what they were really looking for, then their eye would scan the first result (or sometimes the first two or three results) of the organic search.  If the exact thing they were looking for was not in those first results, their eye would go back to the search box, and they’d refine their search and go through the same process again, until finally they would find an exact match for what they were searching.

On the whole, users’ eyes did not scan to sponsored results until the subject matter in the organic results was close to what was being searched.

Consequently, sites that run Adwords campaigns need to ensure they are managed properly.  Broad-brush, short tail search Adwords campaigns will become less effective and those which carefully manage long-tail will do better.

As in nature, the fittest will survive.  And, as in nature, that doesn’t necessarily mean the strongest, but actually those more adaptable to change.

Search marketing, both organic and paid is changing, and those who do nothing may be those who do die.

Here are some more of its points, in just a little more detail.

Using the Free Tools Effectively

Search engines give you a remarkable array of tools to help you understand how people use your website, and how well search engines can index it. These tools include:

• Google Webmaster Tools
• Yahoo Site Explorer
• Bing Webmaster Tools
• Google Analytics

Each of the above should be used in conjuction with the other three. The first three tell you a little about what the search engines know about your website, for example, how many pages they are aware of on your website, how many links they are aware of to your site, and so on.

Such information is extremely helpful in uncovering weaknesses in the SEO profile of your website.

Google Analytics is a different beast.  It tells you a wide range of metrics about the people that find your website, and how they behave within it.  It also indicates which of your marketing channels are successful, and which ones are not.  Used properly, it is extremely helpful in indicating how marketing can be improved.

Remember when using Google Analytics to set up goal recording and hence conversion correctly.  Funnels for goal achievement are also important.  Correctly set up, these help you identify what bottlenecks there are in the conversion process and therefore what you might need to improve to increase conversion.

Telling the Search Engines About Your Website

All the major search engines accept submission using the XML sitemap protocol.  This is a handy way of indicating to a search engine the URLs that make up your website, in order to ensure that the website URLs are crawled effectively.  It is important to understand that the sitemaps do not influence ranking, just whether a search engine knows of a URL or not.  They don’t even guarantee inclusion, because clearly you could include URLs of web pages that contravene the search engine guidelines.

Probably the one thing most neglected by websites that rely on XML sitemaps is that they cannot cause PageRank to flow to pages that cannot be reached by other navigation methods.  From a best practice SEO perspective, XML sitemaps DO NOT replace good, indexable navigation.

Today, I’m going to cover a few more of the issues Ian discussed in his post.  First of all, error pages.

Get Your Errors Right!

From time-to-time, every website has an error.  Whether you’ve got a typo in your code, or a user mistypes a URL, or maybe your database server decides that it’s out to lunch – every site has had to deal with such things.

From a user’s point of view, there’s little more irritating that getting an error page.  Even worse, getting an unhelpful error page with technobabble and no real help.

Therefore, it helps to create a useful, on-topic error page to handle different kinds of error.  If a 404 error is encountered, give a helpful, friendly error message inside a corporate themed web page.  If possible, try to link to what the user might have been looking for – if they seemed like they were looking for a product page, show them some product categories.  Show them a search box. In short, do what you can to help them find what they were actually looking for.

The more you help a user who encounters a problem, the less chance there is that they’ll turn round, head out, and buy their red widget from some other website.

Go For Diet HTML!

Yesterday, we mentioned page loading speed.  One of the factors of loading speed is the amount of bulk in the code on a page.  There are two competing factors that can affect this – DNS lookups (Ian doesn’t talk about this though) and JavaScript/CSS code.

Ideally, all JavaScript and CSS code should be in external files. That is, in your HTML code, you only reference the JavaScript or CSS as being in a separate file, you don’t include within the HTML itself.  This reduces the amount of code in the actual HTML page.

However, each time you request a new file from a server, the browser has to go and fetch it, which slows down the rendering of the page. Additionally, if the file is on a different domain, the browser also has to lookup the address of the server to find where it is before it can fetch it, which slows down the process further.

There is a balance to strike clearly. Look-ups and fetches slow the process of displaying a web-page down.  However, once you have downloaded a file, unless the server instructs the browser NOT to cache the file, it will be cached so that next time (within a reasonable time period) it doesn’t need to be downloaded again.  This means that when you view the next page of the same website, if it uses the same JavaScript files, and the same CSS files, they are already downloaded, and the page will render much quicker.

Ideally then, the balance must be to keep as much JavaScript or CSS as possible in just one or a few files that are shared across all pages of a website.  In this way, few look-ups and fetches are required, but the majority of the CSS and JavaScript is download on the first page and cached for later use – the total code downloaded over the space of a single multi-page visit can be dramatically reduced.

First of all I guess I should say that I agree with every point he makes, although some of the content of the points have changed slightly in the context of what has happened in the SEO marketplace since Ian posted this.

So for now, I’ll take the first few points and hopefully explain them a little more completely than Ian did in that one post.

1) Kill Your Flash Intro!

If you have a Flash intro to your website, you are doing a number of things – annoying the user, wasting the user’s time, using you bandwidth, giving the user an opportunity to give up on your site before they’ve seen your content, and paying your designer more than you need.  What you are not doing is taking those vital first few seconds of a user’s visit to your site to engage them in what they are actually looking for.

Very few website users want to be impressed with how your website looks and behaves – that isn’t why they are visiting you – they want to buy a red widget, or subscribe to your RSS, or whatever it is that they actually visited your site for in the first place.  Let them find how to do that quickly and easily – everything else is secondary.

2) Make Your Pages Load Quickly

The speed pages load on your website is vitally important for two reasons – in fact, that is one more reason than when Ian wrote his post! If your pages take too long to load, your visitors will leave before they have finished loading, or at the very least you are slowing down their path to conversion – i.e. why they came to your site in the first place.  Slow that path down too much, and they’ll head right back out the door and go somewhere else.  The second reason, and the one that didn’t exist before, is that Google now use the perceived loading speed as a quality indicator – that is, your page load speed will potentially influence the ranking in search results pages!

3) Fix Those 404 Errors Now!

Gaining a user’s trust can be a difficult battle.  Anything your site does to reduce the user’s trust will reduce the likelihood of a website user becoming a customer, even in the future. Negative experiences stick in the mind – sometimes a potential customer can’t even remember what the negative experience was, but once you’ve created a negative experience, it can take several positive experiences to counteract the negative effect. Broken links can cause your website users to lose trust in your website.  In their eyes, if you can’t even get a link right, how can they trust that you’ll get someting complicated right, like fulfilling their order?  They might not put it quite so bluntly as that, but the negative subtle correlation is likely.

Also, broken links are bad news for SEO – you’re bleeding PageRank into non-existant URLs instead of ones that could benefit you, and you are creating a big flag for search engines that says “I don’t look after my website properly, don’t rank me so high.” Often, they oblige!

That’s your lot for this post.  I’ll tackle some more points from Ian’s wonderful information next time!

The essense of the piece I read was that if you use JavaScript to “pop-up” sub-menus, then your site navigation won’t be indexed by search engines.  You’ve probably read similar claims hundreds of times also.

Here is why that is an over-simplicification, and to most intents and purposes, completely wrong.

Most (not all mind) JavaScript pop-up menus rely on JavaScript controlling the “display” or “visibility” or “z-Index” CSS of an HTML div layer or layers to control the pop-up.  Others switch the innerHTML in and out of other HTML div layers that remain permanently hidden by one of those same CSS methods.  Another way is to use JavaScript to swap a CSS class from one where the div layer is hidden to one where it displays (and back again).

In each of those examples, the links that make up the sub-menu items are able to be physically on-page, and indexable, as they are just standard HTML. In fact, the JavaScript is by-and-large irrelavant to their rendering (they are only “hidden” by CSS – JavaScript is manipulating that!).

This means that menus created in those ways are indexable in a way that is virually identical to HTML only or HTML/CSS menus.  There only differences perhaps being the subtle nuances of importance implied by the particular HTML mark-up used.

There are a few things to avoid though:

Never use JavaScript for the links themselves.  While Google now makes a good attempt to understand most JavaScript it encounters, each step away from HTML is a step away from a search engines rendering comfort zone.  Stick with using JavaScript to manipulate the HTML and CSS, and you’ll be safe.

Secondly, don’t “hide” HTML in an unsafe manner.  Search engines expect to find hidden div layers for navigation (several people believe hiding HTML will incur a search egnines wrath as an extension of the issue I’m about to highlight here).  However, search engines are also on the lookout for sites “hiding” text content from users for no good reason.  If your hidden HTML contains text significantly more than links, it is likely to look like a cheap attempt to cloak.  Even links set up in a non-menu way might trigger suspicion.  Avoid hiding div layers by poisitoning them with negative left margins greater than their width, and so on.  These are red flags to Matt Cutt’s spam police, even though they could be forgivable for a menu.  Avoid anything that puts you on the radar of the spam police!

Finally, don’t use use CSS to hide any other text on your page without good reason – for example as storage for text used in a marquee display – as this is almost certain to trigger spam filters and bring you to the attention of Matt Cutts’ team.  He’s a nice guy, but he’s a trained spam killer!