Accidental Keyword Optimization and Latent Semantic Indexing

Several weeks ago, I took a class on the internet about Latent Semantic Indexing, and since then, I've been discussing this with many web content writers and web developers, of which I am both a writer and web developer. There are so many people who don't understand how Latent Semantic Indexing (LSI) works, and how that can affect people who write web copy or build websites for search engine optimization.

Latent Semantic Indexing is relatively new on the keyword optimization arena. This technology, which is used almost exclusively by Google at the moment, is an automated way of trying to compete with the new trend of live search sites. Live search sites came about because many ordinary internet surfers have had trouble using Boolean search strings to find content on the internet.

Too many people have gone to Google or another search engine or directory and they type their query in and then receive hundreds of thousand or even millions of returns for the search, but when they start clicking on the search returns, while they share the same keywords that were typed into the query, the results don't contain relevant information to what the searching was browsing.

For example, prior to LSI being used, I wrote an article entitled How to Utilize Keywords and Keyword Density. This article will help you clearly understand how keywords affect the way search engines and directories index your information when it is published on the web. The article is still valid, so I encourage you to read it if you need help with keywords, but more than that, I want you to see how LSI has changed the face of keyword writing and relevancy on the internet. You can right click on that link and open my article in a new window so you can follow along with me.

Back to the live searches now. Live searches were a great idea, in that they took a group of people who were trained in how to search for information that was relevant to a query on the internet and then those people would do the searching for you, and return only results that were relevant to your query.

Why was this necessary? Because, if you will look at my article on keywords linked above, you will see that I intentionally used a high concentration of the following words in that article:

* Jalapeno peppers = 2.18%
* Cooking with jalapeno peppers = 2.32%
* Cooking = 1.09%

Now, when this article first indexed in Google, prior to the implementation of Latent Semantic Indexing, the article could be found on about page six of Google for a search on Keyword Optimization, but it was found on the very first page of Google when searching for the phrase "cooking with jalapeno peppers"!

Yet, if you were to have been looking for information about cooking with jalapenos, you would have been disappointed when you clicked my link and found an article about keywords and search engine optimization. After all, you can't cook with SEO, and jalapeno peppers on your computer keyboard will actually eat through the plastic. (Try it sometime. It's really kind of cool.)

The point here is, live searchers would have clicked that link, seen that it wasn't relevant, and would have excluded my article from the search results for the phrase "cooking with jalapeno peppers".

So Google comes along and says that live searches aren't efficient enough, and that searching should be an automated process. Not to mention, they are leaders in technology on many fronts, so they wanted to once again be the first at something great, and thus enters Latent Semantic Indexing.

Latent Semantic Indexing is a search engine's attempt to automate and replicate relevant live search results without having to use a live person to click the link and look to see if the result was relevant to the query.

Once LSI is being used properly, if you Google the phrase "cooking with jalapeno peppers" you should no longer see my keyword density article showing up on the front page. In fact, it's unlikely it will show up in that particular search on Google at all. Yet, if you read the comments on that previous article, you'll clearly see that it was on the front page when the article first came out, and while it does currently still show up in that search, when LSI is perfected, it should disappear.

Why? Because LSI would determine it was not relevant to cooking, peppers, or jalapenos, and therefore, it doesn't belong in the results for people searching for those phrases.

I'm sure you're wondering now how this all works, right? Glad you asked.

I don't know.

In fact, no one but Google probably knows for sure how it works, and they sure aren't going to give away the secret, but here's what we do know about Latent Semantic Indexing, at least, from my point of view here and from what I've learned in my classes.

LSI will scan the content for keywords the same way that it always has in the past. It will look at what words show up with a high concentration within the indexable words on a page (read the other article I linked to for more information on what is indexable). Then, when that is finished, it will take those keywords and will search on other websites that have a similar or high concentration of those same keywords.

Once it's found those other web pages with the same or similar keywords as your content, LSI will then compare other things such as: grammar, syntax, terminology, etc., and see if the rest of your web page contains similar terminology to those that are already indexed and known to be about whatever topic your keywords are about.

For example, when my article about keyword density was indexed without LSI, the keywords that I purposely used to prove a point were indexed, giving me a non relevant top level slot for cooking with jalapeno peppers. But with LSI, the article I wrote would be matched against other articles with high keyword density for that search phrase, and LSI should determine that my article did not have the other words and terminology to be relevant to cooking.

After all, I merely used the phrase 'cooking with jalapeno peppers' too many times - but there's nothing about cooking, utensils, and other things that go along with a cooking article. I don't want to list too many of those other relevant terms here, or else this article too will index for cooking, and we really don't want that to happen.

The point here is that LSI will match your content with other previously indexed content using the same keywords, and will look for similarities in terminology to make the search results relevant to what the searcher is actually looking to find.

What does this mean to you, the web copy writer or the web developer looking for good ranking in search engines?

What it means is that we are no longer slaves to keywords. Instead of using the same keyword and keyword phrases over and over again to get the highest concentration (density) that we can, now we can mix it up a bit and use other relevant terms and phrases that match our content, making our copy read more smoothly, more the way we are used to reading, without feeling choppy and repetitive.

LSI can bring back good, quality writing, instead of keyword dense, short and choppy writing to get high rankings in search results.

It also means that there is a bigger burden on web designers and developers to find quality content, instead of just keyword dense or keyword rich content. This is a good thing for writers who know how to write. In the past, freelance writers were losing jobs to people who could write SEO content, whether it was good writing or not, and get those all important keywords optimized in the content.

Now, LSI is going to make quality, grammatically correct, relevant writing come back into the picture. Websites will have more valuable information or else they will move to the back of the search results, writers can once again write instead of keyword stuff, and the end user who has trouble finding things on the internet that match what they are looking for will now find more relevant search results, without having to wade through things that just aren't what they wanted.

Another thing LSI should help with is removing blackhatters who try for keyword stuffing and underhanded behind the scenes keyword techniques to get indexed high for non relevant searches just because they were able to stuff a bunch of words onto a page that the user couldn't see, but the bots could. Of course, blackhatters will always find a way around things to break the rules and steal page views and clicks, but LSI will help move many of them way to the back of the line in search results or eliminate them from search results altogether.

LSI will also make it much less likely to accidentally index for phrases and terms that just happen to have a high density in an article but really have nothing to do with what the article means.

One last note before we wrap this up: keywords are still important.

LSI doesn't do away with the need for keywords and keyword density in your writing or web pages. You still want to be sure that the keywords and keyword phrases you want your copy to be found for are still the most prevalent in your content, or else LSI doesn't know what phrases to use in order to compare your content to others.

Whatever keyword phrase or keywords you want to use for your content still need to have the highest density over the other indexable words, but now, there is no set number, as in the past, such as needing to get 2-4.5% keyword density for a particular word.

With LSI in play, all you need to do now if pick your keywords and keyword phrases (and with LSI, keyword phrases mean more than just single keywords) and just ensure they are higher in density than the other indexable words or phrases in your content. As long as they are, LSI will compare them to other content with the same keywords, and you no longer have to worry about percentages and density and the like.

That's really the beauty of Latent Semantic Indexing - writing can be coherent and valuable and interesting again, without keyword stuffing or a focus on keyword density, and the better the writing is, the better the content will perform.