Latent Semantic Indexing: What's Old is News Again

To webmasters, SEO consultants, and many writers, Latent Semantic Indexing (LSI) is a new and baffling concept. To writers like me, who have been using computers to create, process, index, sort, translate, and retrieve information for decades, it's old news. The earliest LSI-related research published was in the April, 1963 issue of the Journal of the Association for Computing Machinery (ACM). The article was "Automatic Document Classification". 1963: the year John Kennedy was assassinated, the Beatles released their first album, Alcatraz prison closed, and George Michael was born. That's extremely old news.

LSI is just information retrieval with a new development. The new development is not LSI, which was patented over 20 years ago, it is using LSI on the internet.

Information retrieval has been a problem for thousands of years because writing things down is not as difficult as finding them later when you need them. Shortly after writing was invented someone probably had a difficult time finding the clay tablet with the inventory of the king's treasure, and the index had to be invented. Indexes have their problems, as anyone who has used a library's card catalog or the index at the back of a book knows. Indexes can only capture broad topics, and they are only as good as the choices made by the creator of the index. The bit of information you need to solve your problem may be in a book, but you will never find it if it didn't get linked to the right heading on the card catalog or selected by the person creating a book's index.

Very soon after computers were invented, people started using them for document management. At first just the equivalent of card catalogs were computerized, because computers were so expensive. As the price of computers dropped and their processing power increased, using computers to do full-text indexing became possible. Early indexing software threw out all the extremely common words and made an index of what document or documents contained the remaining words. When a user entered a word or words in the search field, the information retrieval system responded with a list of all the documents that contained those words.

A breakthrough came from research done for machine translation (translating software). Those researchers stopped trying to understand the text and started looking at the relationships among the words, much like the way a human comprehends language. Translation software picks what word to suggest by looking at the surrounding words, using information like "if spring has words flowers or autumn nearby, it's a season; if spring has the words tension or adjust nearby, it's a coiled bit of metal." It's not a perfect method, but it's darned good.

Latent semantic indexing took the idea one step further, using statistics and matrix manipulation to find the word patterns in large collections of unstructured data, such as business letters, web pages, or newspaper articles. Search engines first create semantic indexes, then use the index of word patterns to decide which documents most closely match the words a user enters for search terms.

If semantic indexing is so good, why did it take so long to be used on the web? Capitalism, dear readers. The persons who discovered how to do it it patented the concept and the process. The licenses to use it were not cheap. Only large companies that could afford the software could index their document collections. As soon as the patent expired in 2006, the method became free to use by everyone, and the major search engines wasted no time putting the concept to work.