The search isn't broken, we're broken - Part I : Search success depends on people and rules

by Toby Ward — “People are lazy,” claims Cory Doctorow, a science fiction writer and technologist who maintains the popular weblog Boing Boing. “People are remarkably cavalier about their information and how it is stored. This laziness is bottomless…”
One of the major and universal frustrations heard from Internet and intranet users alike is that corporate search engines "suck". But, are search engines the real problem, or is the issue more complex?
Despite "leaps and bounds" progress in search technology, which is quite advanced, compared to other Web technologies, inaccurate and irrelevant search results continually defeat users performing search queries.
Though some search engines may be sub-par, the more likely problem is an absence of people processes and rules for managing information. 

Hide and Seek

Recent studies reveal that the average corporate employee spends 25-35% of their productive time searching for information to do their day-to-day job. 
How many times have you been to a website where the title heading is either the URL or is missing altogether? And, how often have you searched a corporate site for product information only to be given endless results that link to press releases that are three or four years old? Poorly organized and often out-of-date information frustrates users and erodes productivity.
“I think searching has become a more difficult process for everyone; this has less to with the quality of search engines and more to do with the meteoric growth in data,” says Josh Mugele, Director of Product Management at Semio, a California-based indexing and categorization technology company. “With more to search through, it's difficult for search engines to maintain high levels of both precision and recall.”

Meta-Tag you're it!

One way of capitalizing on the potential of the search function to insert keywords as meta tags within the actual content pages. But this requires rules and a rulebook, otherwise known as the corporate taxonomy. A taxonomy is a set of rules, or dictionary, for classifying or cataloguing information – whether on the Internet, intranet or shared drives via a LAN or WAN.
Meta tags, simply put, are the tags or data that describe the information contained on a page or site. Think of a meta tag as the tag on your shirt collar – it identifies the type of shirt and describes it with information about the materials and the manufacturer. Meta tags can be used to describe the type of data in terms of keywords, description, department, date, author, etc.
If you read technology columnists and analysts, you might be duped into thinking that a taxonomy is simply a software product -- not unlike a corporate portal -- that builds a navigation hierarchy or directory like Yahoo! While it is true that such software products are in abundance, a taxonomy is actually defined as a ‘classification set’ -- a system popularized by biologists that has been in use for centuries before software was even invented. So while a taxonomy can be used to build a directory or navigation tree, it is in essence a rulebook -- and it’s usually built the old-fashioned way, from scratch.

Taxonomy & Search

If used properly, a taxonomy is used in association with the search engine to enhance its potential and its relevance. Most popular search technologies (though not all) can be directed to index, search and retrieve content based upon the meta tag descriptions (using a taxonomy) for each file, document or Web page.
“If everyone would subscribe to such a system and create good meta data for the purposes of describing their goods, services and information, it would be a trivial matter to search the Internet for highly qualified, context-sensitive results: a fan could find all the downloadable music in a given genre, a manufacturer could efficiently discover suppliers, travelers could easily choose a hotel room for an upcoming trip,” espouses Doctorow in a recent article Metacrap: Putting the torch to seven straw-men of the meta-utopia

Taxonomy & Trust

So why aren’t more organizations using meta tags and taxonomies? “The reason why (Internet) search engines don't rely on meta data has to do with trust,” says Andrew Houghton, a software engineer with the nonprofit Online Computer Library Center, a global library cooperative serving 41,000 libraries in 82 countries. “Anyone could place unrelated or misleading information in them to manipulate the search engine result lists.”
Therein lies the "people" dilemma again: meta data is only as good as the author. This means that, on the public Internet, it would be impossible to create and maintain a global taxonomy.
“However, in the corporate arena it's a different story. Once the trust and/or quality issues are resolved then taxonomies, thesauri, and classification systems can be more effectively used… by categorizing the results,” says Houghton. “Instead of getting thousands of documents on cookies, you could get them grouped by their semantic meaning (snack food, computer code, slang term for a cook, etc.). Depending upon the search query you could use a taxonomy to limit the search results.”

Intranet vs Internet Search  

The search isn't broken, we're broken - Part I : Search

Well Beyond the Search Box

Next generation inference engines

The lost meaning of knowledge management

Toby Ward, a former journalist and a regular e-business columnist and speaker, is the President and Founder of Prescient Digital Media. For more information on Prescient’s CMS Blueprint service, or for a free copy of the white paper “Finding ROI”, please contact us.