Stop Me If You've Seen This Word Before

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

<i>Way back in 2004, I ran a little experiment with Google -- over a period of a week, I searched for an entire dictionary of ~110k individual English words and recorded how many hits Google returned for each.</i><p>Of course, a word can appear on a page multiple times. That's why, I think, folks used to ignore the stopwords. They introduced noise when trying to access the content words. Now, with span constraints, you can incorporate them into the analysis. So "a matrix" and "the matrix" returns very different results, even without quotes.

&#62; "The" is one of the most common words in the English language<p>"the" is THE most common word in English.

It makes sense that low-information terms would have a lower preference when searching without any context. If your index models the context around terms, you can get better results from a low-information search.<p>I think...I'm kind of shooting from the hip here relating it to context modeling in lossless compression schemes like CABAC and PPM.<p>Could you overcome stop words with some sort of Bayesian phrase matching over some learned hidden states?

POPFile has stopwords because people in the community insisted on it. My commercial email filtering software does not because it turned out that in my tests that the accuracy difference they made was so small as to be in the noise. And they were costly in terms of time to check, and to maintain across different languages.

I guess the idea was to help allow English search queries (i.e. exclude words people were using to describe their query but shouldn't be searched for).

it is about how to sort with stop word. Tranditional tf-idf method didn't work well as it didn't contain any information about each word relative location in its context. a simple method is to index "the the", the word group instead of single "the". I guess it is what Google does now with "to be or not to be". However, the word grouping tech is a common method in CJK full text search.

I like how the top Google words are all generic web marketing words, with the two exceptions of "hotels" and "women."

<a href="http://www.google.com/search?hl=en&#38;safe=off&#38;q=%22the+the%22+band&#38;btnG=Search" rel="nofollow">http://www.google.com/search?hl=en&#38;safe=off&#38;q=%22the...</a><p>as soon as the article asserted this wouldn't work, i tried googling, and it worked fine. i stopped reading after that.<p>edit: for whatever reason, if you follow the link directly, the search results are wrong. you might have to submit the query again after the page loads to get the right results. weird! maybe he was onto something (nope)