Ask HN: How do you split strings (to get keywords)?

It depends very heavily on the origin of the string, as that would determine the special cases that needed to be dealt with. Can you provide more details?<p>edit: Based on what you said in your original post, I would say to have a list of possible delimiters (which would probably need to be added to for some time), and tokenize the string according to that, and discard any token that appears in a second list of words that don't matter (conjunctions, articles, prepositions, etc...). Before discarding said strings, you'd also want to check if they're operators used in your app, or anything like that.

How about term extraction?<p><a href="http://developer.yahoo.com/search/content/V1/termExtraction.html" rel="nofollow">http://developer.yahoo.com/search/content/V1/termExtraction....</a>

<i>"Second trial, you use a alpha-numeric whitelist and split on anything else, but what about umlauts? What about hebrew or cyrillic?"</i><p>A multi-lingual version of this could use the Unicode "General Category" character classes (Letter, Mark, Number, Punctuation, Symbol, Separator, Other).

Not sure what your main goal is, but in my compilers project class, we used lexical analyzers to break out tokens from the input stream.<p>Try looking up "Lex" or "Flex"...these were the tools we used. There may be better ones around now.<p>Here's a quick google: <a href="http://dinosaur.compilertools.net/" rel="nofollow">http://dinosaur.compilertools.net/</a>

I just found LingPipe on Monday, and haven't had a chance to try it yet. But it has 'entity extraction' in text mining. Not sure if that is what you're looking for. It's a Java library. <a href="http://alias-i.com/lingpipe/" rel="nofollow">http://alias-i.com/lingpipe/</a><p>anybody have any comments about it?