StreamTokenizer class in .net / C#

I don't think there is any build in way. There are few parser/lexer tools available. Antlr (www.antlr.org) is a good one that would do that for you, but you'd have to write grammar for it. I'm writing somethign similiar to StreamTokenizer, but with more capabilities, but it's not finished yet. I can send you whatever I have now if you're interested. It splits string into tokens, not strings like java version, but you can get the text from those tokens easily.