GitHub - microsoft/BlingFire: A lightning fast Finite State machine and REgular expression manipulation library.

Hi, we are a team at Microsoft called Bling (Beyond Language Understanding), we help Bing be smarter. Here we wanted to share with all of you our FInite State machine and REgular expression manipulation library (FIRE). We use Fire for many linguistic operations inside Bing such as Tokenization, Multi-word expression matching, Unknown word-guessing, Stemming / Lemmatization just to mention a few. B

[braitom]

Bingで使われている自然言語のトークナイザー。

[RabbitBit]

Microsoft BingのTokenizerとして使っているなら、NLTKの10倍の速度と言われても、納得。