Lucene lecture at PisaPackage: org.apache.lucene.analysis An Analyzer is a TokenStream factory. A TokenStream is an iterator over Tokens. input is a character iterator (Reader) A Token is tuple <text, type, start, length, positionIncrement> text (e.g., “pisa”). type (e.g., “word”, “sent”, “para”). start & length offsets, in characters (e.g, <5,4>) positionIncrement (normally 1) standard TokenStream implementations are
