mode The tokenization mode determines how the tokenizer handles compound and unknown words. It can be set to: normal Normal segmentation, no decomposition for compounds. Example output: 関西国際空港 アブラカダブラ search Segmentation geared towards search. This includes a decompounding process for long nouns, also including the full compound token as a synonym. Example output: 関西, 関西国際空港, 国際, 空港 アブラカダブラ extend