Answer (1 of 3): Paul McCann's answer is very good, but to put it more simply, there are two major methods for Japanese tokenization (which is often also called "Morphological Analysis"). * Dictionary-based sequence-prediction methods: Make a dictionary of words with parts of speech, and find th...
![What are some Japanese tokenizers or tokenization strategies?](https://cdn-ak-scissors.b.st-hatena.com/image/square/0254a35f9fba5c3353d62b7e014d0aa594f4cc4e/height=288;version=1;width=512/https%3A%2F%2Fqph.cf2.quoracdn.net%2Fmain-custom-t-980-600x315-sqxxospfciejqkjfytssmfbdcvzbbxla.jpeg)