Difference between revisions of "Language/Multiple-languages/Culture/Text-Processing-Tools"

From Polyglot Club WIKI
Jump to navigation Jump to search
(Created page with "Category:Computer-Knowledge In some languages, words are not separated by spaces, for example: Chinese, Japanese, Lao, Thai. This brings about difficulties for computer p...")
 
Line 3: Line 3:
In some languages, words are not separated by spaces, for example: Chinese, Japanese, Lao, Thai. This brings about difficulties for computer programs like [https://vocabhunter.github.io/ VocabHunter], [https://github.com/jeffkowalski/gritz gritz] and [https://github.com/zg/text-memorize text-memorize], where words are detected only with spaces.
In some languages, words are not separated by spaces, for example: Chinese, Japanese, Lao, Thai. This brings about difficulties for computer programs like [https://vocabhunter.github.io/ VocabHunter], [https://github.com/jeffkowalski/gritz gritz] and [https://github.com/zg/text-memorize text-memorize], where words are detected only with spaces.


The solution is called “[https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation word segmentation]”, which detects words and insert spaces in between. Sounds like easy, but they have to deal with ambiguities and unknown words, the time and accuracy are both to be considered.
The solution is called “[https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation word segmentation]”, which detects words and insert spaces in between. Sounds like easy, but they have to deal with ambiguities and unknown words including proper names, the time and accuracy are both to be considered. It can be based on the dictionary or machine learning.


Here are free and open-source tools to do it:
Here are free and open-source tools to do it:


Chinese:
Chinese:
* Ansj https://github.com/NLPchina/ansj_seg
* CoreNLP https://github.com/stanfordnlp/CoreNLP
* CoreNLP https://github.com/stanfordnlp/CoreNLP
* FoolNLTK https://github.com/rockyzhengwu/FoolNLTK
* FoolNLTK https://github.com/rockyzhengwu/FoolNLTK

Revision as of 13:49, 20 May 2021


In some languages, words are not separated by spaces, for example: Chinese, Japanese, Lao, Thai. This brings about difficulties for computer programs like VocabHunter, gritz and text-memorize, where words are detected only with spaces.

The solution is called “word segmentation”, which detects words and insert spaces in between. Sounds like easy, but they have to deal with ambiguities and unknown words including proper names, the time and accuracy are both to be considered. It can be based on the dictionary or machine learning.

Here are free and open-source tools to do it:

Chinese:

Japanese:

Thai:


If you don't know Python, please try this:

Retrieved from "https://polyglotclub.com/wiki/index.php?title=Language/Multiple-languages/Culture/Text-Processing-Tools&oldid=94606"