Language/Multiple-languages/Culture/Text-Processing-Tools

From Polyglot Club WIKI
Jump to navigation Jump to search
This lesson can still be improved. EDIT IT NOW! & become VIP
Rate this lesson:
5.00
(2 votes)

Multiple-languages-flag-polyglotclub.jpg

In this lesson, several useful linguistic tools useful for common language learners are discussed. They are not always accurate, so keep in mind.

Many of the tools introduced are written in Python, which is an important language in machine learning and easy to learn.

If you don't know Python, please try this:

https://www.youtube.com/watch?v=_uQrJ0TkZlc

In progress.

Diacritisation[edit | edit source]

In Arabic writing system, diacritics indicate the accents, but they are often omitted for writing fluently. The process of restoring diacritics is called diacritisation.

Arabic:

Lemmatisation[edit | edit source]

When you search a word in inflected form, the dictionary program can show you the result as lemma, during which the lemmatisation is done.

multiple languages:

English:

German:

Hungarian:

Persian:

Turkish:

Pitch-Accent Marking[edit | edit source]

In Japanese and other languages, the pitch-accent is important on distinguishing different words. They are unwritten and required.

Japanese:

Stress Marking[edit | edit source]

In Russian and other languages, the stress is important on distinguishing different words. They are usually omitted.

Russian:

Transcription[edit | edit source]

Some languages are written in more than one writing systems. This tool converts them from one to another.

Chinese:

Part of Speech Tagging[edit | edit source]

It tags words in the sentence with parts of speech. Some of them can draw parse trees.

Multiple languages:

Arabic:

Chinese:

Japanese:

Thai:

Vietnamese:

Romanisation[edit | edit source]

multiple languages:

Iranian Persian:

Japanese:

Korean:

Mandarin Chinese:

Standard Arabic:

Thai:

Word Segmentation[edit | edit source]

In some languages, words are not separated by spaces, for example: Chinese, Japanese, Khmer, Lao, Thai. In Vietnamese, spaces are used to divide syllables instead of words. This brings about difficulties for computer programs like VocabHunter, gritz and text-memorize, where words are detected only with spaces.

The solution is called “word segmentation”, which detects words and insert spaces in between or put the segmented words into a list.

Burmese, Khmer, Lao, Thai:

Burmese:

Chinese:

Japanese:

Lao:

Thai:

Vietnamese:

Other Lessons[edit | edit source]

Contributors

GrimPixel and Maintenance script


Create a new Lesson