Difference between revisions of "Language/Multiple-languages/Culture/Text-Processing-Tools"

From Polyglot Club WIKI
Jump to navigation Jump to search
m (Quick edit)
m (Quick edit)
Line 14: Line 14:
== Diacritisation ==
== Diacritisation ==
In Arabic writing system, diacritics indicate the accents, but they are often omitted for writing fluently. The process of restoring diacritics is called diacritisation.
In Arabic writing system, diacritics indicate the accents, but they are often omitted for writing fluently. The process of restoring diacritics is called diacritisation.


Arabic:
Arabic:
Line 26: Line 25:
== Lemmatisation ==
== Lemmatisation ==
When you search a word in inflected form, the dictionary program can show you the result as lemma, during which the lemmatisation is done.
When you search a word in inflected form, the dictionary program can show you the result as lemma, during which the lemmatisation is done.


Multiple languages:
Multiple languages:
Line 36: Line 34:
== Pitch-Accent Marking ==
== Pitch-Accent Marking ==
In Japanese and other languages, the pitch-accent is important on distinguishing different words. They are unwritten and required.
In Japanese and other languages, the pitch-accent is important on distinguishing different words. They are unwritten and required.


Japanese:
Japanese:
Line 44: Line 41:
== Stress Marking ==
== Stress Marking ==
In Russian and other languages, the stress is important on distinguishing different words. They are usually omitted.
In Russian and other languages, the stress is important on distinguishing different words. They are usually omitted.


Russian:
Russian:
Line 52: Line 48:
== Transcription ==
== Transcription ==
Some languages are written in more than one writing systems. This tool converts them from one to another.
Some languages are written in more than one writing systems. This tool converts them from one to another.


Chinese:
Chinese:
Line 62: Line 57:
== Part of Speech Tagging ==
== Part of Speech Tagging ==
It tags words in the sentence with parts of speech. Some of them can draw parse trees.
It tags words in the sentence with parts of speech. Some of them can draw parse trees.


Multiple languages:
Multiple languages:
Line 107: Line 101:


The solution is called “[https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation word segmentation]”, which detects words and insert spaces in between or put the segmented words into a list.
The solution is called “[https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation word segmentation]”, which detects words and insert spaces in between or put the segmented words into a list.


Chinese:
Chinese:
Line 151: Line 144:
* VnCoreNLP https://github.com/vncorenlp/VnCoreNLP
* VnCoreNLP https://github.com/vncorenlp/VnCoreNLP


==Related Lessons==
==Other Lessons==
* [[Language/Multiple-languages/Culture/Internet-Dictionaries|Internet Dictionaries]]
* [[Language/Multiple-languages/Culture/Internet-Dictionaries|Internet Dictionaries]]
* [[Language/Multiple-languages/Culture/Astrology-in-different-Cultures-and-Languages|Astrology in different Cultures and Languages]]
* [[Language/Multiple-languages/Culture/Astrology-in-different-Cultures-and-Languages|Astrology in different Cultures and Languages]]
Line 162: Line 155:
* [[Language/Multiple-languages/Culture/Helpful-Anki-Shared-Decks|Helpful Anki Shared Decks]]
* [[Language/Multiple-languages/Culture/Helpful-Anki-Shared-Decks|Helpful Anki Shared Decks]]
* [[Language/Multiple-languages/Culture/Internet-resources-for-learning-specific-languages|Internet resources for learning specific languages]]
* [[Language/Multiple-languages/Culture/Internet-resources-for-learning-specific-languages|Internet resources for learning specific languages]]
<span links></span>

Revision as of 11:03, 27 March 2023

Multiple-languages-flag-polyglotclub.jpg

In this lesson, several useful linguistic tools useful for common language learners are discussed. They are not always accurate, so keep in mind.

Many of the tools introduced are written in Python, which is an important language in machine learning and easy to learn.

If you don't know Python, please try this:

In progress.

Diacritisation

In Arabic writing system, diacritics indicate the accents, but they are often omitted for writing fluently. The process of restoring diacritics is called diacritisation.

Arabic:

Lemmatisation

When you search a word in inflected form, the dictionary program can show you the result as lemma, during which the lemmatisation is done.

Multiple languages:

Pitch-Accent Marking

In Japanese and other languages, the pitch-accent is important on distinguishing different words. They are unwritten and required.

Japanese:

Stress Marking

In Russian and other languages, the stress is important on distinguishing different words. They are usually omitted.

Russian:

Transcription

Some languages are written in more than one writing systems. This tool converts them from one to another.

Chinese:

Part of Speech Tagging

It tags words in the sentence with parts of speech. Some of them can draw parse trees.

Multiple languages:

Arabic:

Chinese:

Japanese:

Thai:

Vietnamese:

Word Segmentation

In some languages, words are not separated by spaces, for example: Chinese, Japanese, Lao, Thai. In Vietnamese, spaces are used to divide syllables instead of words. This brings about difficulties for computer programs like VocabHunter, gritz and text-memorize, where words are detected only with spaces.

The solution is called “word segmentation”, which detects words and insert spaces in between or put the segmented words into a list.

Chinese:

Japanese:

Lao:

Thai:

Vietnamese:

Other Lessons

Retrieved from "https://polyglotclub.com/wiki/index.php?title=Language/Multiple-languages/Culture/Text-Processing-Tools&oldid=196100"