Editing Language/Multiple-languages/Culture/How-to-make-a-TSV-file
Jump to navigation
Jump to search
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
Tab-separated values (TSV) file is a simple format for storing data in a tabular structure. | |||
You may have visited [[Language/Multiple-languages/Culture/Internet-Dictionaries] | You may have visited [[Language/Multiple-languages/Culture/Internet-Dictionaries]] and want to utilise some downloadable stuff by creating flashcards on [https://apps.ankiweb.net/ Anki] or [https://mnemosyne-proj.org/ Mnemosyne]. But it requires too much effort if you copy-paste entry by entry. If we can use a spreadsheet, things will be much easier. Can we do that? | ||
You may have noticed that both programs have “File -> Import...” option. But they don't support XLS or XLSX files. What should you do? | You may have noticed that both programs have “File -> Import...” option. But they don't support XLS or XLSX files. What should you do? | ||
Line 11: | Line 8: | ||
If you have opened a spreadsheet program (e.g. [https://www.libreoffice.org/ LibreOffice] Calc, [https://www.openoffice.org/ Apache OpenOffice] Calc, [https://www.onlyoffice.com/ ONLYOFFICE] Spreadsheet Editor, [https://www.office.com/ Microsoft Office] Excel) and click on “File -> Save As...”, you can see some other formats to choose from, one of which is “CSV”. | If you have opened a spreadsheet program (e.g. [https://www.libreoffice.org/ LibreOffice] Calc, [https://www.openoffice.org/ Apache OpenOffice] Calc, [https://www.onlyoffice.com/ ONLYOFFICE] Spreadsheet Editor, [https://www.office.com/ Microsoft Office] Excel) and click on “File -> Save As...”, you can see some other formats to choose from, one of which is “CSV”. | ||
“[ | “[[wikipedia:Comma-separated_values|CSV]]” means “Comma-separated values”. It uses commas to separate columns. If you have a comma in the text, then it uses quotation marks to quote your text, so the comma won't be counted as a column separator. If you have quotation marks in your text, then the [[wikipedia:Escape_character|escape character]] backslash “\” need to be placed before the quotation mark. This is an example: https://github.com/skywind3000/ECDICT/blob/master/ecdict.mini.csv. | ||
You may have realised that a CSV file doesn't store any styling data. If you save as a CSV file, all the information about fonts, colours, hyperlinks, etc. will be lost. CSV files are lightweight, so when you just need pure data, this format is ideal. Do Anki and Mnemosyne support it? | You may have realised that a CSV file doesn't store any styling data. If you save as a CSV file, all the information about fonts, colours, hyperlinks, etc. will be lost. CSV files are lightweight, so when you just need pure data, this format is ideal. Do Anki and Mnemosyne support it? | ||
Not seems to be so, but its sibling TSV is supported. In Anki, it is called “Text separated by tabs or semicolons”; in Mnemosyne, it is called “Tab-separated text files”. What is it? | Not seems to be so, but its sibling TSV is supported. In Anki, it is called “Text separated by tabs or semicolons”; in Mnemosyne, it is called “Tab-separated text files”. What is it? | ||
“[ | “[[wikipedia:Tab-separated_values|TSV]]” means “Tab-separated values”. It is similar with CSV and has an advantage over CSV: it uses “tabs” to separate columns, so there is no need to use quotation marks to indicate commas as text instead of column separators. Both “TSV” and “CSV” belong to “[[wikipedia:Delimiter-separated_values|DSV]]”, delimiter-separated values. | ||
You may wonder what a “tab” means. [ | You may wonder what a “tab” means. [[wikipedia:Tab_key|The tabular key]] is the key above the “Caps Lock” key on your keyboard (in most cases). It is used for aligning text in different lines to make it easier to form a table for typewriters and is inherited by computers. If you press this key in a text editor, it will look like some spaces; if you press this key in a browser, it will move the focus to the next element (link, textbox, button, etc.). In a spreadsheet program, you can press the Tab key to move to the next column or the Enter key to move to the next row. TSV files also use these two keys to separate columns and rows. It is more ideal than CSV. This is an example: https://www.eki.ee/litsents/vaba/ies/eestiinglise.txt. | ||
How to save as TSV file? This is a bit confusing, because TSV is not so well-known as CSV. If you are using LibreOffice, click on that “Save As...”, select CSV, then in the dialogue box, choose {Tab} as “Field delimiter” and ignore “String delimiter” | How to save as TSV file? This is a bit confusing, because TSV is not so well-known as CSV. If you are using LibreOffice, click on that “Save As...”, select CSV, then in the dialogue box, choose {Tab} as “Field delimiter” and ignore “String delimiter” (if you have neutral quotation marks ⟨"…"⟩ in the text, they will be converted to typographic quotation marks ⟨“…”⟩). The file you save has “CSV” as its file extension, but it's a TSV file essentially. | ||
== How to convert to TSV == | == How to convert to TSV == | ||
=== Formats === | === Formats === | ||
==== Spreadsheet formats ==== | ==== Spreadsheet formats ==== | ||
Line 47: | Line 36: | ||
===== Different delimiters ===== | ===== Different delimiters ===== | ||
In this case, you need to use a text editor | In this case, you need to use a text editor like Notepad on Windows instead of spreadsheet program. | ||
==== Other formats ==== | ==== Other formats ==== | ||
===== [https://www.sqlite.org/fileformat.html DB] ===== | ===== [https://www.sqlite.org/fileformat.html DB] ===== | ||
You need open-source tools [https://sqlitebrowser.org/ DB Browser for SQLite], [https://sqlitestudio.pl/ SQLiteStudio] or something else to open it. See which tables it contains, export tables you need, make sure “Field separator” is “Tab”. | |||
===== [https://en.wikipedia.org/wiki/DICT DICT/DICT.DZ] ===== | ===== [https://en.wikipedia.org/wiki/DICT DICT/DICT.DZ] ===== | ||
DICT is compressed as DICT.DZ, which is [https://en.wikipedia.org/wiki/Gzip gzip] in essence. Open-source tools [https://github.com/ib/xarchiver XAchiver] for Linux and BSD, [https://www.7-zip.org/ 7-Zip] for Windows can open DICT.DZ files without changing the file extension. DICT files can be edited with a text editor. | DICT is compressed as DICT.DZ, which is [https://en.wikipedia.org/wiki/Gzip gzip] in essence. Open-source tools [https://github.com/ib/xarchiver XAchiver] for Linux and BSD, [https://www.7-zip.org/ 7-Zip] for Windows can open DICT.DZ files without changing the file extension to GZ. DICT files can be edited with a text editor. | ||
===== [https://github.com/itkach/slob SLOB] and some others ===== | |||
It is a format designed for [http://aarddict.org/ Aard 2]. Open-source tool [https://github.com/ilius/pyglossary PyGlossary] can do. | |||
===== [https://en.wikipedia.org/wiki/Xml XML] and some others ===== | |||
Outdated format with incredible popularity. | |||
This sort of conversion is a bit different, because XML, JSON and the rising star [https://yaml.org/ YAML] are not in the form of sheet. They are hierarchical, tree-like. You can find a list of tools at [https://github.com/dbohdan/structured-text-tools Structured text tools], convert to JSON, then TSV. | |||
= | [https://www.youtube.com/watch?v=shc3CFOKp-0 A video about how to import XML into LibreOffice Calc]. | ||
=== Skill training === | |||
It requires the skill of combined use of text editor. You need to look through the file, discover its patterns and differentiate contents through the patterns. It's similar to chemistry. | |||
In a text editor's “replace” function, “Tab” can be represented with “\t”, “Enter” can be represented with “\n” on Linux, BSD, macOS, Solaris, or “\r\n” on Windows. “t” means “tab”, while “r” and “n” mean “return” and “newline” respectively (not to be mentioned in the examples below). The backslash is an “[https://en.wikipedia.org/wiki/Escape_character escape character]”. Further reading on [https://en.wikipedia.org/wiki/Newline newline] and [https://en.wikipedia.org/wiki/Carriage_return return]. What's more, you can use something called “[https://en.wikipedia.org/wiki/Regular_expression regular expression]”, which is complex and powerful. You can visit [https://regexone.com/ this tutorial] and [https://www.regular-expressions.info/ this tutorial] and [https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285 this page]. | |||
If you find [https://en.wikipedia.org/wiki/Mojibake mojibake] in your text editor, you probably need to open the text editor and click on “open file” in it, instead of directly double-clicking on the file; in the encoding selection, try | If you find [https://en.wikipedia.org/wiki/Mojibake mojibake] in your text editor, you probably need to open the text editor and click on “open file” in it, instead of directly double-clicking on the file; in the encoding selection, try some possible [https://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings encodings]. | ||
Several operations require waiting because of the large amount of character replacements. | |||
Please try to solve the problems yourself. See the steps as late as possible. | Please try to solve the problems yourself. See the steps as late as possible. | ||
==== Example 1: [https://freedict.org/downloads/#dictionary-downloads | ==== Example 1: [https://freedict.org/downloads/#dictionary-downloads English-French FreeDict Dictionary] ==== | ||
{| class="wikitable" | {| class="wikitable" | ||
|a /ə/ | |a /ə/ | ||
Line 94: | Line 93: | ||
3. renoncer, résigner | 3. renoncer, résigner | ||
|} | |} | ||
Steps for putting multiple lines into one: | |||
Steps: | # Open with a text editor; | ||
# | # Delete useless information; | ||
# Replace “\n2.” with “; 2.”; | # Replace “\n2.” with “; 2.”; | ||
# Replace “\n3.” with “; 3.”; | # Replace “\n3.” with “; 3.”; | ||
# Increase the number until | # Increase the number until such string not found; | ||
# Replace “/\n1.” with “/\t1.” | # Replace “/\n1.” with “/\t1.”. | ||
# Replace “ /” with “\t/”; | # Replace “ /” with “\t/”; | ||
# Save. | |||
In step 5, you can actually use “/\n” or “\n1.”, but the one shown has a better reliability. | |||
Final result: | |||
{| class="wikitable" | {| class="wikitable" | ||
! | ! | ||
Line 113: | Line 114: | ||
|a | |a | ||
|/ə/ | |/ə/ | ||
|1. à, au milie de, en, dans, parmi; 2. un, quelqu'un; 3. àraisonde, par | |1. à, au milie de, en, dans, parmi; 2. un, quelqu'un; 3. àraisonde, par 4. une | ||
|- | |- | ||
!'''2''' | !'''2''' | ||
Line 126: | Line 127: | ||
|} | |} | ||
==== Example 2: [https://www. | ==== Example 2: [https://www.edrdg.org/jmdict/edict_doc.html JMdict] ==== | ||
{| class="wikitable" | {| class="wikitable" | ||
|一 一 [yi1] /one/single/a (article)/as soon as/entire | |あああ;あーあ;あーー;アアア;アーア;アーー /(int) (expression of despair, resignation, boredom, disgust, etc.) (See 嗚呼・ああ・1) aah!/ooh!/oh no!/oh boy!/EntL2205270X/ | ||
ああいう(P);ああゆう /(exp,adj-pn) that sort of/like that/(P)/EntL2085090X/ | |||
ああいう風に [ああいうふうに] /(exp) (uk) in that way/like that/EntL2424550X/ | |||
|} | |||
Steps: | |||
# Open with a text editor (select Japanese (EUC-JP) as encoding); | |||
# Delete useless information (the first line); | |||
# Replace “ /” with “\ /”; | |||
# Replace “]\” with “]”; | |||
# Replace “\ /” with “ [] /”; | |||
# Replace “] /” with “\t”; | |||
# Replace “ [” with “\t”; | |||
# Select “Regular Expression”, replace “/EntL.+” with “”; | |||
# Save. | |||
Explanation: | |||
Step 3, 4, 5 add a backslash to all entries and delete them in entries with square parenthesis, then replace the backslashes with square parentheses. | |||
Step 8 deletes “/EntL” and its following characters in the line. “.+” in regular expression means any character (at least one) until newline. | |||
Final Result: | |||
{| class="wikitable" | |||
! | |||
!'''A''' | |||
!'''B''' | |||
!'''C''' | |||
|- | |||
!'''1''' | |||
|あああ;あーあ;あーー;アアア;アーア;アーー | |||
| | |||
|(int) (expression of despair, resignation, boredom, disgust, etc.) (See 嗚呼・ああ・1) aah!/ooh!/oh no!/oh boy! | |||
|- | |||
!'''2''' | |||
|ああいう(P);ああゆう | |||
| | |||
|(exp,adj-pn) that sort of/like that/(P) | |||
|- | |||
!'''3''' | |||
|ああいう風に | |||
|ああいうふうに | |||
|(exp) (uk) in that way/like that | |||
|} | |||
==== Example 4: [https://www.mdbg.net/chinese/dictionary?page=cc-cedict CC-CEDICT] ==== | |||
{| class="wikitable" | |||
|一 一 [yi1] /one/1/single/a (article)/as soon as/entire/whole/all/throughout/"one" radical in Chinese characters (Kangxi radical 1)/also pr. [yao1] for greater clarity when spelling out numbers digit by digit/ | |||
一一 一一 [yi1 yi1] /one by one/one after another/ | 一一 一一 [yi1 yi1] /one by one/one after another/ | ||
Line 134: | Line 181: | ||
|} | |} | ||
Steps: | Steps: | ||
# | # Open with a text editor; | ||
# | # Delete useless information (lines starting with hash); | ||
# | # Replace “pr. [” with “pr.[”; | ||
# Replace “ [” with “\t”; | # Replace “ [” with “\t”; | ||
# Replace “pr. | # Replace “pr.[” with “pr. [”; | ||
# Replace “] /” with “\t”; | # Replace “] /” with “\t”; | ||
# Replace “/\n” with “\n”; | # Replace “/\n” with “\n”; | ||
# Delete the last “/” in the file; | |||
# Save; | |||
# Open with a spreadsheet program; | |||
# Add a column between column A and B; | |||
# Select column A; | |||
# Open “Text to Columns”, select “space” in “Separated by”, click on “OK”; | |||
# Save. | |||
Explanation: | |||
Most of the customised field separators are unified to Tab in the first edition. Step 3, 5 guarantee that the square parenthesis in “meaning” field won't be affected, as that “pr. [” is a universal pattern throughout the whole file. | |||
Result: | Result by step 10: | ||
{| class="wikitable" | |||
! | |||
!'''A''' | |||
!B | |||
!'''C''' | |||
|- | |||
!'''1''' | |||
|一 一 | |||
|yi1 | |||
|one/1/single/a (article)/as soon as/entire/whole/all/throughout/"one" radical in Chinese characters (Kangxi radical 1)/also pr. [yao1] for greater clarity when spelling out numbers digit by digit | |||
|- | |||
!'''2''' | |||
|一一 一一 | |||
|yi1 yi1 | |||
|one by one/one after another | |||
|- | |||
!'''3''' | |||
|一一對應 一一对应 | |||
|yi1 yi1 dui4 ying4 | |||
|one-to-one correspondence | |||
|} | |||
Final Result: | |||
{| class="wikitable" | {| class="wikitable" | ||
! | ! | ||
Line 154: | Line 233: | ||
|一 | |一 | ||
|yi1 | |yi1 | ||
|one/single/a (article)/as soon as/entire | |one/1/single/a (article)/as soon as/entire/whole/all/throughout/"one" radical in Chinese characters (Kangxi radical 1)/also pr. [yao1] for greater clarity when spelling out numbers digit by digit | ||
|- | |- | ||
!'''2''' | !'''2''' | ||
Line 171: | Line 250: | ||
== How to combine data with same column from two spreadsheets == | == How to combine data with same column from two spreadsheets == | ||
You have a dictionary file in spreadsheet format, but it has so many entries that you don't want to memorise them all. Then you get a list of common words in spreadsheet format. How to combine them? | You have a dictionary file in spreadsheet format, but it has so many entries that you don't want to memorise them all. Then you get a list of common words in spreadsheet format. How to combine them? | ||
Here's [https://www.youtube.com/watch?v=VmanL-Vf8Eg a guidance for Microsoft Office Excel]. | |||
For LibreOffice Calc and other spreadsheet programs, you need to put the two sheets alongside, then use VLOOKUP (vertical lookup). | For LibreOffice Calc and other spreadsheet programs, you need to put the two sheets alongside, then use VLOOKUP (vertical lookup). | ||
Line 187: | Line 268: | ||
|frequency | |frequency | ||
|word | |word | ||
|meaning | |''meaning'' | ||
| | | | ||
|word | |word | ||
Line 244: | Line 325: | ||
|frequency | |frequency | ||
|word | |word | ||
|meaning | |''meaning'' | ||
| | | | ||
|word | |word | ||
Line 282: | Line 363: | ||
|} | |} | ||
Then you need to copy or cut column C, then paste special (shortcut Shift+Ctrl+V), make sure “formula” is unchecked, click on “OK”. Delete column E and F, save as TSV. | Then you need to copy or cut column C, then paste special (shortcut Shift+Ctrl+V), make sure “formula” is unchecked, click on “OK”. Delete column E and F, save as TSV. | ||