Editing Language/Multiple-languages/Culture/Producing-dictionaries-with-web-scraping

{{Anki-menu}}
[[Category:Computer-Knowledge]]
{{Multiple-languages-flag}}
Have you ever tried this: get a frequency list, copy and paste the entries from a dictionary website to make a flashcard deck? And have you thought about making this process automatic?

There is a thing called “[https://en.wikipedia.org/wiki/Web_scraping web scraping]” that does exactly such work. And the most fascinating part is that '''<big>it might be illegal</big>'''.

A dictionary is a creative work and is protected by copyright laws, unless its author died long enough so that the work enters public domain in your country. References under certain conditions, in a small scale, can be considered as “[https://en.wikipedia.org/wiki/Fair_use fair use]”. Apart from that, if the automatic data collection process generates too many requests in a short time, it would affect the web server's functionality, so some websites explicitly prohibit web scraping in their terms of service.

<b>You are the only one responsible for the consequences of your own web scraping activity.</b>

OK, disclaimer done. Reality:

Unless you are for commercial purpose, most people won't sue you for scraping because there's no profit from that. They'll just block your IP address or do something to exterminate possibilities of scraping there, if you have made real trouble. Apart from that, if your frequency of request is low enough, the website server won't feel uncomfortable. Furthermore, even if the dictionary is protected by copyright, as long as you don't distribute, no one would sue you for copyright infringement.

How slow is good? A quest every 10 seconds won't be a problem. Maybe 150 quests per day, unless you can learn more.

The principle of web scrapping is to read its webpage code, find where the corresponding contents locate by its tags, then look up a list of words on different pages at the same locations.

If you want to try, there is a [https://polyglotclub.com/wiki/Language/Multiple-languages/Culture/Internet-Dictionaries dictionary list]. Just pick a website that claims to have made their content under whatsoever public license and provides no download link. Some websites provide API, good job.

Here is a guy displaying the simple trick with Python (but the code is for old Python versions): 

<youtube>-Yx8q6aKgtw</youtube>

And he's not alone: 

<youtube>zcszMBH6Lvc</youtube>

Someone did it with PHP:

<youtube>VEqlj124Fjo</youtube>

You need to modify the code so that it can automatically process a wordlist and generate a file. If you don't know Python, please try this: 

<youtube>_uQrJ0TkZlc</youtube>

Have fun!
<span links></span>