Language/Multiple-languages/Culture/Producing-dictionaries-with-web-scraping

From Polyglot Club WIKI
Jump to navigation Jump to search
This lesson can still be improved. EDIT IT NOW! & become VIP
Rate this lesson:
5.00
(one vote)

Shared Decks Add-ons External Programs
Beginner Tutorial TSV Creation Web Scrapping Markup Studying
Multiple-languages-flag-polyglotclub.jpg

Have you ever tried this: get a frequency list, copy and paste the entries from a dictionary website to make a flashcard deck? And have you thought about making this process automatic?

There is a thing called “web scraping” that does exactly such work. And the most fascinating part is that it might be illegal.

A dictionary is a creative work and is protected by copyright laws, unless its author died long enough so that the work enters public domain in your country. References under certain conditions, in a small scale, can be considered as “fair use”. Apart from that, if the automatic data collection process generates too many requests in a short time, it would affect the web server's functionality, so some websites explicitly prohibit web scraping in their terms of service.

You are the only one responsible for the consequences of your own web scraping activity.

OK, disclaimer done. Reality:

Unless you are for commercial purpose, most people won't sue you for scraping because there's no profit from that. They'll just block your IP address or do something to exterminate possibilities of scraping there, if you have made real trouble. Apart from that, if your frequency of request is low enough, the website server won't feel uncomfortable. Furthermore, even if the dictionary is protected by copyright, as long as you don't distribute, no one would sue you for copyright infringement.

How slow is good? A quest every 10 seconds won't be a problem. Maybe 150 quests per day, unless you can learn more.

The principle of web scrapping is to read its webpage code, find where the corresponding contents locate by their tags, then look up a list of entries on different pages at the same locations and write them down.

If you want to try, there is a dictionary list. Just pick a website that claims to have made their content under whatsoever public license and provides no download link. Some websites provide API, good job.

Here is a guy displaying the simple trick with Python (but the code is for old Python versions):

And he's not alone:

Someone did it with PHP:

You need to modify the code so that it can automatically process a wordlist and generate a file. If you don't know Python, please try this:

Have fun!

Contributors

GrimPixel, Maintenance script and Vincent


Create a new Lesson