Automatically create a list of new vocabulary from an eBook

Question

I thought of an idea for a software that I would find useful as a foreign language learner, and I am wondering if something similar already exists. The software would (more-or-less) automatically create a list of unknown vocabulary from an eBook. The user could then study this new vocabulary before reading the book. Schematically the process would look something like:

Initial Setup

1. User adds words to their 'Known Vocab' list (e.g. from an Anki deck)
2. User chooses how many of the languages most common words to link with their Known Vocab (can be changed to taste at any stage later on)

Typical Use

1. User opens an eBook with the software
2. The software creates a list of all distinct words (perhaps with some clever filtering to avoid annoying duplicates like 'cat' and 'cats', or 'eat' and 'eating' etc.) in the book, sorted by frequency, and excluding words in the Known Vocab
3. The user can then export this list (perhaps with some further filtering and manual exclusions) for use in things like Anki
4. The user can also update their Known Vocab with this list

Answer 1

I do not know of any existing software; however, I do something similar to what you have described using a selection of open-source software running on a Linux platform. I have not automated the process.

I typically start with either an epub publication or something in PDF format. If the publication is an epub, I use Calibre to convert the document to text. If the publication is in PDF format, I use run:

pdftotext original.pdf text-version-of-original.txt

I then create a list of words from text-version-of-original.txt using:

fmt -1 text-version-of-original.txt

sort it, and strip out duplicates. I typically run the commands as a long pipe:

fmt -1 text-version-of-original.txt | sort | uniq > list-of-unique-words.csv

The only reason for having a csv extension on the list of unique words is because Anki expects it. I then edit the csv file into a form that conforms to my existing Anki deck, and import it into the deck. The process works well, but it could be improved. In particular, rather than having the words sorted into alphabetically order, it would be useful to retain them, without duplicates, in the order in which they first appeared in the original publication. That way, rather than trying to learn all the words, without context, prior to reading the book, I could progress through the book and the Anki deck at a matched pace!

Added later: The process of answering your question spurred me into trying to solve the problem that I described in the last paragraph above ... how to eliminate the duplicates from the file text-version-of-original.txt, while retaining the original ordering of new words. As usual with most things GNU-Linux oriented, the solution is remarkably simple. I should have checked years ago. Instead of using the UNIX-pipe I have shown above, use this instead:

fmt -1 text-version-of-original.txt | cat -n | sort -uk2 | sort -nk1 | cut -f2- > list-of-unique-words.csv

Finally, having imported the file list-of-unique-words.csv into Anki, ensure that the Anki preferences for the deck are set to present new cards in the order of their appearance, and not in random order

Answer 2

This is concordancing. There is a freeware AntConc can do this. However, it is intended for corpus analysis so the usage may not be quite intuitive.