I'm creating a web service for learning words and I need to not make a user to see each English word, for example if you know the word "table" you most likely know words "apple", "door", "hi" and so on. Why? Because they are common and appeared frequently. Also it works for singular and plural, if you know "car", you know "cars" as well, but for the word "mouse" it's not so obvious that you know "mice". There are many such cases, like special terms and so on, I just need a database with the following structure:

word1 word2 prediction

prediction is the value from 0 to 1 showing how likely it is that you know word2 if know word1.

Is there any free database or API? For now I'm looking only for English language, but it would be great if there are for others.

  • 1
    I don't know if this is on-topic or not because it seems to be about the development of software that assists language learning rather than the software itself – Anthony Pham Apr 4 '18 at 22:23
  • 1
    @AnthonyPham Yes, it's about resources required for a specific type of software, but I don't see how the required resource(s) would be used for a different purpose, so I would consider the question on topic. – IkWeetHetOokNiet Apr 5 '18 at 15:02
  • I think it is unlikely that you will find an API or database with this information, let alone a free one. In principle, one could analyze a number of written texts, from the basic reading level to highly technical scientific papers, and calculate "correlation coefficients" between words. In the simplest form, this could be a symmetric relationship, so prediction(word1,word2) = prediction(word2,word1). This kind of text analysis is big nowadays, but you're asking for a fairly specific algorithm and the only people with accurate answers to these questions are big tech companies. – Myridium Apr 6 '18 at 4:16
  • @ChristopheStrobbe - this resource could be used for marketing purposes, to appeal to cliques by using their lingo. This question is asking for big data, and in addition, how to process that data. Nothing specific to the actual learning process of English at all, except for an incidental use case. I believe it is off-topic, and belongs on the data science stack exchange. – Myridium Apr 7 '18 at 1:41
  • 1
    @Myridium The fact that the same resource can also be used for purposes unrelated to language learning does not make the question off topic. For example, spaced repetition can also be used for other purposes than language learning, but that does not make spaced repetition off topic. – IkWeetHetOokNiet Apr 7 '18 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.