What are the main benefits and drawbacks of learning vocabulary only through conversation?
In a YouTube video from September 2016, the British polyglot Olly Richards explains that he had been learning Cantonese without learning to read and write. He had been following the advice to focus on oral skills first, and he found that it kept the momentum going and that it kept him motivated to learn.
But after a few years, he felt he got stuck at an intermediate plateau, with a limited vocabulary. When learning other languages, breaking through that plateau always involved reading. That's why he decided to start learning Chinese characters. (He also created a number of videos about this new learning project; see the playlist Project: Learn To Write Chinese Characters .)
For many/most languages, learning only oral way would not allow you to learn to read and write. Which might (or might not) be included in your goals. (This is obvious: you might speak and understand, but be illiterate and not able to read and write)
Also, learning by conversation does not allow (a learner of English) to distinguish between homophones, who make mistakes like "cell vs sell", "brake vs break" etc in written form of English.
Also, there is a contested theory that there are multiple learning styles: some people prefer oral, some people prefer visual, or kinetic/tactile inputs.
It seems that research results are not consistent so far: it means that approach works for some people but not for others, like so many other theories from cognitive science.
From personal experience, I found out that I strongly prefer visual learning style, limiting the learning to auditory only would be not using my strongest facility. I am much better able to remember a word if I can associate it with an image, and pronunciation (even recognize the correct pronunciation) if I can see it as IPA.
IOW: I can much better learn the correct pronunciation after I've seen it in IPA, than just by listening alone.
So (as linked Wikipedia page mentions), none of the learning approaches works 100% for everybody. I recommend every person to try and see which of the approaches works best. It is nothing wrong to use method which works best for you personally, even if it is not universal. Build on your own strengths.