For many/most languages, learning only oral way would not allow you to learn to read and write. Which might (or might not) be included in your goals. (This is obvious: you might speak and understand, but be illiterate and not able to read and write)
Also, learning by conversation does not allow (a learner of English) to distinguish between homophones, who make mistakes like "cell vs sell", "brake vs break" etc in written form of English.
Also, there is a contested theory that there are multiple learning styles: some people prefer oral, some people prefer visual, or kinetic/tactile inputs.
It seems that research results are not consistent so far: it means that approach works for some people but not for others, like so many other theories from cognitive science.
From personal experience, I found out that I strongly prefer visual learning style, limiting the learning to auditory only would be not using my strongest facility. I am much better able to remember a word if I can associate it with an image, and pronunciation (even recognize the correct pronunciation) if I can see it as IPA.
IOW: I can much better learn the correct pronunciation after I've seen it in IPA, than just by listening alone.
So (as linked Wikipedia page mentions), none of the learning approaches works 100% for everybody. I recommend every person to try and see which of the approaches works best. It is nothing wrong to use method which works best for you personally, even if it is not universal. Build on your own strengths.