It should increase the difficulty of a language slightly (in fact, I speak Tamil as an L2, and didn't actually even realize it had a case system until I found your question). Using the Wikipedia list of language with a case system and the FSI's ranking of the easiest languages to learn (which accounts for the difficulty of the case system), we can see that languages with a case system are in all four categories of difficulty, although they do tend to be in the harder categories. For example, Romanian and German have a case system and both are considered fairly easy to learn for English speakers. Most of the other languages with case systems, like Greek, Russian and Polish, are in Category 3, meaning they should take quite a bit of time and effort to become fluent in. Finally, both Japanese and Korean are in Cateogry 4, meaning they will require an exceptional amount of dedication to speak.
Case system seems to increase the difficulty of a language for an English speaker (partly because we ourselves don't have a case system), but other factors of the language are more important in affecting the difficulty than case system.