Firstly, it does seem that some languages might be inherently more difficult than other languages to learn in the classroom. This is because many languages have large amounts of irregulars, exceptions to grammar rules, and things like that. Some languages also have larger useful lexicons than others, but that seems pretty marginal.
However, given that babies tend to have the neuroplasticity necessary to learn any language, the difficulty really comes out after that skill has been dulled by age. In these cases, it seems much more likely that "difficulty" is related more to what languages the person knew before. For example, the reason that many accents emerge from non-native speakers is because, among other things, people who learn languages later in life have to filter pronunciation through the systems of speaking that they learned when they were children (and in some cases the shapes of their palates and vocal chords also affect this). In these cases, it is probably easier to learn a Romance language if you know one already, but it would be much more difficult to switch to German, or a tonal language like Mandarin.
It seems as though children can pick up on most languages pretty easily, given the way their brains are prepped for language. Noam Chomsky writes a lot about biolinguistics in the context of this linguistic theory.
The only notable exception to this is if a created language doesn't follow a clear grammatical system that Chomsky outlines. For example
"Creoles are languages that develop and form when disparate societies come together and are forced to devise a new system of communication. The system used by the original speakers is typically an inconsistent mix of vocabulary items, known as a pidgin. As these speakers' children begin to acquire their first language, they use the pidgin input to effectively create their own original language, known as a creole. Unlike pidgins, creoles have native speakers (those with acquisition from early childhood) and make use of a full, systematic grammar."