There is at least one natural language (as opposed to the conlang Ithkuil mentioned by michau) that is impossible for non-native speakes to learn: Sentinelese. The reason is that it will probably cost your life.
The Sentinelese language is spoken by the inhabitants of North Sentinel Island, a small island (47.5 km2) that is part of the Andaman and Nicobar Islands.
The Andaman & Nicobar Administration describes the Sentinelese are "very hostile" and that little is known about them. The Government of India's Ministry of Tribal Affairs writes that the Andaman & Nicobar Administration
has adopted an ‘eyes-on and hands-off’ policy to ensure that no poachers enter into the island. A protocol of Circumnavigation of the North Sentinel Island has been made and notified in consultation with Government of India, which have yielded meaningful results.
Forbes wrote in September 2015 that
The North Sentinel island made headlines in 2006 after the tribe murdered two fishermen who had illegally approached the island. After the incident, a 3 mile zone has been imposed around the island, and the Sentinelese have since kept a low profile.
So Sentinelese is impossible for non-natives to learn because the Sentinelese try to kill anyone who tries to access the island. As a consequence, nothing is known for certain about the language, let alone how to learn it.
From the point of view of linguistic characteristics: I have read many articles about the "hardest language" and I have never seen any mention of a language that is too difficult for non-native speakers to learn.