According to Beyond Highbrow – Robert Lindsay,
Icelandic is very hard to learn, much harder than Norwegian, German or Swedish. Part of the problem is pronunciation. The grammar is harder than German grammar, and there are almost no Latin-based words in it. The vocabulary is quite archaic. Modern loans are typically translated into Icelandic equivalents rather than borrowed fully into Icelandic.
There are four cases: nominative, accusative, dative and genitive – as
in German – and there are many exceptions to the case rules, or
“quirky case,” as it is called. In quirky case, case can be marked on
verbs, prepositions and and adjectives. The noun morphology system is
highly irregular. Articles can be postfixed and inflected and added to
the noun. In fact, Icelandic in general is highly irregular, not just
the nouns. Verbs are modified for tense, mood, person and number, as
in many other IE languages (this is almost gone from English).
There are up to ten tenses, but most of these are formed with auxiliaries as
in English. Icelandic also modifies verbs for voice – active, passive
and medial. Furthermore, there are four different kinds of verbs –
strong, weak, reduplicating and irregular, with several conjugation
categories in each division. Many verbs just have to be
memorized. Adjectives decline in an astounding 130 different ways, but
many of these forms are the same.
In essence, Icelandic is an archaic and isolated languages with an immensely tricky pronunciation, as well as completely unfamiliar and irregular sentence structure, vocabulary, and grammar.
Personally, though I suspect the fact that while Swedish and other Scandinavian languages influenced the development of modern English because the Vikings, who invaded the British Isles several times, were from Scandinavia, Iceland remained completely isolated from Britannia. Since Iceland had no influence over English, it seems logical that it would be more difficult than other Scandinavian languages.