Some of the ways to determine how difficult it is to learn a language is to compare how different the motherlanguage of the learner is to the language which is learned. In addition, you can compare how long it takes a person on average to reach a general ability to speak and write in a given language. The Foreign Service Institue publishes a ranked list if you are interested. http://www.effectivelanguagelearning.com/language-guide/lang...
The study they cite is for monolingual English speakers. They concluded that the easiest language to learn was Swedish/Norwegian IIRC, and the hardest was Japanese by a factor 10.
It certainly is subjective, but it seems that Duolingo focuses on reading the language, so from that perspective, Japanese (and related languages such as Chinese and Korean) will be more difficult.
Unfortunately, I find learning the written portion of languages to be the easy part - learning to speak and hear the language is much harder.
I'm not sure how Polish could be considered that hard for English speakers. The writing system is completely phonetic, there are a ton of cognates (more so than other Slavic languages I think because of its proximity to Western Europe), and the ways of phrasing things are definitely more similar than your average non-Indo-European language. OTOH, I've heard the other Slavic languages have simpler grammar with fewer exceptions.