[sid-]

In java to be able to crawl a https url you have to do the following - http://www.coderanch.com/t/134619/Security/JDK-trust-Certifi... Its easier in other languages but its not that hard in java. Just export certificate via IE and save to disk and from the jre/lib/security folder and issue one command keytool -import -alias mycert -keystore cacerts -file d:\mycert.cer. (default password is changeit) Done.

[scotty79]

Why must you do that? And why you don't have to do that for environments of modern llanguags?

[jcape]

tl;dr: Other environments are insecure out of the box, and require applications specifically opt-in to security. Java requires you opt-out of the security.

HTTPS is built on top of PKI, which involves a list of trusted root authorities who verify that the certificate for blahblah.com is actually for blahblah.com. A self-signed certificate won't have that, and any application that doesn't validate that the certificate is signed by a trusted authority and not expired, etc. has no security.

If an application doesn't validate it's certificate, anybody sitting between you and the HTTPS server can step in between you and your traffic, give you a phony certificate, and then proxy all your "secure" traffic to the HTTPS server. And, of course, "sitting between you and the HTTPS server" means not only the NSA with their low-latency network specifically built to conduct these types of attacks, it also means the guy in the corner at Starbucks too (because WiFi is a radio).

Java only actually started checking if certificates were valid very recently (IIRC it was J7, r51). Prior to that, Java was just as lax as every other toolkit---probably specifically to address complaints like Bray's: "testing HTTPS is tough".

[scotty79]

I never understood how trustworthy is cert that you could buy for 100$. What that certificate proves? That whoever signed the stuff had a 100$ at some point?

Besides ... why can't java just pull the certs out of the system (like you did manually) or ship with them like every browser does (I presume).

[jcape]

They typically ask that you perform some step of the transaction using an e-mail address tied to the domain, so it's not quite that terrible. The 700USD EV certs actually require corporate registration paperwork, tax IDs, etc. and are far closer to a credit check in terms of depth.

I agree that Java should use the certs the system provides, and that is a PITA to wrestle with keytool, but I also know that the self-signed cert that apache is using is not trusted by your PC either (so you've got work to do regardless).