Wednesday, November 9, 2016

Jenkins fails Maven/Java builds when running API tests using HTTPS

Recently discovered the problem with Jenkins, when projects builds were failing due to REST API tests (RestAssured) fails for an unknown reason.

We've recently switched our infrastructure to HTTPS and that change was obviously the root cause as the tests on HTTP were passing successfully.

On local developer's machines, the QA servers or even on Jenkins machine tests were passing when running from bash.

Also tried to run them from the "jenkins" user, on Jenkins machine, from the build's workspace -
in the exact same way Jenkins runs during a build, and all tests were passing successfully on both HTTP and HTTPS.

HTTPS tests were failing only when started from Jenkins UI "Build Now" button. We were not able to reproduce these fails any other environment except this Jenkins instance.




JVM networking can be switched to debug mode using "-Djavax.net.debug=all" and there were the results after analyzing the output diff. (it produces a lot of garbage, so it's better to dump everything into text file and use some diff tools to find the spots of interest)

When running from CLI we've got the following:
trustStore is: /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

When running from Jenking "Build Now" we've got the following:
trustStore is:  /var/lib/jenkins/tools/hudson.model.JDK/jdk1.8.0_40/jre/lib/security/cacerts

So Jenkins has its own bundled version of JDK, that is used for running the builds and because trusted certificates are stored inside the JDK - those may be outdated if Jenkins wasn't updated recently.

The quick workaround would be to override the trustStore is for each Jenkins build where you would like to use HTTPS:
 -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

The best solution would be to upgrade your Jenkins on a regular basis.