Archive for the ‘Troubleshooting’ Category

How to Troubleshoot and Fix Certificate Validation Issues in Java

Posted on 07/25/2019 , Alexander,

Certificate validation errors are a frequent cause of issues when dealing with APIs and Web services calls, especially when self-signed certificates are used.
The error message is usually javax.net.ssl.SSLHandshakeException: PKIX path building failed.

How to Troubleshoot

Run your code with the following system property: javax.net.debug="ssl,handshake,trustmanager". This trace will provide the most relevant output. You may want to save the entire output to a file.

At the top you will the exact trust store file used by the JVM as well as the available certificates:

javax.net.ssl|DEBUG|01|main|2019-07-24 16:11:55.923 EDT|TrustStoreManager.java:112|trustStore is: ./src/test/resources/com/myarch/sec/cryptofiles/app_truststore.pkcs12

Later on, after "ServerHello", you will see the certificate chain (could be a single certificate too, in the case of a self-signed certificate, it's always one cert) presented by the server:

javax.net.ssl|DEBUG|01|main|2019-07-24 16:11:56.511 EDT|CertificateMessage.java:358|Consuming server Certificate handshake message (
"Certificates": [
  "certificate" : {
    "version"            : "v3",
    "serial number"      : "00 FD FF 7C 02 1F DF B2 51",
    "signature algorithm": "SHA256withRSA",
    "issuer"             : "CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O="GoDaddy.com, Inc.", L=Scottsdale, ST=Arizona, C=US",
    "not before"         : "2017-10-27 19:32:01.000 EDT",
    "not  after"         : "2019-10-27 19:32:01.000 EDT",
    "subject"            : "CN=myarch.com, O=My Arch Inc., L=Centreville, ST=Virginia, C=US,

To see an abbreviated list of serial numbers of certificates in the trust store and certificates from the server, run:

cat log_file | grep  "serial\|ServerHello"

The output will look like this

   "serial number"      : "00 86 9A E3 5A 4D FE 72 BA",
javax.net.ssl|DEBUG|01|main|2019-07-24 16:11:56.415 EDT|ServerHello.java:866|Consuming ServerHello handshake message (
"ServerHello": {
javax.net.ssl|DEBUG|01|main|2019-07-24 16:11:56.416 EDT|ServerHello.java:962|Negotiated protocol version: TLSv1.2
    "serial number"      : "00 FD FF 7C 02 1F DF B2 51",
    "serial number"      : "07",
    "serial number"      : "1B E7 15",
    "serial number"      : "00",

One of the certificates (and one of the serials) in the trust store must match one of the certificates from the server.

How to Fix

You have a choice of adding the end entity's certificate (with the subject's CN usually matching the domain/hostname) or one of its issuers to the trust store.

Going with the end-entity certificate is more secure, however, you will need to update it when it changes/expires on the server. CA certificates have a much longer validity period, but having the CA cert in the trust store will make ALL of the certificates issued by that CA trusted by default (unless it was revoked by the CA).

The easiest is to obtain the certificates from the server is by using openssl:

openssl s_client -connect myarch.com:443  -showcerts

Copy the certificate that you want to import (starting with "-----BEGIN CERTIFICATE-----" and including "-----END CERTIFICATE-----") into a file. This gives you a PEM-encoded certificate.

Import the certificate into the truststore:

keytool -import -alias cert_alias -file cert_file -keystore truststore_file

Background

By default, Java uses a separate "truststore" (the same concept as the keystore, only for certificates, it does not contain any private keys), called "cacert". It is pre-populated with all known CAs (trust anchors). So all certificates issued by Verisign or Digicert will be trusted. As an aside, from the security standpoint, it is much better to use separate application-specific truststore.

You can see all the certificates in the default trustore by running

keytool -list -cacerts -storepass changeit

For a non-default trustore:

keytool -list -v -keystore trustore_file -storepass truststore_password

Certificates issued by internal CAs or self-issued certificates (a.k.a. self-signed) are never trusted by default.

During the TLS handshake a server presents its certificates (usually including the issuer's certificate) to the client as part of the "ServerHello" message. One of the certificates presented by the server must be in the truststore. This could be the end-entity cert (most secure) or one of its issuers.

The Java client is still going to validate the entire certificate chain, starting with the end-entity certificate, following the Certification path validation algorithm. This includes many additional checks, including the expiration date and the revocation (using OCSP starting with Java 9). If none of the certificate from the chain is in the truststore, the validation will fail in the very beginning with the "PKIX path building failed" exception.

ClassNotFoundException: A List of Dumb Things to Check

Posted on 01/14/2010 , Alexander Ananiev, 9 Comments ( Add )

You deploy a new version of your application into production environment, hit the application’s URL and get a 500 error with a long error stack and nasty “java.lang.ClassNotFoundException” in bold at the top.

“Class Not Found” exceptions could be quite tricky to troubleshoot because of the complexity of Java Web applications and application servers they run on. An average web application nowadays comes bundled with dozens of jar file (and probably thousands of classes). An average application server’s classpath is many pages long. Not to mention separately deployed libraries containing jar files shared by a group of applications. There should be little surprise that it is quite common for all these different jars and classloaders to clash with each other, get out of sync or become otherwise corrupt and mis-configured.

The list below represents a subset of all the possible causes of “ClassNotFoundException”. Hopefully this list could serve as a starting point for attacking the problem. The list was inspired by A List of Dumb Things to Check.

  • To start, determine a type of the offending class. Is it a an application class, a third-party library class, a class provided by the application server or a JDK class? Determine the jar file that should contain the class. Determine where that jar should be located on the file system. Is it part of application installation, application server installation or some shared library installation? You may need to search for the class within multiple jars. Here is the command to do it (source): find . <del>name *.jar -print -exec jar -tvf {} ; | awk '/YOURSEARCHSTRING/ || /jar/ {print} ' (note-it won’t search within EAR and WAR files)
  • Does the jar that’s supposed to contain the class exist on the file system?
  • Are you able to “unjar” the jar using jar -xvf? Does the jar indeed contain the package and class in question?
  • Check the version of the jar if you can’t find the class there. To determine the version, look at the jar’s MANIFEST.MF. Usually (but, unfortunately, not always) you will find some version information there. You can also compare the file size with the “baseline”.
  • Does the account that the application server’s JVM was started with have read access to the jar? An application server usually runs under some sort of a system account. The jar might have been copied to the file system using a personal account from a different group.
  • Have all application jars been updated during deployment? Are all the jars (including shared libraries) at the right version? Manual deployment process is quite common, so missing to update a jar is always a possibility.
  • Is the class in question packaged with the application (e.g., under WEB-INF/lib) and being loaded by one of the parent classloaders? Most application servers utilize a classloader hierarchy where WAR file’s classloader is a child of EAR classloader which in turn is a child of the system (JVM) classloader. Parent classloaders can’t go down to request a class from a child classloader. The problem occurs if, for example, some jars were moved to a shared library but they still depend on classes packaged with the application.
    In order to diagnose this situation, you’ll need to have a good understanding of your application server’s classloader hierarchy. Here is the information for WebSphere and here is the WebLogic documentation on classloaders.
  • Is any of the jars packaged with the application also present on any of the parent classloader’s classpath? Running different versions of the same jar or library can cause all kinds of issues, including ClassNotFoundException. Some app servers allow overriding default classloader behavior so that the jars packaged with the application are loaded first. This could fix the problem.
  • If the jar with the class in question is part of a shared library (as opposed to packaged with the application), check if this library was made available to the application via the classloader configuration. For example, WebSphere configuration involves setting up a separate classloader for the library and explicitly associating it with the application.
  • Is the version and patch level of the application server correct? Does it match your development environment? Look at the detailed version information for all the different components of your app servers and also get a list of installed patches. E.g., for WebSphere run versionInfo -long command.
  • Is the application server running under the right JDK? E.g., check if the server startup script relies on JAVA_HOME and see which JDK the JAVA_HOME points to.
  • If the application runs in a cluster, does the problem occur on all nodes or just on some? Are you trying to troubleshoot the problem on the right node?
  • If the classname is driven from a string, either in java source or some other file, have you spelled the class name correctly? (Steve Loughran)

Once again, this is by no means a complete list. If anybody wants to contribute, please add a comment below and I’ll update the post.