A keystore is primarily a database for storing application secrets. Keystores can also be used for storing “trust certificates” and CA chains.
Trust certificates are public keys and they don’t necessarily need to be protected with the same rigor as private keys. However, they identify all trusted parties for an application. Java automatically looks for a certificate (or its root CA) in its keystores for all TLS requests. If it’s found, the TLS handshake will be successfully established. The keystores that are used for trust only (i.e., these databases do not contain any keys) are also called “truststores”.
Java comes bundled with the default keystore called “cacerts”. This keystore is pre-populated with many well-known root CAs. This means that any TLS call to a server whose certificate was issued by a well-known CA will be trusted.
Do not use “cacerts” or any other keystore bundled with Java. Instead, create the keystore specifically for your application. This keystore must only contain the keys/certs (including CAs) that are used by the application, it shall not contain any unused certs (including CAs), expired certs, etc.
Use javax.net.ssl.trustStore system property to point to your custom truststore file.
Any Java developer knows that the default password for Java keystores is “changeit”. If you want to use the keystore files bundle with Java (which we do not recommend), please change it to a strong password.
keytool -storepasswd -keystore cacert
Same goes for IBM JDK and WebSphere products (including a more modern Liberty app server) – the default password (“WebAS”) is well-known. Changing this password is the first thing that any WebSphere administrator must do.
As with any password, frequent password refresh is the key. Some organizations treat keystore passwords as service accounts and thus change them infrequently. This creates risks.
We recommend updating keystore password for every deployment/release.
Very often, keystore passwords are stored in open text in various config files.
E.g., here is a typical configuration of a Tomcat connector:
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
keystoreFile="${catalina.home}/conf/keystore.p12"
keystorePass="mypass" keyPass="mypass"
keyAlias="main-key"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
/>
This is obviously completely unacceptable from the security standpoint.
Save sensitive secrets elsewhere, ideally use a secret manager, like Hashicorp vault. At the very least, the master password could be stored in an environment variable for a properly protected account.
Keep the private keys in a separate keystore file, do not use the same keystore for trust certificates and for the keys. The keystore with the keys must be protected using the filesystem permission. It must also be secured using a complex password. The keystore with the keys must only contain the key/certificate pairs and CAs (if needed), nothing else.
Set permissions for keystore files to read-only. The account used to run the application should be the owner of the file.
chown account_id keystore_file_name
chmod 400 keystore_file_name
Make sure that the keystores contain only active keys, certificates and CA chains. Expired certificates, unused CAS must be removed.
This requires implementing a process for cleaning keystores on a periodic basis or as part of every application release.
Very often developers bundle keystore files with all the other application resources so they end up on the application classpath and then automatically picked up my Maven/Gradle build tools and added to Jar files.
This makes it difficult to update keys/certificates when needed. Keys/certificates can be (and should be) updated on a lifecycle which is different from the application’s lifecycle. Therefore, keeping them outside of application archives is a good idea – this allows for updating keystores completely independently from the application itself.
Even when there is a truly continuous delivery process in place and the application code is updated very frequently, it is still worth it to de-couple keystores from applications from the packaging standpoint so that the keystores could be quickly updated in case of a breach.
This also means that the deployment toolchain for keystores should be separate from the regular application deployment toolchain. In other words, keystores should be treated as a completely separate deployment artifact, independent of the application.
Keystores and PEM files should be placed on an external docker volume so they can be updated without modifying the docker image. This allows for updating keystores independently from docker images in case of a security breach or for a regular key/certificate rotation/refresh.
The files on the external volume could simply be replaced on the host and then the docker container (or multiple containers) can be restarted.
If there are multiple docker containers running on the same host, they could all share the same volume with keystores.
All keys and secrets must be de-coupled from source code and stored separately.
The production environment must always have its own dedicated keystore file.
PKCS12 is compatible with other tools, such as openSSL. It is the default for Java 9 and above but it must be explicitly set for previous Java versions.