From a PEM to a Running Stack — Part 4: When One Service Must Trust Two Worlds
The finale of the series. The certificates were clean, the passwords unified, the paths correct. One service still wouldn't start — and this time the reason was not a mistake but a misconception about the architecture of trust.
A step forward that looked like a step back
After the middleware finally trusted the internal auth server, a new trust error appeared. Frustrating at first glance — PKIX again. But the issuer in the error had changed:
Before: https://keycloak.example.local:5443/... <- the internal auth server Now: https://auth-provider.example.com/... <- a public cloud service
This was in truth a success: the internal server was no longer being distrusted. The error now concerned a second authentication provider — a public cloud service needed for a paid add-on module.
Lesson: when a detail changes in a stubborn error (here: the hostname in the error), it's a sign that the previous hurdle has fallen. Read error messages closely — the differences are the information.
The real problem: a truststore that's too narrow
The configuration bound the service to my self-built truststore via a Java option:
-Djavax.net.ssl.trustStore=/mw/root.p12
Here lies a property of Java you have to know: as soon as you set your own truststore, it replaces the default truststore completely. So the service trusted exclusively my internal CA — and not a single public CA anymore.
For the internal connection to the auth server, that was perfect. But the public cloud provider uses a certificate from a perfectly ordinary public CA (the kind every browser knows). That one was now missing — so, trust broken.
The solution: both worlds in one truststore
The requirement was clear: the service must trust my internal CA and the public CAs at the same time. The clean solution is to take Java's bundled default truststore (cacerts, which already contains all public CAs) as the basis and add your own CA to it.
The default cacerts lives in the JDK and usually has the password changeit. But since our service expects a different value, we have to switch the store password when copying and then import our CA — best done in the container, where cacerts and keytool are already present anyway:
docker compose run --rm -v /host/ssl:/out:ro --entrypoint sh rest-api -c '
CACERTS=$(find / -name cacerts 2>/dev/null | head -1)
cp "$CACERTS" /tmp/work.p12
# switch store password from "changeit" to our value
keytool -importkeystore -noprompt \
-srckeystore /tmp/work.p12 -srcstorepass changeit \
-destkeystore /tmp/root.p12 -deststorepass <TRUSTSTORE_PW> -deststoretype PKCS12
# add our own CA
keytool -importcert -noprompt -alias corp-ca \
-file /out/corp-ca-root.pem \
-keystore /tmp/root.p12 -storepass <TRUSTSTORE_PW>
cat /tmp/root.p12
' > /host/installation/certs/root.p12
The verification showed the goal reached:
keytool -list -keystore root.p12 -storetype PKCS12 -storepass <TRUSTSTORE_PW> \ | grep -iE 'entries|corp-ca' # -> Your keystore contains 151 entries # -> corp-ca, ..., trustedCertEntry
151 entries: the ~150 public standard CAs plus my one internal CA. With that, the service trusts both worlds. And indeed — the next start brought the longed-for line:
Tomcat started on port 8080 (http) with context path '/mw'
Lesson: a custom Java truststore replaces the default, it doesn't extend it. If a service reaches both internal and public targets, build the truststore from cacerts as a base plus your CA — otherwise you accidentally lock out half the world.
As if it were one last test, the nginx web server also hung itself up:
nginx: [emerg] cannot load certificate "/etc/nginx/certs/issued.crt": PEM_read_bio_X509_AUX() failed ... wrong tag ... nested asn1 error
We knew the pattern from Part 1: a double Base64-encoded file. But the real lesson here is a different one — nginx fundamentally ticks differently from the Java services.
EVIDENCE: while Keycloak and the middleware wanted PKCS#12 containers with a password, nginx expects plain PEM in cleartext: the certificate as -----BEGIN CERTIFICATE----- blocks, the key as -----BEGIN PRIVATE KEY-----, without a password and without container packaging.
Conveniently, I could extract both from the already-working PKCS#12 server keystore:
# full chain (server certificate + CA) as PEM openssl pkcs12 -in issued.p12 -passin pass:<TRUSTSTORE_PW> -nokeys -out full_chain.pem # private key as unencrypted PEM openssl pkcs12 -in issued.p12 -passin pass:<TRUSTSTORE_PW> -nocerts -nodes -out private.key
A small stumbling block at the end: OpenSSL likes to write Bag Attributes cleartext lines in front of the PEM blocks, which some nginx versions trip over. You cut them away cleanly:
sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' full_chain_raw.pem > full_chain.pem
And — almost most importantly — always check that certificate and key are a matching pair. The two hashes must be identical:
openssl x509 -in full_chain.pem -noout -modulus | openssl md5 openssl rsa -in private.key -noout -modulus | openssl md5
VERDICT: if they differ, you have readable files but TLS fails anyway — a mistake that would otherwise only surface during the handshake.
The big takeaway
After a day full of error messages, the essence can be cast into a few principles that hold far beyond this one project:
- "Certificate" is not a uniform thing. PEM, DER, PKCS#12, truststore, keystore, CA root, end certificate — these are different things with different rules. Most lost time comes from confusing them.
- The file extension lies. What's really in a file is told by
file,head,openssl x509 -textandkeytool -list— not the name. - Different tools have different truths. That OpenSSL accepts a file doesn't mean Java does. Verify with the tool that ultimately uses the file.
- Error messages are hints, not verdicts. "Wrong password" actually meant "wrong algorithm." "Connection refused" meant "service isn't running." Read them literally and question what they mean technically.
- If nothing changes after a change, the change was in the wrong place. This one realization would have saved me hours. Check the path, the file, and the value before repeating the same repair.
- Verify via fingerprints, not via names. An alias or filename proves nothing. A certificate's fingerprint is the only reliable identity.
- Understand the trust architecture, not just the commands. A custom truststore replaces the default. A truststore needs the CA, not an end certificate. Different services expect different formats. Whoever understands the model barely needs to memorize the commands.
In the end, the complete stack ran: the auth server, the middleware with its two trust anchors, the web server. A journey from a single .pem file to a dozen interacting containers — and every hurdle taught something that, next time, costs only minutes.
End of series. Thanks for reading along — and may your next certificate error be an instructive one.