Caddy does not trust its own Local CA - x509

[tctlrd]

Issue Details

Running the latest caddy in a container. Caddy does not trust its own local CA. Even though it is successfully installing the local CA's root into the system trust store.

I am pointing acme_ca at a local openbao acme server which is serving with caddy's local CA signed cert (acme_ca will not accept a non-tls hostname endpoint: #1592)

{"level":"info","ts":1763811706.332481,"msg":"certificate installed properly in linux trusts"}
...
ERROR tls.obtain could not get certificate from issuer ...with server: provisioning client: performing request: Get \"...pki_int/acme/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

This acme_ca_root config works after a caddy restart (it does not yet exist at first launch):
acme_ca_root /data/caddy/pki/authorities/local/root.crt

Thank you for your attention to this issue!

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[mohammed90]

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍 I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

[tctlrd]

thanks @mohammed90

3. Tutorial (minimal steps to reproduce the bug)

Create first Caddyfile: ./client

{
    local_certs
    on_demand_tls {
        ask http://localhost:5555/
    }
}

http://localhost:5555 {
	respond 200
}

https://client {
    tls {
        on_demand
        issuer acme https://server/acme/local/directory
    }
    root * /usr/share/caddy
    file_server
}

https://server {
    root * /usr/share/caddy
    file_server
}

Start the first container

mkdir client_data

podman run -d --name client \
  --network debug \
  -v "$(pwd)/client:/etc/caddy/Caddyfile:z" \
  -v "$(pwd)/client_data:/data:z" \
  -p 4443:443 \
  docker.io/caddy

Create second Caddyfile: ./server

{
    debug
}

https://server {
    tls /client/caddy/certificates/local/server/server.crt /client/caddy/certificates/local/server/server.key
    acme_server
}

Start the second container.

mkdir server_data

podman run -d --name server \
  --network debug \
  -v "$(pwd)/server:/etc/caddy/Caddyfile:z" \
  -v "$(pwd)/server_data:/data:z" \
  -v "$(pwd)/client_data:/client:z" \
  docker.io/caddy

Add 127.0.0.1 client to /etc/hosts file.

Send query to first container: curl https://client:4443

Check container logs: podman logs client

{"level":"info","ts":1763832993.1185694,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1763832993.1207433,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1763832993.1207626,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1763832993.1233463,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1763832993.124353,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000523d80"}
{"level":"info","ts":1763832993.1424704,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1763832993.1425338,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1763832993.143412,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1763832993.1437027,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1763832993.143878,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1763832993.144031,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":5555"}
{"level":"warn","ts":1763832993.1440434,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":5555"}
{"level":"info","ts":1763832993.1440506,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1763832993.14415,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1763832993.1441598,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1763832993.1441672,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1763832993.144178,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["client","server"]}
{"level":"warn","ts":1763832993.1443417,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1763832993.1449661,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1763832993.1449852,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1763832993.1461627,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1763832993.146519,"logger":"tls.obtain","msg":"acquiring lock","identifier":"server"}
{"level":"info","ts":1763832993.1474688,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1763832993.1477306,"logger":"tls.obtain","msg":"lock acquired","identifier":"server"}
{"level":"info","ts":1763832993.147893,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"server"}
{"level":"info","ts":1763832993.155135,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"server","issuer":"local"}
{"level":"info","ts":1763832993.1553261,"logger":"tls.obtain","msg":"releasing lock","identifier":"server"}
{"level":"warn","ts":1763832993.156047,"logger":"tls","msg":"stapling OCSP","identifiers":["server"]}
{"level":"info","ts":1763832993.2070396,"msg":"certificate installed properly in linux trusts"}
{"level":"info","ts":1763832993.2076292,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1763832993.2076485,"msg":"serving initial configuration"}
{"level":"info","ts":1763833010.4826975,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"10.89.0.1","remote_port":"35030","server_name":"client"}
{"level":"info","ts":1763833010.4905872,"logger":"tls.obtain","msg":"acquiring lock","identifier":"client"}
{"level":"info","ts":1763833010.4915922,"logger":"tls.obtain","msg":"lock acquired","identifier":"client"}
{"level":"info","ts":1763833010.4917595,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"client"}
{"level":"info","ts":1763833010.4928184,"logger":"tls.issuance.acme","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://server/acme/local/directory","error":"open /data/caddy/acme/server-acme-local-directory/users/default/default.json: no such file or directory"}
{"level":"info","ts":1763833010.4929159,"logger":"tls.issuance.acme","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
{"level":"info","ts":1763833010.4939184,"logger":"tls.issuance.acme","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://server/acme/local/directory","error":"open /data/caddy/acme/server-acme-local-directory/users/default/default.json: no such file or directory"}
{"level":"warn","ts":1763833010.5002823,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"warn","ts":1763833010.7548432,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"warn","ts":1763833011.009524,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"error","ts":1763833011.0097425,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"client","issuer":"server-acme-local-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"error","ts":1763833011.0097966,"logger":"tls.obtain","msg":"will retry","error":"[client] Obtain: registering account [] with server: provisioning client: performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority","attempt":1,"retrying_in":60,"elapsed":0.518183028,"max_duration":2592000}

check second container logs: podman logs server

{"level":"debug","ts":1763831940.6168392,"logger":"events","msg":"event","name":"tls_get_certificate","id":"b7801690-ec5e-41d8-9ecc-9e82ea7039b0","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,4865,4866,4867],"ServerName":"server","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.89.0.7","Port":59698,"Zone":""},"LocalAddr":{"IP":"10.89.0.8","Port":443,"Zone":""}}}}
{"level":"debug","ts":1763831940.6169198,"logger":"tls.handshake","msg":"choosing certificate","identifier":"server","num_choices":1}
{"level":"debug","ts":1763831940.6169329,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"server","subjects":["server"],"managed":false,"issuer_key":"","hash":"1b22c11cb59348e27d61ab9931c542c3a034d0ac0e6a376fd9737a68fea207ee"}
{"level":"debug","ts":1763831940.6169462,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.89.0.7","remote_port":"59698","subjects":["server"],"managed":false,"expiration":1763875109,"hash":"1b22c11cb59348e27d61ab9931c542c3a034d0ac0e6a376fd9737a68fea207ee"}
{"level":"debug","ts":1763831940.618525,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.89.0.7:59698: remote error: tls: bad certificate"}

[mholt]

I'm pretty sure in containers the trust store installation can't succeed, right?

[francislavoie]

It can succeed when the container runs as root (default in Docker) but they're using podman here and trying to run it as non-root.

[tctlrd]

@mholt @francislavoie - I was running as root in podman and the trust store install succeeds as seen above.

{"level":"info","ts":1763832993.2070396,"msg":"certificate installed properly in linux trusts"}

I jumped into the container and /etc/ssl/certs/ca-certificates.crt included the caddy root CA cert.

My best guess is that caddy grabs the trust store into memory before installing it's own root CA and thus it does not trust its own CA, though it was properly installed.

[mholt]

Are you sure the cert is installed before the client(s) making the connection are started/running at all?

[tctlrd]

@mholt - Are you asking if the root CA cert is "installed in linux trusts" before the caddy process starts running in that same container? In my tutorial, this is the first container: "client". If so the log indicates that caddy initiates the install of its root CA after generating it... so the answer is no.