Skip to content

Commit 34ee3d0

Browse files
elskowelskow
authored
Add nginx ingress SSL certificate crisis detection (#100)
* feat(rules): add nginx ingress SSL certificate crisis detection Add new rule CRE-2025-0120 to detect critical SSL certificate failures in NGINX Ingress Controllers * chore: rename into failure instead of crisis
1 parent 61d507e commit 34ee3d0

File tree

3 files changed

+143
-3
lines changed

3 files changed

+143
-3
lines changed

‎rules/cre-2025-0121/nginx-ingress-ssl-failure.yaml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
rules:
2+
- cre:
3+
id: CRE-2025-0120
4+
severity: 0
5+
title: NGINX Ingress Controller SSL Certificate Failure
6+
category: load-balancer-problems
7+
author: Prequel
8+
description: |
9+
Critical NGINX Ingress Controller SSL certificate validation failure detected. This pattern indicates
10+
cascading SSL failures where certificate verification errors lead to upstream connection failures
11+
and service unavailability. The failure sequence shows SSL handshake failures, certificate verification
12+
errors, and resulting HTTP error responses that affect client connectivity.
13+
cause: |
14+
- SSL certificate expiration without renewal
15+
- Invalid or self-signed certificates in production
16+
- Broken certificate chain or missing intermediate certificates
17+
- Certificate authority (CA) bundle misconfiguration
18+
- Certificate hostname mismatch (CN/SAN validation failure)
19+
- Upstream services using untrusted or expired certificates
20+
- SSL/TLS protocol version incompatibility
21+
- Certificate revocation or CA compromise
22+
- Misconfigured SSL verification settings
23+
- Network time synchronization issues affecting certificate validity
24+
impact: |
25+
- CRITICAL: Complete service unavailability for SSL-enabled endpoints
26+
- All HTTPS traffic fails with SSL handshake errors
27+
- Client applications receive 502 Bad Gateway or 503 Service Unavailable responses
28+
- Loss of secure communication channels and encrypted data transmission
29+
- Potential security exposure if SSL verification is disabled as workaround
30+
- Cascading failures across SSL-dependent microservices
31+
- Browser security warnings and user trust degradation
32+
- API integrations and automated systems fail due to SSL verification
33+
- Revenue loss from e-commerce and secure transaction failures
34+
- Compliance violations for systems requiring encrypted communication
35+
impactScore: 10
36+
tags:
37+
- nginx
38+
- ingress-controller
39+
- ssl-certificate
40+
- tls-handshake
41+
- certificate-verification
42+
- load-balancer
43+
- kubernetes
44+
- security
45+
- high-availability
46+
- service-unavailability
47+
mitigation: |
48+
IMMEDIATE ACTIONS:
49+
- Check SSL certificate expiration: `openssl x509 -in cert.pem -text -noout | grep -A2 Validity`
50+
- Verify certificate chain: `openssl verify -CAfile ca-bundle.pem server.crt`
51+
- Test SSL connectivity: `openssl s_client -connect hostname:443 -servername hostname`
52+
- Check NGINX SSL configuration: `nginx -t && nginx -s reload`
53+
- Monitor SSL handshake errors in real-time: `tail -f /var/log/nginx/error.log | grep SSL`
54+
55+
RECOVERY STEPS:
56+
1. Replace expired/invalid certificates with valid ones
57+
2. Update certificate chain with proper intermediate certificates
58+
3. Verify CA bundle contains trusted root certificates
59+
4. Restart NGINX Ingress Controller: `kubectl rollout restart deployment/nginx-ingress-controller`
60+
5. Test SSL endpoints: `curl -v https://your-domain.com/health`
61+
6. Monitor certificate auto-renewal processes
62+
63+
PREVENTION:
64+
- Implement automated certificate monitoring and alerting
65+
- Set up certificate auto-renewal with cert-manager or similar tools
66+
- Configure certificate expiration alerts (30, 7, 1 days before expiry)
67+
- Implement SSL health checks in monitoring systems
68+
- Use certificate transparency monitoring for unauthorized certificates
69+
- Regular SSL configuration audits and security scans
70+
- Implement proper certificate lifecycle management
71+
- Set up backup certificate authorities and failover mechanisms
72+
references:
73+
- https://kubernetes.github.io/ingress-nginx/user-guide/tls/
74+
- https://nginx.org/en/docs/http/ngx_http_ssl_module.html
75+
- https://cert-manager.io/docs/
76+
- https://www.openssl.org/docs/man1.1.1/man1/verify.html
77+
- https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
78+
applications:
79+
- name: nginx-ingress-controller
80+
version: ">= 1.0.0"
81+
- name: nginx
82+
version: ">= 1.15.0"
83+
mitigationScore: 7
84+
metadata:
85+
gen: 1
86+
id: L67VM4E8FeoKezgjWfo9gt
87+
kind: prequel
88+
rule:
89+
set:
90+
event:
91+
source: cre.log.nginx
92+
match:
93+
- regex: "SSL.*certificate.*verify.*failed|SSL.*handshake.*failed|certificate.*verify.*error|upstream.*SSL.*certificate.*verify.*failed|SSL.*connect.*error|upstream.*ssl.*handshake|502.*Bad.*Gateway|503.*Service.*Unavailable|upstream.*timed.*out.*SSL"

‎rules/cre-2025-0121/test.log‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
2024/07/01 14:30:15 [error] 123#123: *456 SSL certificate verify failed (18: self signed certificate) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/status HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/status", host: "api.example.com"
2+
2024/07/01 14:30:15 [error] 123#123: *456 upstream SSL certificate verify failed (18: self signed certificate) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/status HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/status", host: "api.example.com"
3+
2024/07/01 14:30:16 [error] 123#123: *789 SSL connect error while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/data HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/data", host: "api.example.com"
4+
10.244.0.1 - - [01/Jul/2024:14:30:16 +0000] "GET /api/v1/status HTTP/1.1" 502 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="0.005"
5+
2024/07/01 14:30:17 [error] 123#123: *890 SSL handshake failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /health HTTP/1.1", upstream: "https://172.20.0.3:443/health", host: "api.example.com"
6+
2024/07/01 14:30:17 [error] 123#123: *890 upstream SSL certificate verify failed (20: unable to get local issuer certificate) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /health HTTP/1.1", upstream: "https://172.20.0.3:443/health", host: "api.example.com"
7+
10.244.0.1 - - [01/Jul/2024:14:30:17 +0000] "GET /health HTTP/1.1" 502 Bad Gateway 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="0.003"
8+
2024/07/01 14:30:18 [error] 123#123: *901 certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/secure HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/secure", host: "api.example.com"
9+
2024/07/01 14:30:18 [error] 123#123: *901 SSL connect error (certificate verify failed) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/secure HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/secure", host: "api.example.com"
10+
10.244.0.1 - - [01/Jul/2024:14:30:18 +0000] "GET /api/v1/secure HTTP/1.1" 503 Service Unavailable 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="503" upstream_response_time="0.002"
11+
2024/07/01 14:30:19 [error] 123#123: *912 SSL certificate verify failed (19: self signed certificate in certificate chain) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/data HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/data", host: "api.example.com"
12+
2024/07/01 14:30:19 [error] 123#123: *912 upstream SSL certificate verify failed (19: self signed certificate in certificate chain) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/data HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/data", host: "api.example.com"
13+
10.244.0.1 - - [01/Jul/2024:14:30:19 +0000] "GET /api/v1/data HTTP/1.1" 502 Bad Gateway 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="0.004"
14+
2024/07/01 14:30:20 [error] 123#123: *923 SSL handshake failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown) while SSL handshaking to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "https://172.20.0.4:443/", host: "web.example.com"
15+
2024/07/01 14:30:20 [error] 123#123: *923 SSL connect error (handshake failure) while connecting to upstream, client: 10.244.0.1, server: web.example.com, request: "GET / HTTP/1.1", upstream: "https://172.20.0.4:443/", host: "web.example.com"
16+
10.244.0.1 - - [01/Jul/2024:14:30:20 +0000] "GET / HTTP/1.1" 503 Service Unavailable 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.4:443" upstream_status="503" upstream_response_time="0.001"
17+
2024/07/01 14:30:21 [error] 123#123: *934 certificate verify error: (62:hostname mismatch) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "POST /api/v1/submit HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/submit", host: "api.example.com"
18+
2024/07/01 14:30:21 [error] 123#123: *934 upstream SSL certificate verify failed (62: hostname mismatch) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "POST /api/v1/submit HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/submit", host: "api.example.com"
19+
10.244.0.1 - - [01/Jul/2024:14:30:21 +0000] "POST /api/v1/submit HTTP/1.1" 502 Bad Gateway 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="0.006"
20+
2024/07/01 14:30:22 [error] 123#123: *945 SSL certificate verify failed (21: unable to verify the first certificate) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/metrics HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/metrics", host: "api.example.com"
21+
2024/07/01 14:30:22 [error] 123#123: *945 SSL connect error (certificate chain verification failed) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/metrics HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/metrics", host: "api.example.com"
22+
10.244.0.1 - - [01/Jul/2024:14:30:22 +0000] "GET /api/v1/metrics HTTP/1.1" 503 Service Unavailable 157 "-" "Prometheus/2.45.0" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="503" upstream_response_time="0.002"
23+
2024/07/01 14:30:23 [error] 123#123: *956 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/slow HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/slow", host: "api.example.com"
24+
2024/07/01 14:30:23 [error] 123#123: *956 upstream timed out SSL while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/slow HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/slow", host: "api.example.com"
25+
10.244.0.1 - - [01/Jul/2024:14:30:23 +0000] "GET /api/v1/slow HTTP/1.1" 502 Bad Gateway 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="5.001"
26+
2024/07/01 14:30:24 [error] 123#123: *967 SSL handshake failed (SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/legacy HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/legacy", host: "api.example.com"
27+
2024/07/01 14:30:24 [error] 123#123: *967 upstream SSL handshake failed (no shared cipher) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/legacy HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/legacy", host: "api.example.com"
28+
10.244.0.1 - - [01/Jul/2024:14:30:24 +0000] "GET /api/v1/legacy HTTP/1.1" 503 Service Unavailable 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="503" upstream_response_time="0.003"
29+
2024/07/01 14:30:25 [error] 123#123: *978 certificate verify error: (9:certificate is not yet valid) while SSL handshaking to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/future HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/future", host: "api.example.com"
30+
2024/07/01 14:30:25 [error] 123#123: *978 SSL connect error (certificate not yet valid) while connecting to upstream, client: 10.244.0.1, server: api.example.com, request: "GET /api/v1/future HTTP/1.1", upstream: "https://172.20.0.3:443/api/v1/future", host: "api.example.com"
31+
10.244.0.1 - - [01/Jul/2024:14:30:25 +0000] "GET /api/v1/future HTTP/1.1" 502 Bad Gateway 157 "-" "curl/8.7.1" ssl_protocol="TLSv1.3" ssl_cipher="TLS_AES_256_GCM_SHA384" upstream_addr="172.20.0.3:443" upstream_status="502" upstream_response_time="0.004"
32+
33+
# Backend API Logs
34+
backend-api_1 | 2024/07/01 14:30:15 [error] 25#25: *1 SSL_do_handshake() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca) while SSL handshaking, client: 172.20.0.2, server: 0.0.0.0:443
35+
2024/07/01 14:30:16 [error] 25#25: *2 no suitable peer certificate available while SSL handshaking, client: 172.20.0.2, server: 0.0.0.0:443
36+
2024/07/01 14:30:17 [error] 25#25: *3 SSL_do_handshake() failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown) while SSL handshaking, client: 172.20.0.2, server: 0.0.0.0:443
37+
2024/07/01 14:30:18 [error] 25#25: *4 SSL_do_handshake() failed (SSL: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired) while SSL handshaking, client: 172.20.0.2, server: 0.0.0.0:443
38+
2024/07/01 14:30:19 [error] 25#25: *5 SSL_do_handshake() failed (SSL: error:14094419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied) while SSL handshaking, client: 172.20.0.2, server: 0.0.0.0:443

‎rules/tags/tags.yaml‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,9 @@ tags:
345345
- name: nginx
346346
displayName: Nginx
347347
description: Problems related to Nginx, such as weak ciphers, configuration errors, or performance issues
348+
- name: ingress-controller
349+
displayName: Ingress Controller
350+
description: Problems related to Kubernetes Ingress Controllers, such as SSL certificate validation failures, routing issues, or backend connectivity problems
348351
- name: tls
349352
displayName: TLS
350353
description: Problems related to TLS, such as weak ciphers, configuration errors, or performance issues
@@ -818,6 +821,12 @@ tags:
818821
- name: critical-failure
819822
displayName: Critical Failure
820823
description: Failures that cause immediate service termination or data loss
821-
- name: ingress-controller
822-
displayName: Ingress Controller
823-
description: Problems related to Kubernetes Ingress Controllers, such as SSL certificate validation failures, routing issues, or backend connectivity problems
824+
- name: ssl-certificate
825+
displayName: SSL Certificate
826+
description: Problems related to SSL/TLS certificate validation, expiration, trust chain issues, or handshake failures
827+
- name: tls-handshake
828+
displayName: TLS Handshake
829+
description: Problems during TLS/SSL handshake process including cipher negotiation, protocol version mismatches, and connection establishment failures
830+
- name: certificate-verification
831+
displayName: Certificate Verification
832+
description: Issues with SSL/TLS certificate verification including trust chain validation, certificate authority verification, and hostname matching

0 commit comments

Comments
 (0)