keycloak/docs/guides/server/haproxy-reencrypt.adoc

<#import "/templates/guide.adoc" as tmpl>
<#import "/templates/kc.adoc" as kc>
<#import "/templates/options.adoc" as opts>
<#import "/templates/links.adoc" as links>

<@tmpl.guide title="HAProxy with TLS re-encrypt"
summary="Configure HAProxy as a TLS re-encrypt load balancer for {project_name}."
includedOptions="proxy-headers health-enabled metrics-enabled https-certificate-file https-certificate-key-file https-client-auth https-trust-store-file https-management-client-auth shutdown-delay"
tileVisible="false">

This {section} describes how to configure link:https://www.haproxy.com/[HAProxy] as a TLS re-encrypt load balancer in front of {project_name}.

In TLS re-encrypt mode, HAProxy terminates the incoming TLS connection and establishes a new encrypted connection to {project_name}.
HAProxy operates at Layer 7 (HTTP). Unlike passthrough, end-to-end TLS encryption between client and {project_name} is not preserved, but the proxy gains the ability to inspect and modify HTTP traffic.

For a general overview of TLS re-encrypt and how it compares to passthrough, see the <@links.server id="reverseproxy"/> {section}.

A ready-to-run example with a Compose file is available in the link:https://github.com/keycloak/keycloak-quickstarts/tree/main/proxy/haproxy/reencrypt[quickstart repository].

[[haproxy-configuration-reencrypt]]
== HAProxy configuration

The following `haproxy.cfg` shows a configuration for TLS re-encrypt with two {project_name} backend servers.

[source,haproxy]
----
global
    log stdout format raw local0

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  50s
    timeout server  50s
    retries 3

frontend https_front
    bind *:8443 ssl crt /path/to/haproxy-external-certificate # <1>
    mode http # <2>

    # Prevent external spoofing
    http-request del-header Forwarded <3>
    http-request del-header x-forwarded-.* -m reg
    http-request del-header x-original-.* -m reg
    http-request del-header x-real-ip

    # Prevent external tracing context injection (W3C Trace Context / Baggage)
    http-request del-header traceparent
    http-request del-header tracestate
    http-request del-header baggage

    # Prevent external tracing context injection (Zipkin, Jaeger, OpenTracing)
    http-request del-header b3
    http-request del-header x-b3-.* -m reg
    http-request del-header uber-trace-id
    http-request del-header x-ot-span-context

    # Optional. Forward client identities so that client certificate lookups work as expected
    http-request set-header Client-Cert %[ssl_c_der,base64] if { ssl_c_used } { ssl_c_verify 0 }
    http-request set-header Client-Cert-Chain %[ssl_c_chain_der,base64] if { ssl_c_used } { ssl_c_verify 0 }

    # Public paths. Revisit the reverse proxy guide for the latest guidance. # <4>
    # With these settings, the redirect to the welcome screen or Admin UI will not work from external IP addresses, and this is expected.
    acl is_public_path path_beg /realms/
    acl is_public_path path_beg /resources/
    acl is_public_path path_beg /.well-known/

    # Allowed source IP ranges. Replace with your internal IP address ranges. # <4>
    acl is_allowed_src src 192.168.0.0/16
    acl is_allowed_src src 172.16.0.0/12
    acl is_allowed_src src 10.0.0.0/8
    acl is_allowed_src src 127.0.0.0/8

    http-request deny unless is_public_path or is_allowed_src # <4>

    default_backend keycloak_back

backend keycloak_back
    mode http # <5>
    balance roundrobin # <6>
    option forwarded host by by_port for # <7>
    option httpchk GET /health/ready # <8>
    http-check expect status 200

    server keycloak1 keycloak1:8443 ssl verify required crt /path/to/haproxy-internal-certificate ca-file /path/to/keycloak-1-certificate check port 9000 check-ssl verify none inter 5s fall 3 rise 2 slowstart 60s # <9>

    server keycloak2 keycloak2:8443 ssl verify required crt /path/to/haproxy-internal-certificate ca-file /path/to/keycloak-2-certificate check port 9000 check-ssl verify none inter 5s fall 3 rise 2 slowstart 60s

----

<1> The port HAProxy listens on for incoming TLS connections.
The `ssl crt /path/to/haproxy-external-certificate` directive sets the certificate that the proxy will use to identify itself on the frontend.
<2> Enables link:https://docs.haproxy.org/3.2/configuration.html#4-mode[HTTP mode] on the frontend.
This tells HAProxy to operate at Layer 7, decrypting and re-encrypting HTTP traffic.
HAProxy has access to the plaintext HTTP traffic in this mode.
<3> The `http-request del-header` directives remove HTTP headers from incoming requests before forwarding them to {project_name}.
This prevents external clients from spoofing proxy identity headers (such as `Forwarded`, `+X-Forwarded-*+`, and `X-Real-IP`), injecting authentication-related headers (such as `X-Forwarded-Access-Token`), or injecting distributed tracing context (such as W3C Trace Context, Zipkin B3, or Jaeger headers).
For the full list of recommended headers to filter, see the <@links.server id="reverseproxy" anchor="header-filtering-recommendations"/> {section}.
<4> Restricts access so that only public {project_name} paths are reachable from external networks. Requests to non-public paths (such as the Admin API or Admin Console) are only allowed from the configured internal IP ranges. For the full list of paths and recommendations, see the <@links.server id="reverseproxy" anchor="_exposed_path_recommendations"/> {section}.
<5> The backend must also use HTTP mode to match the frontend.
<6> Distributes connections across backend servers using link:https://docs.haproxy.org/3.2/configuration.html#4-balance[round-robin] load balancing.
<7> This option adds a `Forwarded` header containing the correct client information.
<8> Configures link:https://docs.haproxy.org/3.2/configuration.html#4.2-option%20httpchk[HTTP health checks] against {project_name}'s readiness endpoint.
<9> Defines a backend {project_name} server.
The parameters on this line control mTLS settings, health checks, and failure detection:

The `server` directive parameters are explained below:

`ssl verify required`:: Enables a secure connection to Keycloak and verifies the certificate.

`crt /path/to/haproxy-internal-certificate`:: Configures the certificate used to authenticate HAProxy to Keycloak.

`ca-file /path/to/keycloak-N-certificate`:: Configures the backend server certificate (or issuing CA certificate) used to authenticate the particular Keycloak server to HAProxy.

`check port 9000 check-ssl verify none`::
Directs health checks to the link:https://docs.haproxy.org/3.2/configuration.html#5.2-check[management port] (`9000`) over HTTPS.
The `verify none` option skips TLS certificate verification for the health check connection, which is acceptable because the health check is an internal communication between HAProxy and {project_name}.
The management port is separate from the main application port (`8443`), as described in the <@links.server id="management-interface"/> {section}.

`inter 5s fall 3 rise 2`::
Configures the link:https://docs.haproxy.org/3.2/configuration.html#5.2-inter[health check frequency]:
poll every 5 seconds, mark a server as down after 3 consecutive failures, and mark it as up again after 2 consecutive successes.
These values affect how quickly HAProxy detects that a {project_name} instance is shutting down (see <<graceful-shutdown-considerations-haproxy-reencrypt>>).

`slowstart 60s`::
Gradually increases the server's weight from 0 to its full value over 60 seconds after the server becomes available.
This gives {project_name} and the JVM time to warm up (JIT compilation, class loading, cache population) before receiving full traffic, preventing performance degradation and timeouts during the warmup phase.

[[keycloak-configuration-haproxy-reencrypt]]
== {project_name} configuration

With TLS re-encrypt, {project_name} requires the following configuration:

<@kc.start parameters="--proxy-headers forwarded --health-enabled true --metrics-enabled true --https-certificate-file=/path/to/certificate --https-certificate-key-file=/path/to/key --https-client-auth=required --https-trust-store-file=/path/to/https-truststore --https-management-client-auth=none"/>

`--proxy-headers forwarded`::
Restricts the forwarding headers that {project_name} will parse to the standard `Forwarded` header as per https://www.rfc-editor.org/rfc/rfc7239.html[RFC 7239].
Do not set `--proxy-protocol-enabled` when using TLS re-encrypt. The PROXY protocol is only relevant for TLS passthrough (TCP mode).
The `--proxy-headers` option and the `--proxy-protocol-enabled` option are mutually exclusive.

`--health-enabled true`::
Enables the health check endpoints on the management port (`9000`).
Without this, the `/health/ready` endpoint that HAProxy polls will not be available, and HAProxy will mark all {project_name} instances as down.

`--metrics-enabled true`::
Enables the metrics endpoints on the management port.
Enabling metrics also enables an additional status check for the database, so it is recommended to enable metrics.

`--https-certificate-file=/path/to/certificate` and `--https-certificate-key-file=/path/to/key`::
Configure the certificate and the private key {project_name} will use for HTTPS.
See the <@links.server id="enabletls"/> guide for additional details.

`--https-client-auth=required`::
Configures {project_name} to require client authentication, for mutual TLS.

`--https-trust-store-file=/path/to/https-truststore`::
Configures a truststore for client authentication.
In this case the truststore should contain the public certificate of the HAProxy.
See the <@links.server id="mutual-tls"/> guide for additional details.

`--https-management-client-auth=none`::
Disables the client authentication requirement for the management endpoint.

=== Enabling client certificate lookups

To enable the client certificate lookups, the following CLI options are required when starting {project_name}:

<@kc.start parameters="--spi-x509cert-lookup--provider=haproxy --spi-x509cert-lookup--haproxy--ssl-client-cert=Client-Cert --spi-x509cert-lookup--haproxy--ssl-cert-chain=Client-Cert-Chain --spi-x509cert-lookup--haproxy--certificate-chain-length=2"/>

If the number of intermediate certificates in your certificate chain is longer than the default, you must set the
`certificate-chain-length` option to an appropriate value. Otherwise, the provider will discard the request.

See <@links.server id="reverseproxy" /> for more details.

[[admin-dedicated-port-haproxy-reencrypt]]
== Optional: Admin UI and Admin API on a dedicated hostname and port

By default, the Admin UI and Admin API are served on the same port as the public endpoints, protected only by the source IP filtering (`is_allowed_src`).
For stricter network-level isolation, you can move them to a dedicated hostname and port so they can be independently firewalled.

Because the admin port and the public port share the same HAProxy frontend, the header filtering and client certificate forwarding described above apply to both automatically.

To enable this, add the configurations outlined in the following subsections.

=== {project_name} configuration

`+--hostname-admin=https://...+`::
Add the full admin URL with its hostname and the port (if it is a non-standard port) to the {project_name} configuration.

[NOTE]
====
Setting `hostname-admin` only changes the URLs {project_name} generates for the Admin Console and Admin API.
It does not stop the Admin API from also answering on the public frontend URL — restricting administration access to the dedicated port must be enforced at the reverse proxy, which is what the rules below do.
See the <@links.server id="hostname"/> {section}.
====

=== HAProxy configuration

TLS certificate::
If you choose to have a different hostname for the admin API, update your certificate to include the admin hostname as an additional Subject Alternative Name in the certificate, or issue a separate certificate.

`haproxy.cfg`::
* Add the second `bind *:8444 ...`, which makes the frontend also listen on the admin port.
If you choose a different port, replace `8444` in the examples below with your chosen port.
* Replace the default `http-request deny unless is_public_path or is_allowed_src` rule with the three isolation rules:
+
[source,haproxy]
----
acl is_admin_port dst_port 8444
http-request deny if !is_admin_port !is_public_path
http-request deny if is_admin_port !is_allowed_src
----

These rules route by destination port:

* *Public port:* only the public paths (`/realms/`, `/resources/`, `/.well-known/`) are served; any other path, including the Admin Console and Admin API, is denied.
* *Admin port:* all paths are served, but only to the allowed source IP ranges.

[NOTE]
====
With this setup, any request to an admin path on the public port is denied with an HTTP 403 response.
====

[[graceful-shutdown-considerations-haproxy-reencrypt]]
== Graceful shutdown considerations

The HAProxy health check settings determine how long it takes for the proxy to detect that a {project_name} instance is shutting down and that connections should no longer be routed to it.

With the health check settings from the configuration above (`inter 5s fall 3`), it takes up to 15 seconds (3 failures x 5-second interval) for HAProxy to mark a {project_name} instance as down.
During this period, {project_name} must remain running to serve in-flight requests.
Therefore, you need to configure the `--shutdown-delay` to be at least as long as the detection time:

<@kc.start parameters="--proxy-headers forwarded --health-enabled true --metrics-enabled true --https-certificate-file=/path/to/certificate --https-certificate-key-file=/path/to/key --https-client-auth=required --https-trust-store-file=/path/to/https-truststore --https-management-client-auth=none --shutdown-delay=16s"/>

For a detailed explanation of shutdown phases and how to tune the delay and timeout values, see the Graceful HTTP shutdown section in the <@links.server id="reverseproxy"/> {section}.

</@tmpl.guide>