Configure Infinispan for Keycloak 17

Question

I want to run Keycloak 17 (Quarkus Edition) in HA mode with the provided infinispan. Because we are running Keycloak on serveral stages, I want to specify a infinispan cluster name. As I understood from the documentation I should configure this in the given infinispan config xml ./conf/cache-ispn.xml

I altered

     <transport lock-timeout="60000"/>

to

     <transport cluster="myClusterName" lock-timeout="60000"/>

After that I ran .\kc.bat build --cache=ispn --cache-config-file=conf/cache-ispn.xml

and started up the server with .\kc.bat start

Sadly the output logging shows this:

[org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
[org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [MyHostName-14281|0] (1) [MyHostName-14281]

As seen in the logs, the cluster name still is the default "ISPN".

I already consulted the infinispan docs here: https://infinispan.org/docs/stable/titles/configuring/configuring.html

as well as the Keycloak docs:

https://www.keycloak.org/server/caching

https://www.keycloak.org/server/configuration

Can anyone help me out? Is this a bug related to Keycloak 17 or am I missing something in the infinispan config?

Full Infinispan Config:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright 2019 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

    <cache-container name="keycloak">
        <transport cluster="myClusterName" lock-timeout="60000"/>
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>

Answer 1

I figured it out:

First I copied the cache-ispn.xml to a new file in the same directory and named it cache.xml I changed the build paramter --cache-config-file=conf/cache-ispn.xml to --cache-config-file=cache.xml So I just removed the folder specification as Keycloak seems to auto set the config directory to the conf folder.

Answer 2

can you share your keycloak startup scipt and its arguments? i am facing the same issue, i can create a custom cluster but two instances not discovering each other. startup command and its logs as below. I have copied cache.xml under conf folder. I have also created postgresql database for storage.

[keycloak-18.0.0]$ ./bin/kc.sh start --auto-build --cache=ispn --cache-config-file=cache.xml  --cache-stack=udp --http-enabled=true --http-port=8080  --http-host=127.0.0.1 --hostname=localhost
2022-05-07 12:52:37,097 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: localhost, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: false
2022-05-07 12:52:38,034 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-05-07 12:52:38,076 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-05-07 12:52:38,095 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-05-07 12:52:38,365 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.8.Final
2022-05-07 12:52:38,510 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `uiot`
2022-05-07 12:52:38,611 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-05-07 12:52:38,612 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
2022-05-07 12:52:38,612 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-05-07 12:52:38,612 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB
2022-05-07 12:52:40,629 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) localhost-22258: no members discovered after 2010 ms: creating cluster as coordinator
2022-05-07 12:52:40,637 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel uiot: [localhost-22258|0] (1) [localhost-22258]
2022-05-07 12:52:40,641 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `uiot` local address is `localhost-22258`, physical addresses are `[10.0.2.15:56671]`
2022-05-07 12:52:41,195 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: localhost-22258, Site name: null
2022-05-07 12:52:41,645 INFO  [io.quarkus] (main) Keycloak 18.0.0 on JVM (powered by Quarkus 2.7.5.Final) started in 7.790s. Listening on: http://127.0.0.1:8080
2022-05-07 12:52:41,646 INFO  [io.quarkus] (main) Profile prod activated. 
2022-05-07 12:52:41,646 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
^C2022-05-07 12:54:03,839 INFO  [org.infinispan.CLUSTER] (Thread-14) ISPN000080: Disconnecting JGroups channel `uiot`
2022-05-07 12:54:03,908 INFO  [io.quarkus] (Shutdown thread) Keycloak stopped in 0.118s

keycloak conf as below...

# Basic settings for running in production. Change accordingly before deploying the server.

# Database

# The database vendor.
db=postgres

# The username of the database user.
db-username=postgres

# The password of the database user.
db-password=password

# The full database JDBC URL. If not provided, a default URL is set based on the selected database vendor.
db-url=jdbc:postgresql://localhost/iotdb

# Observability

# If the server should expose healthcheck endpoints.
#health-enabled=true

# If the server should expose metrics endpoints.
#metrics-enabled=true

# HTTP

# The file path to a server certificate or certificate chain in PEM format.
#https-certificate-file=${kc.home.dir}conf/server.crt.pem

# The file path to a private key in PEM format.
#https-certificate-key-file=${kc.home.dir}conf/server.key.pem

# The proxy address forwarding mode if the server is behind a reverse proxy.
#proxy=reencrypt

# Do not attach route to cookies and rely on the session affinity capabilities from reverse proxy
#spi-sticky-session-encoder-infinispan-should-attach-route=false

# Hostname for the Keycloak server.
hostname=localhost