Deploying Redis Sentinel for a Three-Server GateKeeper Cluster
Purpose
This guide describes how to configure a highly available Redis backend for three GateKeeper servers.
Each server will run:
- GateKeeper;
- Redis Server;
- Redis Sentinel;
- HAProxy, providing the local Redis endpoint
127.0.0.1:6380.
GateKeeper does not connect directly to an individual Redis server. Each GateKeeper instance uses its local HAProxy:
Redis Sentinel monitors the Redis instances and, if the master fails, promotes one of the replicas to become the new master.
HAProxy checks the actual role of each Redis instance and routes new connections only to the current master.
Three Sentinel instances with a quorum of 2 allow automatic failover if one Redis node fails.
Important
This guide provides Redis high availability, but it does not configure load balancing for public HTTP/HTTPS traffic across the GateKeeper servers. Public traffic requires a separate load balancer, DNS failover, or another ingress mechanism with GateKeeper health checks.
1. Example Topology
The following addresses are used in the examples:
| Server | Hostname | Private IP | Components |
|---|---|---|---|
| GK1 | gk1 |
10.114.0.11 |
GateKeeper, Redis, Sentinel, HAProxy |
| GK2 | gk2 |
10.114.0.12 |
GateKeeper, Redis, Sentinel, HAProxy |
| GK3 | gk3 |
10.114.0.13 |
GateKeeper, Redis, Sentinel, HAProxy |
During the initial deployment:
- GK1 is the initial Redis master;
- GK2 is a Redis replica;
- GK3 is a Redis replica;
- the Sentinel quorum is
2; - GateKeeper accesses Redis through
127.0.0.1:6380.
After the first failover, no specific server should be treated as the permanent master. Redis Sentinel determines the current role.
2. Important Warning
Warning
After Redis Sentinel has been started, do not replace
/etc/redis/sentinel.confor restore it from the original template. Replacing the active file with the initial configuration may return Sentinel to an outdated view of the cluster.
3. Prerequisites
Before starting, make sure that:
- all three servers are installed and accessible over SSH;
- bidirectional network connectivity is available between the servers' private IP addresses;
- the private IP addresses do not change after a reboot;
- all servers maintain accurate time using NTP;
- GateKeeper is already installed;
- ports
6379/tcpand26379/tcpare not accessible from the internet;
4. Creating Passwords
Use two different passwords:
REDIS_PASSWORD— for connecting to Redis;SENTINEL_PASSWORD— for connecting to Redis Sentinel.
To simplify HAProxy health checks, use passwords without spaces, quotation marks, backslashes, or newline characters. Hexadecimal values are suitable, for example.
Generate the passwords on a secure administrative machine:
Store the generated values in a secure secrets-management system.
The following placeholders are used in this guide:
Do not put real passwords in shell history, tickets, public documentation, or version control.
For the command examples below, load Redis CLI passwords from local root-only files instead of typing passwords directly in shell commands.
On each GateKeeper server, create the helper files:
sudo install -o root -g root -m 0600 /dev/null /root/.redis-cli-redis.env
sudo install -o root -g root -m 0600 /dev/null /root/.redis-cli-sentinel.env
Open the Redis CLI password file:
Add:
Open the Sentinel CLI password file:
Add:
5. Installing Packages
Run the following commands on GK1, GK2, and GK3:
sudo apt update
sudo DEBIAN_FRONTEND=noninteractive apt install -y \
redis-server \
redis-sentinel \
redis-tools \
haproxy
Check the installed versions:
Create backup copies of the configuration files:
sudo cp -a /etc/redis/redis.conf \
/etc/redis/redis.conf.before-gk-cluster
sudo cp -a /etc/redis/sentinel.conf \
/etc/redis/sentinel.conf.before-gk-cluster 2>/dev/null || true
sudo cp -a /etc/haproxy/haproxy.cfg \
/etc/haproxy/haproxy.cfg.before-gk-cluster
Stop Redis Sentinel until the initial configuration is complete:
6. Configuring iptables
The following connections must be allowed between the GateKeeper servers:
| Port | Purpose |
|---|---|
6379/tcp |
Redis replication and Redis management by Sentinel |
26379/tcp |
communication between Sentinel instances |
Port 6380/tcp must listen only on 127.0.0.1 and must not be opened in iptables.
6.1. Rules for GK1
Run on GK1:
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.11 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.11 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.11 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.11 --dport 26379 -j ACCEPT
6.2. Rules for GK2
Run on GK2:
sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.12 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.12 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.12 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.13 -d 10.114.0.12 --dport 26379 -j ACCEPT
6.3. Rules for GK3
Run on GK3:
sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.13 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.13 --dport 6379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.11 -d 10.114.0.13 --dport 26379 -j ACCEPT
sudo iptables -I INPUT -p tcp -s 10.114.0.12 -d 10.114.0.13 --dport 26379 -j ACCEPT
Review and persist the rules on each server:
7. Configuring Redis
On each server, open the Redis configuration file:
Find the existing directives and replace them. Do not leave multiple active versions of the same directive.
7.1. Redis Configuration on GK1
Use:
bind 127.0.0.1 10.114.0.11
protected-mode yes
port 6379
requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>
appendonly yes
appendfsync everysec
replica-read-only yes
replica-priority 100
During the initial deployment, do not add a replicaof directive on GK1.
7.2. Redis Configuration on GK2
Use:
bind 127.0.0.1 10.114.0.12
protected-mode yes
port 6379
requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>
appendonly yes
appendfsync everysec
replica-read-only yes
replica-priority 100
replicaof 10.114.0.11 6379
7.3. Redis Configuration on GK3
Use:
bind 127.0.0.1 10.114.0.13
protected-mode yes
port 6379
pidfile "/run/redis/redis-server.pid"
logfile "/var/log/redis/redis-server.log"
dir "/var/lib/redis"
requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>
appendonly yes
appendfsync everysec
replica-read-only yes
replica-priority 100
replicaof 10.114.0.11 6379
8. Allowing Redis Configuration Changes
Redis Sentinel changes the roles of Redis instances during failover. Redis must be able to persist the updated configuration by using CONFIG REWRITE.
Run the following commands on all three servers:
Verify the owner and permissions:
The /etc/redis directory must be owned by root, and the redis user must not be able to create or delete files in it.
The /etc/redis/redis.conf file must be writable by the redis group.
9. Starting the Initial Redis Master
Run on GK1:
sudo systemctl enable redis-server
sudo systemctl restart redis-server
sudo systemctl is-active redis-server
Test the connection:
Expected result:
Check the role:
GK1 should return the following role:
Verify that the configuration can be persisted:
Expected result:
10. Starting the Redis Replicas
Run on GK2:
sudo systemctl enable redis-server
sudo systemctl restart redis-server
sudo systemctl is-active redis-server
Run on GK3:
sudo systemctl enable redis-server
sudo systemctl restart redis-server
sudo systemctl is-active redis-server
On GK2 and GK3, check the replication status:
Expected parameters:
Some Redis versions may use the term replica, while the role field in INFO replication may still be displayed as slave.
On GK1, check the number of connected replicas:
Expected parameters:
Do not proceed to the Sentinel configuration until both replicas show:
11. Verifying Data Replication
Create a temporary key on GK1:
. /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
SET gk-cluster-bootstrap-check "$(date -Is)"
Check the key on GK2:
Check the key on GK3:
Both replicas must return the same value.
Delete the test key on GK1:
12. Configuring Redis Sentinel
Create a separate /etc/redis/sentinel.conf file on each server.
12.1. Sentinel on GK1
Open the file:
Add:
port 26379
bind 127.0.0.1 10.114.0.11
protected-mode yes
requirepass <SENTINEL_PASSWORD>
sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>
sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
12.2. Sentinel on GK2
Use:
port 26379
bind 127.0.0.1 10.114.0.12
protected-mode yes
requirepass <SENTINEL_PASSWORD>
sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>
sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
12.3. Sentinel on GK3
Use:
port 26379
bind 127.0.0.1 10.114.0.13
protected-mode yes
requirepass <SENTINEL_PASSWORD>
sentinel monitor mymaster 10.114.0.11 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>
sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
The same SENTINEL_PASSWORD must be used by all three Sentinel instances.
Set the permissions on each server:
13. Starting Sentinel Sequentially
Start Sentinel on GK1 first:
sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel
Check the current master:
. /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster
Expected result:
Then start Sentinel on GK2:
sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel
Wait until GK1 and GK2 discover each other, and then start Sentinel on GK3:
sudo systemctl enable redis-sentinel
sudo systemctl restart redis-sentinel
sudo systemctl is-active redis-sentinel
14. Verifying Redis Sentinel
Run the following command on each server:
The command must confirm that a quorum and a majority sufficient for failover are available.
Check the current master:
Check the discovered replicas:
Check the discovered Sentinel instances:
Make sure that each Sentinel sees:
- one Redis master;
- two Redis replicas;
- the other two Sentinel instances;
- a quorum of
2.
15. Configuring HAProxy
HAProxy must listen for the Redis endpoint only on the local address:
HAProxy will:
- connect to each Redis instance;
- authenticate with Redis;
- request replication information;
- consider only the server with the master role to be available;
- close existing connections if a server is no longer the master.
Open the HAProxy configuration file on each server:
Use the following configuration:
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
daemon
maxconn 4096
defaults
log global
mode tcp
retries 3
timeout connect 3s
timeout client 1m
timeout server 1m
frontend redis_local
bind 127.0.0.1:6380
mode tcp
default_backend redis_current_master
backend redis_current_master
mode tcp
option tcp-check
tcp-check connect
tcp-check send AUTH\ <REDIS_PASSWORD>\r\n
tcp-check expect string +OK
tcp-check send INFO\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
server gk1 10.114.0.11:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions
server gk2 10.114.0.12:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions
server gk3 10.114.0.13:6379 check inter 2s fall 2 rise 1 on-marked-down shutdown-sessions
Restrict access to the configuration file because it contains the Redis password:
Validate the syntax:
If the configuration is valid, enable and start HAProxy:
16. Verifying the Local HAProxy Endpoint
Run the following command on each GateKeeper server:
Expected result:
Check the role of the Redis server selected by HAProxy:
The result must begin with:
Test writing and reading through HAProxy:
. /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6380 \
SET gk-haproxy-check "$(hostname)-$(date -Is)"
redis-cli -h 127.0.0.1 -p 6380 \
GET gk-haproxy-check
redis-cli -h 127.0.0.1 -p 6380 \
DEL gk-haproxy-check
Check the listener:
The endpoint must listen only on:
It must not listen on 0.0.0.0:6380 or on the server's private IP address.
17. Configuring GateKeeper
Open the following file on each server:
Add or update:
Make sure the file does not contain multiple active REDIS_HOST or REDIS_PASSWORD lines.
Check the available GateKeeper services:
```bash
systemctl list-unit-files | grep -E '^(botguard|gatekeeper)'
Restart GateKeeper:
Check the status:
All expected services must start successfully.
18. Final Cluster Verification
Verifying GK1
Run on GK1:
. /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|connected_slaves:'
Initially, the expected output is:
Verifying GK2 and GK3
Run on GK2 and GK3:
. /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|master_host:|master_link_status:'
Initially, the expected output is:
Verifying the Local Redis Endpoint
Run on each server:
Expected result:
Verifying the Current Master Through Sentinel
Run on each server:
. /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster
All three Sentinel instances must return the same master address.
Verifying the Quorum
Run on each server:
Verifying the Services
systemctl status \
redis-server \
redis-sentinel \
haproxy \
botguard-controller \
botguard-apiserver \
--no-pager
19. Testing Automatic Failover
Perform this test during an approved maintenance window.
19.1. Identifying the Current Master
Run the following command on any server:
. /root/.redis-cli-sentinel.env
redis-cli -h 127.0.0.1 -p 26379 \
SENTINEL get-master-addr-by-name mymaster
With the initial configuration, the result should point to:
19.2. Creating a Test Value
Create a key through the local HAProxy instance:
19.3. Stopping the Current Redis Master
If GK1 is the current master, run the following command on GK1:
Do not stop Sentinel when testing only a Redis process failure.
19.4. Verifying the Selection of a New Master
Run on GK2 or GK3:
watch -n 1 \
'. /root/.redis-cli-sentinel.env && redis-cli -h 127.0.0.1 -p 26379 SENTINEL get-master-addr-by-name mymaster'
After failover is complete, the address must change to:
or:
Check the local HAProxy endpoint:
After the switchover, the expected result is:
Check the existing key:
Verify that writes are possible:
19.5. Bringing the Stopped Redis Instance Back Online
Run the following commands on the stopped server:
Do not manually assign the recovered server as the master.
Sentinel must automatically reconfigure the recovered former master as a replica of the current master.
Check the role of the recovered node:
. /root/.redis-cli-redis.env
redis-cli -h 127.0.0.1 -p 6379 \
INFO replication | grep -E 'role:|master_host:|master_link_status:'
Expected output:
Verify that all three Sentinel instances point to the same master.
20. Testing Sequential Server Reboots
After the failover test completes successfully, test rebooting each node one at a time.
Use the following sequence:
- make sure that
SENTINEL CKQUORUM mymastercompletes successfully; - reboot one Redis replica;
- wait for it to return;
- make sure that the replica shows
master_link_status:up; - check the Sentinel quorum again;
- only then proceed to the next node.
Do not reboot two Sentinel instances at the same time.
After rebooting each server, run:
sudo systemctl is-active redis-server
sudo systemctl is-active redis-sentinel
sudo systemctl is-active haproxy
Check the local endpoint:
Expected result: