Deploy as a cluster

A GateKeeper cluster provides redundancy by running multiple GateKeeper servers together in the same cluster. A clustered deployment requires at least two GateKeeper servers. However, Blackwall recommends that you configure three or more so that cluster coordination services, such as Redis Sentinel, can be configured reliably. The more GateKeeper servers there are, the higher the overall throughput of the cluster. The cluster helps ensure that traffic can continue to be processed if one server is unavailable.

When to deploy as a cluster

Use a GateKeeper cluster when you need:

Redundancy across multiple GateKeeper servers.
Higher availability during maintenance or unexpected server issues.
Consistent protection behaviour across servers in the same cluster.
A scalable deployment model for higher-traffic environments.

Prerequisites

Before creating a clustered GateKeeper deployment, confirm that:

You have at least two GateKeeper servers available.
Network connectivity is available between your GateKeeper servers.
You are able to configure Redis Sentinel, or the required cluster coordination service across your servers.
Your GateKeeper API endpoint and authentication details are available to Blackwall Operations.
Your DNS A and AAAA records for blackwall.com resolve to the designated floating IP.
You must have either root-level or sudo-privileged account access via SSH.
You have valid API credentials for the infrastructure platform that manages your floating IP or failover IP.
You have valid API credentials to manage Blackwall services programmatically.

Example deployment - overview

└── GateKeeper Cluster  
    ├── GateKeeper Server 1 (GK1)
    ├── GateKeeper Server 2 (GK2) 
    └── GateKeeper Server 3 (GK3)

For the purposes of this example, consider the following details:

GateKeeper	Public IP	VPC IP	Components
GK1	198.51.100.11	10.114.0.12/20	Redis (Initial master), Sentinel, HAProxy, Failover script
GK2	198.51.100.12	10.114.0.5/20	Redis, Sentinel, HAProxy, Failover script
GK3	198.51.100.13	10.114.0.2/20	Redis, Sentinel, HAProxy, Failover script

Example deployment - configuration steps

Perform the following steps to configure Redis, Redis Sentinel, HAProxy, and Nginx across the three GateKeeper servers, so that the three servers can communicate with each other over the required internal network ports. This enables the GateKeeper servers to operate as a cluster rather than as independent servers.

Deploy three GateKeeper servers in the target infrastructure environment. Follow the steps provided in the Getting Started guide. By default, local Redis is provisioned for each server automatically.
Refresh the server’s package list so that the latest available package versions can be installed.
```
    sudo apt update
```
Run the following command to install Redis, HAProxy, cURL, jq, and UFW, which provide caching, load balancing, command-line request handling, JSON processing, and firewall management.
```
    sudo apt install -y redis-server haproxy curl jq ufw
```
Configure Redis on GK1 as the initial Redis master:
1. On GateKeeper server 1, open the Redis configuration file:
```
sudo nano /etc/redis/redis.conf
```
2. Update the Redis configuration so that it listens on the GK1 VPC IP address and localhost:
```
bind 10.114.0.12 127.0.0.1
protected-mode no
port 6379
pidfile "/run/redis/redis-server.pid"
logfile "/var/log/redis/redis-server.log"
dir "/var/lib/redis"
requirepass <REDIS_PASSWORD>
masterauth <REDIS_PASSWORD>
appendonly yes
```
3. Save and close the file. This configures GK1 as the Redis master, so that Redis listens on the GK1 VPC IP address, requires authentication, and enables append-only persistence.

Configure Redis on the GK2 server so that it replicates from Redis master, GK1:

On GateKeeper server 2, open the Redis configuration file:
```
sudo nano /etc/redis/redis.conf
```

Update the Redis configuration so that it listens on the GK2 VPC IP address and localhost:

    bind 10.114.0.5 127.0.0.1
    protected-mode no
    port 6379
    pidfile "/run/redis/redis-server.pid"
    logfile "/var/log/redis/redis-server.log"
    dir "/var/lib/redis"
    requirepass <REDIS_PASSWORD>
    masterauth <REDIS_PASSWORD>
    appendonly yes
    replicaof 10.114.0.12 6379

Save and close the file. GK2 is now configured as a Redis replica of GK1.

Configure Redis on the GK3 server so that it replicates from Redis master, GK1:

On GateKeeper server 3, open the Redis configuration file:
```
sudo nano /etc/redis/redis.conf
```

Update the Redis configuration so that it listens on the GK3 VPC IP address and localhost:

    bind 10.114.0.2 127.0.0.1
    protected-mode no
    port 6379
    pidfile "/run/redis/redis-server.pid"
    logfile "/var/log/redis/redis-server.log"
    dir "/var/lib/redis"
    requirepass <REDIS_PASSWORD>
    masterauth <REDIS_PASSWORD>
    appendonly yes
    replicaof 10.114.0.12 6379

Save and close the file. GK3 is now configured as a Redis replica of GK1.

Configure Redis Sentinel on each Gatekeeper server so that all servers can monitor the Redis master and automatically coordinate failover if the master becomes unavailable. In this example, Sentinel monitors the initial Redis master on GK1 at 10.114.0.12:6379. A quorum of 2 means that at least two Sentinel instances must agree that the master is unavailable before failover can proceed. If failover occurs, Sentinel promotes one of the replicas to become the new master and runs the fip-switch.sh script to update the floating IP routing.
1. On each of the three Gatekeeper servers, open the Redis Sentinel configuration file:
```
sudo nano /etc/redis/sentinel.conf
```
2. Add or update the following configuration:
```
port 26379
protected-mode no
sentinel monitor mymaster 10.114.0.12 6379 2
sentinel auth-pass mymaster <REDIS_PASSWORD>
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1
daemonize no
supervised systemd
pidfile "/run/sentinel/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis/redis-sentinel.log"

sentinel client-reconfig-script mymaster /usr/local/bin/fip-switch.sh
```
3. Save and close the file.

Create the Redis Sentinel failover script. In this script, FLOATING_IP is the stable public IP address that clients use to reach the GateKeeper cluster. During failover, this IP address is reassigned to the GateKeeper server that becomes the new Redis master. Replace the example 198.51.100.10 with the floating, reserved, or failover IP address provided by your infrastructure platform.

Open the failover script file:
```
sudo nano /usr/local/bin/fip-switch.sh
```

Add the following script content:

#!/usr/bin/env bash
#
# Redis Sentinel client-reconfig script
# Called as:  <master-name> <role> <state> <old-ip> <old-port> <new-ip> <new-port>
#             $1            $2     $3      $4       $5         $6       $7
# Sentinel only fires the script when $3 == "switch" and $2 == "master".
#

API_TOKEN="<INFRASTRUCTURE_API_TOKEN>"

# Map private Redis IPs to infrastructure instance IDs
declare -A INSTANCE_IDS=(
["10.114.0.12"]="<GK1_INSTANCE_ID>"
["10.114.0.5"]="<GK2_INSTANCE_ID>"
["10.114.0.2"]="<GK3_INSTANCE_ID>"
)

set -eu

new_master_ip="$6"
instance_id="${INSTANCE_IDS[$new_master_ip]:-}"

if [[ -z "$instance_id" ]]; then
logger -t fip-switch "No instance ID found for $new_master_ip – aborting."
exit 1
fi

logger -t fip-switch "Assigning $FLOATING_IP to instance $instance_id ($new_master_ip)"

# Replace this command with the API call required by your infrastructure platform.
curl -sS -X POST \
"<INFRASTRUCTURE_API_ENDPOINT_FOR_FLOATING_IP_ASSIGNMENT>" \
-H "Authorization: Bearer ${API_TOKEN}" \
-H "Content-Type: application/json" \
-d "<INFRASTRUCTURE_PROVIDER_REQUEST_BODY>" \
| logger -t fip-switch

exit 0

Save and close the file.

Tip

The example script above shows the failover logic, but the API request used to reassign the floating IP depends on your infrastructure provider. Replace the placeholder API endpoint and request body with the equivalent floating IP assignment command for your environment.

Update the file permissions and ownership so Redis Sentinel can execute the script.

sudo chmod 750 /usr/local/bin/fip-switch.sh

sudo chown redis:redis /usr/local/bin/fip-switch.sh

On each GateKeeper server, allow services to bind to non-local IP addresses.

echo 'net.ipv4.ip_nonlocal_bind = 1' | sudo tee /etc/sysctl.d/99-nonlocal-bind.conf
sudo sysctl --system
sysctl net.ipv4.ip_nonlocal_bind

Update the systemd service configuration for Redis and Redis Sentinel to reduce what each service can access on the server. This restricts access to user home directories and makes most of the filesystem read-only for the service. The ReadWritePaths entries then explicitly allow Redis and Sentinel to write only to the directories they need for data, logs, and runtime files.
1. For Redis, run:
```
sudo systemctl edit redis-server
```
2. Add the following settings to the relevant Redis systemd service override files:
```
[Service]
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=-/var/lib/redis
ReadWritePaths=-/var/log/redis
ReadWritePaths=-/var/run/sentinel
```
3. Save all changes.
4. Then, for Redis Sentinel, run:
```
sudo systemctl edit redis-sentinel
```
5. Add the following settings to the relevant Redis Sentinel systemd service override files:
```
[Service]
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=-/var/lib/redis
ReadWritePaths=-/var/log/redis
ReadWritePaths=-/var/run/sentinel
```
6. Save all changes.
7. Now reload systemd:
```
    sudo systemctl daemon-reload
```

Configure HAProxy on each Gatekeeper server. HAProxy routes Redis traffic to the current Redis master and forwards public HTTP and HTTPS traffic to the local Gatekeeper service.

Open the HAProxy configuration file:
```
    sudo nano /etc/haproxy/haproxy.cfg
```

Update the file to include the following configuration:

    global
            maxconn 200

    defaults
            log     global
            mode    tcp
            retries 2
            timeout connect 5000
            timeout client  50000
            timeout server  50000

    frontend redis_frontend
        bind *:16379
        default_backend redis_backend

    backend redis_backend
        mode tcp
        option tcp-check
        tcp-check send AUTH\ <REDIS_PASSWORD>\r\n
        tcp-check expect string +OK
        tcp-check send PING\r\n
        tcp-check expect string +PONG
        tcp-check send info\ replication\r\n
        tcp-check expect string role:master
        tcp-check send QUIT\r\n
        tcp-check expect string +OK
        server redis1 10.114.0.12:6379 maxconn 4096 check inter 3s
        server redis2 10.114.0.5:6379 maxconn 4096 check inter 3s
        server redis3 10.114.0.2:6379 maxconn 4096 check inter 3s

    # -------------------------------------------------------
    #  Public HTTP  :80 ->  GateKeeper 127.0.0.1:8180
    # -------------------------------------------------------
    frontend frontend_http
        bind *:80
        mode tcp
        default_backend backend_http

    backend backend_http
    mode tcp
    server gk_http 127.0.0.1:8180 send-proxy-v2

    # -------------------------------------------------------
    #  Public HTTPS :443 ->  GateKeeper 127.0.0.1:8443
    # -------------------------------------------------------
    frontend frontend_https
    bind *:443
    mode tcp
    default_backend backend_https

    backend backend_https
    mode tcp
    server gk_https 127.0.0.1:8443 send-proxy-v2-ssl check-ssl verify none

Before restarting HAProxy, validate the configuration. If the configuration is valid, HAProxy returns a message confirming that the file is valid. Run this command:
```
    sudo haproxy -c -f /etc/haproxy/haproxy.cfg
```
Restart HAProxy to apply the changes:
```
    sudo systemctl restart haproxy
```
You can confirm that HAProxy is running with the following command:
```
    sudo systemctl status haproxy
```

HAProxy is now configured to route Redis traffic through port 16379, forward public HTTP traffic from port 80 to Gatekeeper on 127.0.0.1:8180, and forward public HTTPS traffic from port 443 to Gatekeeper on 127.0.0.1:8443.

Execute the following command to enable and start Redis, Redis Sentinel, and HAProxy:

    sudo systemctl enable redis-server.service redis-sentinel.service haproxy.service
    sudo systemctl restart redis-server.service redis-sentinel.service haproxy.service

Check that the required services are running:

    systemctl status redis-server redis-sentinel haproxy nginx --no-pager

Update the Gatekeeper Nginx configuration so that public HTTP and HTTPS traffic is received by HAProxy first, then forwarded to Nginx on local loopback ports. Make these changes before adding domains to the cluster.

Open the website vhost template:

    sudo nano /opt/botguard/lib/botguard/templates/website.conf

Update the HTTP listeners from:

    listen 80;
    listen [::]:80;

to:

    listen 127.0.0.1:8180 proxy_protocol;
    listen [::1]:8180 proxy_protocol;

Update the HTTPS listeners from:

    listen 443 ssl;
    listen [::]:443 ssl;

to:

    listen 127.0.0.1:8443 ssl proxy_protocol;
    listen [::1]:8443 ssl proxy_protocol;

Save and close the file.

Apply the same listener changes to the API server:

Open the API server Nginx configuration:

    sudo nano /opt/nginx/etc/nginx/sites-enabled/api-server

Update the HTTP listeners from:

    listen 80;
    listen [::]:80;

to:

    listen 127.0.0.1:8180 proxy_protocol;
    listen [::1]:8180 proxy_protocol;

Update the HTTPS listeners from:

    listen 443 ssl;
    listen [::]:443 ssl;

to:

    listen 127.0.0.1:8443 ssl proxy_protocol;
    listen [::1]:8443 ssl proxy_protocol;

Save and close the file.

Check that the Nginx configuration is valid.
1. The following command must return a successful syntax test before you reload Nginx.
```
    sudo nginx -t
```
2. If Nginx reports an error, check the error log before proceeding to the next step:
```
    sudo tail -n 50 /opt/nginx/var/logs/error.log
```
3. If the syntax test described in the previous step is successful, reload Nginx:
```
    sudo systemctl reload nginx
```
Update the Blackwall configuration so that GateKeeper uses the new Redis endpoint through the floating IP and HAProxy.
1. Open the Botguard default configuration file:
```
    sudo nano /etc/default/botguard
```
2. Add or update the following values:
```
    REDIS_HOST=198.51.100.10:16379
    REDIS_PASSWORD=<REDIS_PASSWORD>
```
3. Save and close the file.
Restart the Gatekeeper services so that the updated Redis settings take effect:
```
    sudo systemctl restart botguard-controller botguard-apiserver nginx
```

The Gatekeeper cluster is now ready to add new domains.

Post-configuration checks

In the GateKeeper user interface, confirm that all servers display the same list of domains and that all domains have the same status across all GKs in the cluster. This confirms the shared Redis backend is being used.
We recommend that if you are able to do so, you test protected traffic, confirming that:
- Traffic reaches the GateKeeper servers.
- Protection rules are applied.
- Allowed traffic is forwarded to the origin.
- Blocked traffic is handled as expected.
Where failover is configured, test that traffic can continue to be served if one GateKeeper server is unavailable.

Tip

For each GateKeeper server, navigate to Settings in the user interface. Ensure that each GateKeeper server shows the central Redis configuration.

Verify Redis and floating IP status

At any time, you can check which Gatekeeper server is currently acting as the Redis master:

Run the following command on a Gatekeeper server:

redis-cli -h 127.0.0.1 -a <REDIS_PASSWORD> INFO REPLICATION

In the output, check the role value.
- If the server is the current Redis master, the output includes:
```
role:master
```
- If the server is a replica, the output includes:
```
role:slave
```
  or:
```
role:replica
```
To confirm which GateKeeper instance the floating IP is assigned to, use the API, CLI, or control panel provided by your infrastructure platform. For example, check the current floating IP assignment and confirm that it points to the GateKeeper instance that is currently acting as the Redis master. If your infrastructure platform provides an API, the command may look similar to the following:
```
curl -s -H "Authorization: Bearer <INFRASTRUCTURE_API_TOKEN>" \
"<INFRASTRUCTURE_API_ENDPOINT_FOR_FLOATING_IP_STATUS>"
```
Match the returned instance ID against the GateKeeper server list to confirm which instance is currently receiving traffic through the floating IP.

Feedback