GitLab Redis Failure: No Route to Host and Firewall Rule Order

January 20, 2016

After security hardening on a test server, GitLab returned HTTP 500. Application logs pointed to Redis. Its process was running and its port listening, yet redis-cli connecting to loopback reported No route to host.

Narrow the application error to a connection path

redis-cli -h 127.0.0.1 -p 6379 PING

Use the host, port and authentication actually configured in the application. No route to host can indicate firewall rejection rather than a missing route. Refusal, timeout and authentication failure provide different clues.

Check Redis’s actual listening address and distinguish TCP from Unix sockets. Establish whether GitLab runs in a container or another network namespace. A process’s 127.0.0.1 may not be the host’s endpoint.

An allow rule after rejection may never match

In this incident, connectivity returned when the firewall was disabled. The appended allow rule was after an earlier rejection. Inspecting INPUT, OUTPUT and loopback rules and correcting their order restored both the connection and GitLab.

This demonstrates ordering, not that disabling a firewall is the answer to all local connection failures. Inspect rules and counters first, identify the exact match, and make a minimal change for the required path.

Do not expand Redis exposure incidentally

When a TCP client and Redis share a host, loopback binding usually suffices. Changing to 0.0.0.0 expands listening scope without fixing rules rejecting loopback traffic.

Allowing local access likewise does not imply opening 6379 to every source. Remote clients need explicit network scope, authentication and access control. Verify runtime and post-reboot rules, then test actual GitLab requests rather than stopping at a successful redis-cli connection.

References