HAProxy, Lua & Redis: Basic usage

This article shows a basic usage of Redis with HAProxy and Lua. This way is absolutely expensive because a network connection is open for each request.

I suppose that a local Redis server is installed.

First step is getting the Redis project. Checkout this project in the directory containing HAProxy Lua scripts. The lin to the library and the doc are here: https://github.com/nrk/redis-lua.

$ git clone https://github.com/nrk/redis-lua.git
Cloning into 'redis-lua'...
remote: Counting objects: 1439, done.
remote: Total 1439 (delta 0), reused 0 (delta 0), pack-reused 1439
Receiving objects: 100% (1439/1439), 337.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (647/647), done.
Checking connectivity... done.

For using this library in Lua script, we must adapt the search Lua library path, and obviously load the package.

package.path  = package.path  .. ";redis-lua/src/?.lua"

redis = require("redis")

Now we want to use redis for doing something. Two attention points:

  • First, we must use HAProxy cosocket in place of LuaSocket package. This is done in the configuration of the connexion. Look for {socket=tcp}
  • Second, the Lua call must be protected. This Redis library doesn't return error: it directly fail :-(. We must catch the fail for a clean termination of the process. Look for pcall(client.incrby, client, ip, 1)
  • Third, never use the quit() functions. This function call the LuaSocket function shutdown() which is not implemented in HAProxy.

We register an action which perform accounting on the IP source.

core.register_action("redis-accounting", { "http-req", "http-res", "tcp-req", "tcp-res" }, function(txn)

    -- create and connect new tcp socket
    local tcp = core.tcp();
    if tcp == nil then
        return
    end
    tcp:settimeout(1);
    if tcp:connect("127.0.0.1", 6379) == nil then
        return
    end

    -- use the redis library with this new socket
    local client = redis.connect({socket=tcp});

    -- Send redis accouting command
    local ip = txn.sf:src()
    pcall(client.incrby, client, ip, 1);

    -- Close connection
    tcp:close()
end)

And the HAProxy configuration

global
    lua-load samples.lua
    stats socket /tmp/haproxy.sock mode 644 level admin
    tune.ssl.default-dh-param 2048

defaults
    timeout client 1m
    timeout server 1m

listen sample5
    mode http
    bind *:10050
    http-request lua.redis-accounting
    http-request redirect location /ok

Now you can test:

$ redis-cli get '127.0.0.1'
(nil)
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"1"
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"2"
$ curl -s http://127.0.0.1:10050/
$ redis-cli get '127.0.0.1'
"3"

benchmark

I bench this solution on my laptop. It have a i7-4600U CPU @ 2.10GHz. 2 core, 4 threads. I reserve one core for haproxy, one thread for the injector, and one thread fr redis. The setp is:

A reference test: With the same HAProxy configuration without the Lua process (# http-request lua.redis-accounting), we reach about 70 000 HTTP request per second with an approximate ratio of CPU consummation 25% user and 75% system. Note that the test is limited by the injector who reach 100% CPU.

The results are not surprising:

We are limited by the HAproxy CPU. The consomation is about 98% user for the HAProxy process. Redis and the injector does nothing: about 15% cpu fr redis and 10% for the injector.

HAProxy process 4300 requests / second. HAProxy is very slow because the lib Redis is initialized to each request, the initialization takes a lot of CPU. In other way, the TCP connection is also initialized for each connection.