HAProxy & Lua: Fifo and asynchronous actions

In some cases, it will be great to process asynchronously some task. Maybe our email server is very slow and you dont want to add this server processing time to the response time of the HTTP request.

The solution is provided in two parts:

• A Lua FIFO
• An asychronous action action using tasks and the FIFO.

The FIFO

The fifo is a simple script inspired from the "PIL" manual. There is the library:

-- This library provides fifo functions
--
-- Usage:
--
--   fifo = Fifo.new()
--   fifo:push(data)
--   daa = fifo:pop()

Fifo = {}
Fifo.meta = {}
Fifo.meta.__index = {}
Fifo.new = function()
        local fifo = {}
        fifo.first = 1 -- Always the first data
        fifo.last = 1 -- Alway the last available + 1
        fifo.data = {}
        setmetatable(fifo, Fifo.meta)
        return fifo
end
Fifo.meta.__index.push = function(fifo, data)
        fifo.data[fifo.last] = data
        fifo.last = fifo.last + 1
end
Fifo.meta.__index.pop = function(fifo, data)
        if fifo.first == fifo.last then
                return nil
        end
        local data = fifo.data[fifo.first]
        fifo.data[fifo.first] = nil
        fifo.first = fifo.first + 1
        if fifo.first == fifo.last then
                fifo.first = 1
                fifo.last = 1
        end
        return data
end

The usage of this library is easy. Create new Fifo object. Push and pop elements. Like this

require("fifo")
fifo = Fifo.new()
fifo:push("a")
fifo:push("b")
fifo:push("c")
print(fifo:pop())
print(fifo:pop())
print(fifo:pop())
print(fifo:pop())

This Little sample displays:

$ lua ./test.lua
a
b
c
nil

Asynchronous tasks with Lua & HAProxy

Now, we will use this library to stack task executed asynchronously. We use:

• A FIFO for storing tasks
• An HAProxy Lua task for executing it
• The email library to send email

Fisrt step is initializing the FIFO. The FIFO is initialized in the main part of the Lua file. It it just:

require("fifo")

fifo_email = Fifo.new()

The second step is creation an action which send emails. This action put data where sent to the user. In this example, data is a copy of received request.

core.register_action("async_send_email", { "tcp-req", "http-req", "tcp-res", "http-res" }, function(txn)
    fifo_email:push("Request received:\n" .. txn.req:dup())
end)

And now, the task function which effectively send the data. This function pop the FIFO each seconds looking for jobs.

core.register_task(function()
    local ret
    local reason
    local server = "127.0.0.1"
    local port = 25
    local domain = "arpalert.org"
    local from = "haproxy@arpalert.org"
    local to = "admin@arpalert.org"
 
    while true do
       -- Process queue
       local data = fifo_email:pop()
       if data == nil then
          core.sleep(1)
       else
          -- Execute action
          local msg = "From: " .. from .. "\r\n" ..
                 "To: " .. to .. "\r\n" ..
                 "Subject: test - " .. os.date() .. "\r\n" ..
                 "\r\n" ..
                 data .. "\r\n"
          ret, reason = smtp_send_email(server, port, domain, from, to, msg);
          if ret == false then
             txn:Warning("Can't send email: " .. reason)
          end
       end
    end
end)

Finaly a little bit of haproxy configuration:

global
       lua-load samples.lua
       stats socket /tmp/haproxy.sock mode 644 level admin
    
    defaults
       timeout client 1m
       timeout server 1m
    
    listen sample4
       mode http
       bind *:10040
       http-request lua.async_send_email
       http-request redirect location /ok

That's all