HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04


Categories: Containers, Linux

This article has been updated in October 2018 and is now tested for HAProxy 1.8.14.

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy.  Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

This article is a recipe for deploying and then validating the latest HAProxy 1.8 with zero downtime support on Ubuntu 14.04 using sysV init scripts.  If you want this recipe for Ubuntu 16.04 with Systemd, see my other article.

Here is a Vagrantfile that can build the custom HAProxy for you, or continue reading for manual steps.

Install HAProxy 1.8.x

Install haproxy 1.8 using a ppa.

$ sudo apt-get install python-software-properties software-properties-common -y
$ sudo add-apt-repository ppa:vbernat/haproxy-1.8 -y

$ sudo apt-get update

$ sudo apt-cache policy haproxy
$ sudo apt-get install haproxy -y

$ haproxy -v
HA-Proxy version 1.8.14-1ppa1~xenial 2018/09/23

# add to syslog group 
$ sudo usermod -a -G syslog haproxy

# which haproxy
/usr/sbin/haproxy

Custom Startup Scripts

Place a modified version of the init script into ‘/etc/init.d/haproxy’.

$ sudo wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/ubuntu1404/haproxy -O /etc/init.d/haproxy
$ sudo chmod 755 /etc/init.d/haproxy

In addition to the standard stop|start|restart|reload, our init script provides an extra action named ‘reload-socket’.

  • restart – does a hard stop quits and closes all established connections
  • reload – attempts a port rebind to the new HAProxy process
  • reload-socket – retrieves listening sockets from the old process

The ‘-x’ option in the ‘reload-socket’ action specifies the unix socket used to transfer the listening sockets from the old process.  This is the same socket referenced later in the haproxy.cfg  ‘expose-fd listeners’.

SSL cert

As a prerequisite to configuring HAProxy for TLS traffic, we need to create a self-signed cert for the server host.

You can run the script below or follow the instructions I describe in my article here:

$ sudo wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/selfsigned.sh
$ sudo chmod 755 selfsigned.sh
$ ./selfsigned.sh

HAProxy configuration

Download the custom haproxy.cfg then modify the filename to your secure certificate.  Instead of ‘REPLACEME.pem’, use the cert name generated in the last section.

$ cd /etc/haproxy
$ sudo touch haproxy.cfg
$ sudo cp haproxy.cfg haproxy.cfg.orig
$ sudo wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/haproxy.cfg -O haproxy.cfg
$ sudo sed -i "s/REPLACEME/`hostname -f`/" haproxy.cfg

The important line in this config is ‘expose-fd listeners’ which enables the seamless reload functionality.

This would also be a good time to place a quick shell script that allows us to restart HAProxy in different modes and at different frequencies.

$ sudo wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/haproxytest.sh -O /usr/sbin/haproxytest.sh
$ sudo chmod 755 /usr/sbin/haproxytest.sh

This script is a simple loop where we pass the type of HAProxy restart we want (restart|reload|reload-socket) as well as the number of seconds between restarts.  Usage is described later in this article.

HAProxy logging

First, create the log file manually to avoid permissions issues.

$ sudo touch /var/log/haproxy.log
$ sudo chown haproxy:syslog /var/log/haproxy.log
$ sudo chmod ug+r+w /var/log/haproxy.log

Then modify ‘/etc/rsyslog.conf’ so that it can listen on UDP port 514 by uncommenting the UDP server.

$ sudo sed -i '/ModLoad imudp/s/^#//' /etc/rsyslog.conf
$ sudo sed -i '/UDPServerRun/s/^#//' /etc/rsyslog.conf

And then restart the rsyslog service:

$ sudo service rsyslog restart

HAProxy logs will go to “/var/log/haproxy.log” and will look similar to below.

Oct 28 12:17:50 haproxy1404 haproxy[2996]: 192.168.2.108:60668 [28/Oct/2018:12:17:48.293] www-https~ www-backend/mypool1 2439/0/0/1/2440 200 672 - - ---- 2/1/0/0/0 0/0 "GET / HTTP/1.1"

Node.js backend content

What we have currently is an HAProxy instance that has no content to serve.  The haproxy.cfg has a backend that grabs content from a pool of localhost ports at 9000, 9001, and 9002.

We will use a simple Node.js server to deliver content on these ports.  Open a new console to the HAProxy host so that we can run the server in the foreground and see its console output as calls are made.

$ sudo apt-get install nodejs nodejs-legacy -y

$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/server.js -O /tmp/server.js

$ sudo ufw allow 9000:9002/tcp

$ node /tmp/server.js 9000
Server started at 9000

started server on 0.0.0.0:9000
started server on 0.0.0.0:9001
started server on 0.0.0.0:9002

Smoke Test

We have taken a lot of steps to get here, it would be prudent to make sure that Node.js and HAProxy are correctly serving pages.

$ sudo service haproxy restart


$ sudo apt-get install curl -y
$ curl https://localhost --insecure

{"user-agent":"curl/7.35.0","host":"localhost","accept":"*/*","x-forwarded-proto":"https","x-forwarded-for":"127.0.0.1","connection":"close"}
endpoint: 0.0.0.0:9000
requestcount: 1
echo: undefined

$ curl https://localhost --insecure

{"user-agent":"curl/7.35.0","host":"localhost","accept":"*/*","x-forwarded-proto":"https","x-forwarded-for":"127.0.0.1","connection":"close"}
endpoint: 0.0.0.0:9001
requestcount: 2
echo: undefined

And in the other console where the Node.js server is running  you should see output that looks like:

served request: 1 on port 9000
served request: 2 on port 9001

This shows that our HAProxy instance is serving as the SSL termination point, going back to a pool of HTTP instances.

Load Tester Installation

With the setup and smoke test complete, now we want to put a high-concurrency load on the system while we reload HAProxy at a high frequency. Run the load generator from a different VM than where HAProxy is installed to minimize resource interference and simulate true network access.

For this article I will use Apache Workbench which has proven to be good at reporting back errors; whether those be at connection, handshake, or mid-stream.

The one issue with Apache Workbench is that it has hardcoded fatal failure after 10 exceptions, which doesn’t work for our purposes since we are purposely testing failures. So I have patched the program so that we can pass an ‘-R’ argument that allows execution to continue despite the number of individual errors.

Apache Workbench is part of the Apache HTTP download, so below are instructions for creating a custom binary.  Or you can use my Vagrantfile (vagrant up) or Dockerfile (docker build -t my-ab . && docker run -it my-ab) .

$ wget http://archive.apache.org/dist/httpd/httpd-2.4.37.tar.gz
$ tar xvfz httpd-2.4.37.tar.gz
$ cd httpd-2.4.37

$ cp support/ab.c support/ab.c.old
$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/ab.c -O support/ab.c

$ sudo apt-get install libapr1-dev libaprutil1-dev libpcre3 libpcre3-dev build-essential libssl-dev -y

# TLS1.2 option not available unless explicit with openssl
$ ./configure --enable-ssl --with-ssl=/usr/include/openssl
$ make

$ support/ab -V
This is ApacheBench, Version 2.3mymod <$Revision: 1807734 $> 

# should list -f protocol switch if openssl enabled
$ support/ab -?

$ sudo cp support/ab /usr/sbin/ab

Notice that the version reported back is “2.3mymod” which proves it is our custom binary. Here is the syntax for running the load test:

$ ab -r -R -c <numberUsers> -n <numberOfRequests> -f TLS1.2 -l https://<HAProxyHost>:443/

As an example, running as 10 parallel users with 100 requests total, the results should look something like below, and make sure you have the trailing slash on the URL as shown below or you will get “Invalid URL”.

$ ab -r -R -c 10 -n 100 -f TLS1.2 -l https://src1:443/
.
.
.
Server Software:        
Server Hostname:        src1
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        src1

Document Path:          /
Document Length:        Variable

Concurrency Level:      10
Time taken for tests:   0.340 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      36100 bytes
HTML transferred:       18600 bytes
Requests per second:    293.96 [#/sec] (mean)
Time per request:       34.018 [ms] (mean)
Time per request:       3.402 [ms] (mean, across all concurrent requests)
Transfer rate:          103.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        6   16   7.9     14      39
Processing:     6   18   5.5     17      31
Waiting:        6   17   5.5     16      31
Total:         21   34   9.5     30      65

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     33
  75%     36
  80%     38
  90%     44
  95%     62
  98%     65
  99%     65
 100%     65 (longest request)

Run Load Tests

Here are the four variations on restart/reload we will test:

  1. No restarts or reloads – running 10k requests with absolutely no restarts or reloads.  Expect: no errors and fast response times
  2. Hard restart – the HAProxy service will be hard stopped and restarted.  Expect: lots of errors as connections are killed abruptly
  3. Legacy reload  – attempts a port rebind to the new HAProxy process. Expect: some level of errors during high traffic or high frequency reload
  4. Reload using socket – retrieves and transfers listening sockets from the old process to the new process.  Expect: no errors

The “/usr/sbin/haproxytest.sh” script we placed unto the host earlier will assist in running these various tests.  It takes the parameters:

sudo haproxytest.sh <testType> <delaySeconds>

For our first batch of tests we will run with 100 parallel users, running 10k requests total.  Instead of showing detailed reports here, below are the results in table form:

 Test #req/users #fail resp avg 95% resp
<no restarts/reloads> 10k/100 0 301ms 316ms
haproxytest.sh restart 0.2 10k/100 8305 118ms 261ms
haproxytest.sh reload 0.2 10k/100 1 334ms 523ms
haproxytest.sh reload-socket 0.2 10k/100 0 580ms 471ms

These results match well to our expectations.  There was a high failure count for hard restarts, and a single error for the legacy reload mechanism.  The new reload based on socket transfer was a success and yielded 100% success rates.

There were some test runs using the legacy reload mechanism where I didn’t get a single error, so I wondered if I could change that by increasing the concurrent load to 200 users.

Test #req/users #fail resp avg 95% resp
<no restarts/reloads> 10k/100 0 612ms 700ms
runtest.sh reload 0.2 10k/200 21 573ms 1057ms
runtest.sh reload-socket 0.2 10k/200 0 688ms 1040ms

The failures were much more prevalent in legacy reload when concurrency was higher.  And the new reload using socket transfer continued to show success at not dropping any connections.

Summary

The older reload mechanism used by HAProxy was sufficient for most cases, but under high load or triggering reloads of high frequency it would drop connections.

If you are using HAProxy 1.8 and Linux kernel >=3.9 then you can take advantage of a new reload mechanism which can transfer sockets from the old process to the new process without dropping any connections.

This enables dynamic scalability and continuous integration all the way into production datacenters.

 

REFERENCES

https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/

https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html

https://engineeringblog.yelp.com/amp/2017/05/taking-zero-downtime-load-balancing-even-further.html

https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/

https://serverfault.com/questions/580595/haproxy-graceful-reload-with-zero-packet-loss

http://inside.unbounce.com/product-dev/haproxy-reloads/

https://www.mail-archive.com/haproxy@formilux.org/msg25632.html (original patch mailing list discussion)

http://louwrentius.com/how-to-compile-haproxy-from-source-and-setup-a-basic-configuration.html

http://www.haproxy.org/download/1.4/src/snapshot/?C=M;O=A (explains that haproxy-ss has latest source code with applied patches)

http://zhgwenming.blogspot.com/2012/09/tcp-sysctl.html (tcp related sysctl)

https://cbonte.github.io/haproxy-dconv/1.8/snapshot/configuration.html (haproxy 1.8 docs)

https://httpd.apache.org/docs/2.4/programs/ab.html (apache workbench docs)

https://launchpad.net/~ondrej/+archive/ubuntu/apache2 (ppa for apache)

http://apache.mirrors.pair.com/httpd/ (download apache src)

http://ibm-blue-box-help.github.io/help-documentation/troubleshooting/How_many_connections_can_HAProxy_handle/ (using gobench)

https://serverfault.com/questions/580595/haproxy-graceful-reload-with-zero-packet-loss (draining using iptables)

https://stackoverflow.com/questions/9436860/apache-httpd-setup-and-installation (apache install)

 

NOTES

Building latest version of haproxy 1.8

Download the latest 1.8 source snapshot and prepare it.

$ sudo apt-get install build-essential libssl-dev -y

$ cd /usr/src

$ sudo wget http://www.haproxy.org/download/1.8/src/snapshot/haproxy-ss-LATEST.tar.gz

$ sudo tar xfz haproxy-ss-LATEST.tar.gz
$ sudo mv haproxy-ss-20171014 haproxy

$ cd haproxy

$ cat VERSION
1.8-dev2
$ sudo chmod ugo+w VERSION
$ sudo echo "1.8-dev2mymod" > VERSION

Then make and compile the HAProxy 1.8 binary and copy it into the location expected by the init scripts.

$ sudo make TARGET=linux2628 CPU=native USE_LIBCRYPT=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_OPENSSL=1 USE_LIBPCRE=1
$ sudo make install
$ /usr/local/sbin/haproxy -v
HA-Proxy version 1.8-dev2mymod-64cc49c 2017/10/13

$ ./haproxy -v
HA-Proxy version 1.8-dev2mymod-64cc49c 2017/10/13

$ service haproxy stop
$ sudo cp ./haproxy /usr/sbin/haproxy

Docker build of custom Apache Workbench

wget https://raw.githubusercontent.com/fabianlee/blogcode/master/haproxy/ab/Dockerfile

docker build -t my-ab .
docker image rm my-ab
docker run -it my-ab -c 2 -n 10 https://www.google.com/