Background: We have a load balancer powered by LVS + ldirectord. You can find the guide on how to set it up by your own Here.
Recently when we increased the web server pool to 10 servers, we found that the load is not perfectly balanced. After reading some documentation, we found that it could be related to LVS persistence.
Below are some very interesting findings.
quiescent = yes|no
If yes, then when real or failback servers are determined to be down,
they are not actually removed from the kernel’s LVS table. Rather,
their weight is set to zero which means that no new connections will be
This has the side effect, that if the real server has persistent
connections, new connections from any existing clients will continue to
be routed to the real server, until the persistent timeout can expire.
See ipvsadm for more information on persistent connections.
This side-effect can be avoided by running the following:
echo 1 > /proc/sys/net/ipv4/vs/expire_quiescent_template
If the proc file isn’t present this probably means that the kernel
doesn’t have LVS support, LVS support isn’t loaded, or the kernel is
too old to have the proc file. Running ipvsadm as root should load LVS
into the kernel if it is possible.
If no, then the real or failback servers will be removed from the
kernel’s LVS table. The default is yes.
If defined in a virtual server section then the global value is
maintain entry in table (but silently drop any packets sent), allowing service to continue if the ipvsadm table entries are restored.
expire the entry in table immediately and inform client that connection is closed. This is the expected behaviour by some people when running `ipvsadm -C`
expire_quiescent_template - BOOLEAN
0 - disabled (default)
not 0 - enabled
When set to a non-zero value, the load balancer will expire
persistant templates when the destination server is quiescent. This
may be useful, when a user makes a destination server quiescent by
setting its weight to 0 and it is desired that subsequent otherwise
persistant connections are sent to a different destination server.
By default new persistant connections are allowed to quiescent
If this feature is enabled, the load balancer will expire the
persistance template if it is to be used to schedule a
new connection and the destination server is quiescent.