system toolbox

http://upstre.am/2008/01/09/using-haproxy-with-multiple-backends-aka-content-switching/

http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

After reading about HAProxy (there’s nice blog Rails-oriented blog post here), I felt the itch to try out this product myself. HAProxy has a handsome feature set:
http://affectioncode.wordpress.com/2008/06/11/comparing-nginx-and-haproxy-for-web-applications/

haproxy:

• It’s is proxy — and only a proxy. It can’t serve files, for example: proxying is all its does.
• It can proxy anything TCP-based — not just HTTP.
• Plenty of load-balancing algorithms, including a “least connections” strategy that picks the backend with the fewest pending connections. Which happens to be just what we want.
• Backends can be sanity- and health-checked by URL to avoid routing requests to brain-damaged backends. (It can even stagger these checks to avoid spikes.)
• A dedicated status page gives you backend status, uptime and lots of yummy metrics. There’s also a way to read metrics from a Unix domain socket.
• Requests can be routed based on all sorts of things: cookies, URL substrings, client IP, etc.

http://www.howtoforge.com/haproxy_loadbalancer_debian_etch

http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/

Using the normal HAProxy you can have real servers anywhere on the internet because the source address always points back at the HAProxy units IP address. However if the clients source IP address is going to be used then the HAProxy server MUST BE IN THE PATH of the return traffic ?.
The easiest way to do this is to put the backend servers in a different subnet to the front end clients and make sure that the default gateway points back at the HAProxy load balancer.

http://bluhaloit.wordpress.com/2009/04/29/how-to-quickly-setup-a-load-balanced-high-availability-apache-cluster/

http://iamseanmurphy.com/2009/04/22/a-better-haproxy-health-check-for-dynamic-websites/

A Better HAProxy Health Check For Dynamic Websites

Nobody wants their website to go down, or worse, for users to notice the site is down. Because of this most larger websites will run on multiple servers to provide some level of high availability. In a multi-server architecture there is typically a load balancer (or cluster of load balancers) to distribute the load among a pool of web servers. When a server goes down it’s taken out of the pool until it is once again ready to handle requests. HAProxy (a software load balancer) has the ability to perform this task by doing periodic health checks on all the servers in a cluster. The default settings, though, could give false positives in some cases, and thus create a bad user experience by allowing ill application servers to continue receiving requests.

When in HTTP mode HAProxy’s default health check is a simple OPTIONS request. This has the advantage of being a very lightweight request, and is easy to identify and filter from server logs. Consider this scenario though: HAProxy balances the load between several web servers running nginx and PHP-FastCGI. If nginx is up but PHP-FastCGI goes down, nginx will still properly handle the OPTIONS request from HAProxy, giving the impression that all is well. HAProxy continues sending requests to the ill server which in turn get a 504 Gateway Timeout (or similar) response. Not a very good situation.

A solution would be to use a deeper health check, one that goes beyond nginx to the PHP-FastCGI process. That way if PHP-FastCGI goes down, the whole server is presumed ‘down’.

backend appservers

 mode http

 option httpchk HEAD /health_check.php HTTP/1.1\r\nHost:\ example.com

 server web1 x.x.x.x:80 weight 5 check inter 2000

 server web2 x.x.x.x:80 weight 5 check inter 2000

 server web3 x.x.x.x:80 weight 5 check inter 2000

In the above example I’m using a custom health check request which will be processed by PHP-FastCGI. health_check.php is a lightweight script that contains simply <?php echo "I'm healthy"; ?>. I also added a host header so that the health check will be handled by a specific nginx virtual host. The nginx vhost config has this in it:

location = /health_check.php {

 access_log        off;

 fastcgi_pass    127.0.0.1:9000;

 fastcgi_index    index.php;

 include    /etc/nginx/fastcgi_params;

}

http://www.joeandmotorboat.com/2009/07/01/introducing-haproxy_join-and-how-to-use-it-with-chef/

http://www.joeandmotorboat.com/2009/08/20/haproxy-stats-socket-and-fun-with-socat/

http://github.com/joewilliams/haproxy_join

http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/

http://pgbouncer.projects.postgresql.org/doc/usage.html

#! /bin/sh
#
# /etc/init.d/myapp_rep.sh
#
# chkconfig: 35 99 99
# description: Starts or stops the myapp replication
#
# Assumes that application specific files
# * replication launcher script
# * rubyrep configuration file
# * rubyrep log file
# * rubyrep JRuby package
# are kept in /opt/myapp_rep/

case "$1" in
    start)
        echo "Starting myapp replication" 
        nohup bash /opt/myapp_rep/replicate 2>&1 |gawk '{print strftime("%Y-%m-%d %T",systime()),$0; fflush();}' >>/opt/myapp_rep/replication.log &
        ;;
    stop)
        echo "Shutting down myapp replication" 
        kill `ps -Af |grep myapp_rep|grep replicate |grep java |awk '{print $2}'`
        ;;
    restart)
        $0 stop
        $0 start
        ;;
    status)
        echo -e "myapp replication \\c" 
        ps -Af |grep myapp_rep|grep replicate |grep java >/dev/null || echo -e not \\c
        echo running
        ;;
    *)
        echo "Usage: $0 {start|stop|status}" 
        exit 1
        ;;
esac

http://1wt.eu/articles/2006_lb/

1. DNS¶

The easiest way to perform load balancing is to dedicate servers to predefined groups of users. This is easy on Intranet servers, but not really on internet servers. On common approach relies on DNS roundrobin. If a DNS server has several entries for a given hostname, it will return all of them in a rotating order. This way, various users will see different addresses for the same name and will be able to reach different servers. This is very commonly used for multi-site load balancing, but it requires that the application is not impacted to the lack of server context. For this reason, this is generally used to by search engines, POP servers, or to deliver static contents. This method does not provide any means of availability. It requires additional measures to permanently check the servers status and switch a failed server's IP to another server. For this reason, it is generally used as a complementary solution, not as a primary one.