23.8. Setting up HAProxy as a load balancer
In the Neo4j HA architecture, the cluster is typically fronted by a load balancer. In this section we will explore how to set up HAProxy to perform load balancing across the HA cluster.
For this tutorial we will assume a Linux environment with HAProxy already installed. See http://haproxy.1wt.eu/ for downloads and installation instructions.
Configuring HAProxy
HAProxy can be configured in many ways. The full documentation is available at their website.
For this example, we will configure HAProxy to load balance requests to three HA servers. Simply write the follow
configuration to /etc/haproxy.cfg
:
global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j backend neo4j option httpchk GET /db/manage/server/ha/available server s1 10.0.1.10:7474 maxconn 32 server s2 10.0.1.11:7474 maxconn 32 server s3 10.0.1.12:7474 maxconn 32 listen admin bind *:8080 stats enable
HAProxy can now be started by running:
/usr/sbin/haproxy -f /etc/haproxy.cfg
You can connect to http://<ha-proxy-ip>:8080/haproxy?stats to view the status dashboard. This dashboard can be moved to run on port 80, and authentication can also be added. See the HAProxy documentation for details on this.
Optimizing for reads and writes
Neo4j provides a catalogue of health check URLs (see Section 23.7, “REST endpoint for HA status information”) that HAProxy (or any load balancer for that matter) can use to distinguish machines using HTTP response codes.
In the example above we used the /available
endpoint, which directs requests to machines that are generally available for transaction processing (they are alive!).
However, it is possible to have requests directed to slaves only, or to the master only.
If you are able to distinguish in your application between requests that write, and requests that only read, then you can take advantage of two (logical) load balancers: one that sends all your writes to the master, and one that sends all your read-only requests to a slave.
In HAProxy you build logical load balancers by adding multiple backend
s.
The trade-off here is that while Neo4j allows slaves to proxy writes for you, this indirection unnecessarily ties up resources on the slave and adds latency to your write requests. Conversely, you don’t particularly want read traffic to tie up resources on the master; Neo4j allows you to scale out for reads, but writes are still constrained to a single instance. If possible, that instance should exclusively do writes to ensure maximum write performance.
The following example excludes the master from the set of machines using the /slave
endpoint.
global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves option httpchk GET /db/manage/server/ha/slave server s1 10.0.1.10:7474 maxconn 32 check server s2 10.0.1.11:7474 maxconn 32 check server s3 10.0.1.12:7474 maxconn 32 check listen admin bind *:8080 stats enable
Note
In practice, writing to a slave is uncommon. While writing to slaves has the benefit of ensuring that data is persisted in two places (the slave and the master), it comes at a cost. The cost is that the slave must immediately become consistent with the master by applying any missing transactions and then synchronously apply the new transaction with the master. This is a more expensive operation than writing to the master and having the master push changes to one or more slaves. |
Cache-based sharding with HAProxy
Neo4j HA enables what is called cache-based sharding. If the dataset is too big to fit into the cache of any single machine, then by applying a consistent routing algorithm to requests, the caches on each machine will actually cache different parts of the graph. A typical routing key could be user ID.
In this example, the user ID is a query parameter in the URL being requested. This will route the same user to the same machine for each request.
global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves balance url_param user_id server s1 10.0.1.10:7474 maxconn 32 server s2 10.0.1.11:7474 maxconn 32 server s3 10.0.1.12:7474 maxconn 32 listen admin bind *:8080 stats enable
Naturally the health check and query parameter-based routing can be combined to only route requests to slaves
by user ID. Other load balancing algorithms are also available, such as routing by source IP (source
),
the URI (uri
) or HTTP headers(hdr()
).