Leveraging HAProxy Programs to Test URL Rewrite Rules

Explore how to leverage the program directive in HAProxy to test HAProxy routing and request rewrite configurations. Using the program directive to run Goreplay , a traffic relay binary that utilizes eBPF to capture traffic generated by HAProxy, allows forwarding the traffic to a test server for verification. By utlizing the same test harness to both, send requests to the HAProxy frontend and listen to the traffic captured by gor, we can easily verify that HAProxy generates the correct backend calls by exercising all the options and acls.

URL Manipulation in HAProxy with http-request Directives

HAProxy offers powerful directives such as http-request set-path, set-uri, and set-header to manipulate incoming requests before they are sent to backend servers. These directives enable dynamic request modifications, allowing for complex routing and rewriting logic. Here’s a brief overview of each directive:

• http-request set-path: Changes the path of the incoming request.
• http-request set-uri: Modifies the entire URI of the incoming request.
• http-request set-header: Sets or updates HTTP headers in the request.

These capabilities are essential for implementing sophisticated routing and rewriting rules, which can significantly enhance the flexibility and functionality of your HAProxy setup.

Why Testing HAProxy Configurations is Important

As HAProxy configurations become more complex, the risk of misconfigurations and unintended behaviors increases. Thorough testing is crucial to ensure that your HAProxy setup functions as expected, especially when dealing with intricate URL rewrite rules and routing logic. Automated tests can help identify issues early, reduce downtime, and maintain the reliability of your load balancer setup.

Our Testing Approach Using Gor and Program Directive

To address the challenges of testing HAProxy configurations, we use a technique that combines the program directive with Gor, a powerful traffic replay tool. Gor captures and forwards live traffic, making it ideal for testing and debugging HAProxy configurations.

+-------------+                       +---------  ---------+
|  Test       |   craft a test req    |                    |
|  Harness    +---------------------->+       HAProxy      |
+------^------+                       |                    |
       |                              +--------------------+
       |Verify Request                            |
       |                                          |
       |          +-------------+   +-------------v----------+
       +-----------  Goreplay   |   |     Request Shaping &  |
                  +-------^-----+   |     Filtering          |
                          |         +--------------|---------+
             Sniff Traffic|                        |
                          |                        |
                  +------- -----+                  |
                  | Backend call|<-----------------+
                  +-------------+

This approach has the benefit of not only testing that the call was formulated correctly, but also actually getting a response from the backend code as an integration test loop.

First, let’s explore a simple approach of mirroring the request back to the test client - this is very useful for quick debugging loops locally.

Prefer to jump to the full integration test approach? See the traffic capture implementation

Simple Approach: Setting Up HAProxy to Mirror the Modified Request

If you want to test that HAProxy performs the correct request manipulation and your ACLs are working properly, you can have HAProxy simply mirror the constructed request URL like this:

defaults
    log global
    mode http
    timeout connect 20s
    timeout client 30s
    timeout server 30s
    timeout http-request 30s

frontend http_front
    bind *:80
    mode http

    # Collect the query params as variables
    http-request set-var(req.width) url_param(w)
    http-request set-var(req.height) url_param(h)

    # Rewrite as path parameters
    http-request set-path "%[path]/width/%[var(req.width)]/height/%[var(req.height)]" if TRUE

    # To remove the original query parameters and just use the path, override with the newly calculated path
    http-request set-uri %[path]

    # Short-circuit and just respond with the path in the response body
    http-request return status 200 content-type "text/plain" lf-string "%[path]" if TRUE

    # Not actually triggered due to the early response above
    default_backend be_servers

backend be_servers
    balance roundrobin

In this setup:

• Frontend and Backend: Define the basic HAProxy frontend and backend configuration.
• Modify the request: Add/Modify/Drop headers, Apply ACLs, Edit the URI etc
• http-request return: Simply bounce back the rewritten request with modified headers/URI back to the test harness.

Verifying HAProxy Backend Calls

The test server receiving the response back from HAProxy can then verify that your configuration logic correctly modified the request before it would’ve been sent to the backend. This approach allows for a quick unit test of the HAProxy’s URL rewrite rules and routing logic, ensuring that your configuration behaves as expected.

Benefits of this Approach

• Local Testing: Enables fast local testing of HAProxy configurations.
  • Run as haproxy -f test.cfg
  • Quickly verify with:

  curl "localhost:80/foo?w=10&h=133"
  /foo/width/10/height/133
• Real Traffic Replay: Uses real traffic for testing, providing accurate insights.
• Easy Integration: Leverages existing tools and HAProxy capabilities.

Setting Up Gor with HAProxy Program for full integration test

The program directive in HAProxy allows you to run external programs directly from within the HAProxy configuration. By using Gor, we can capture traffic generated by HAProxy and forward it to a test server, which can then verify the backend calls. The same time, we can actually forward the requests to the real backend servers for an end-to-end test.

Here’s an example configuration snippet that demonstrates how to set this up:

defaults
    log global
    mode http
    timeout connect 20s
    timeout client 30s
    timeout server 30s
    timeout http-request 30s

frontend http_front
    bind *:80
    mode http

    # Collect the query params as variables
    http-request set-var(txn.width) url_param(w)
    http-request set-var(txn.height) url_param(h)

    # Rewrite as path parameters
    http-request set-path "%[path]/width/%[var(txn.width)]/height/%[var(txn.height)]" if TRUE

    # To remove the original query parameters and just use the path, override with the newly calculated path
    http-request set-uri %[path]

    # Short-circuit and just respond with the path in the response body
    # http-request return status 200 content-type "text/plain" lf-string "%[path]" if TRUE

    # Not actually triggered due to the early response above
    default_backend be_servers

backend be_servers
    server foo localhost:8080
    balance roundrobin

{{ if eq .Env.TEST "true" }}
program gor
    # Use Gor to capture traffic - and verify that the generated request is correct
    # Capture the requests going to port 8080 (destined for the actual backend): --input-raw :8080
    # Send them to our test harness: --output-http http://127.0.0.1:8081
    # Also print them to stdout:  --output-stdout
    command /usr/local/bin/gor --input-raw :8080 --output-http http://127.0.0.1:8081 --output-stdout
{{ end }}

TEMPLATING NOTE: Ideally use a templating engine to add the program directive for tests as shown. I recommend using something like https://docs.gomplate.ca/ and wrapping the program section in a toggle for generating your config in a test/prod env.

Along with an example matching test suite:

import (
	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"
	"net/http"
	"net/http/httptest"
	"time"
)

type capture struct {
	url     string
	headers http.Header
}

var ts *httptest.Server
var capturedRequest = make(chan capture, 1)
var _ = BeforeSuite(func() {
	ts = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		capturedRequest <- capture{
			url:     r.URL.String(),
			headers: r.Header,
		}
		w.WriteHeader(http.StatusOK)
	}))
	go ts.Start()
	DeferCleanup(func() {
		ts.Close()
	})
})
var _ = Describe("HAProxy", func() {
	defer GinkgoRecover()
	It("converts query params to path params", func() {
		// Send a request to HAProxy with query params
		res, err := http.Get("http://localhost:80/path?w=10&h=20")
		Expect(err).ToNot(HaveOccurred())
		Expect(res.StatusCode).To(Equal(http.StatusNotFound))
		// Verify that the request was proxied to the backend with path params
		// At this point, we wait for our test server to receive the request captured by Gor
		Eventually(capturedRequest).WithTimeout(5 * time.Second).Should(Receive(Equal(capture{
			url: "/path/width/10/height/20",
			headers: http.Header{
				"User-Agent":      []string{"Go-http-client/1.1"},
				"Accept-Encoding": []string{"gzip"},
			},
		})))
	})
	It("exercises ACLs", func() {
        // ACL Tests...
		Expect(1).To(Equal(1))
	})
})

In this setup:

• Frontend and Backend: Define the basic HAProxy frontend and backend configuration.
• Unique ID Headers: Add unique ID headers to requests and responses for tracking.
• Program Directive: Use the program directive to run Gor, capturing traffic on port 80 and forwarding it to the test server. You can also run a BPF expression like --input-raw-bpf-filter 'dst net 10.1.0.0/16' to capture all requests headed to you local VPC block for example.

Verifying HAProxy Backend Calls

The test server receiving the traffic from Gor can then verify that HAProxy generated the correct backend calls by inspecting the captured traffic. This approach allows for end-to-end testing of HAProxy URL rewrite rules and routing logic, ensuring that your configuration behaves as expected.

Benefits of This Approach

• Automated Testing: Enables automated testing of HAProxy configurations.
• Real Traffic Replay: Uses real traffic for testing, providing accurate insights.
• Easy Integration: Leverages existing tools and HAProxy capabilities.

Conclusion

By leveraging the program directive in HAProxy and utilizing Gor for traffic capture and forwarding, you can implement a robust testing framework for your HAProxy configurations. This approach ensures that your URL rewrite rules and routing logic function correctly, enhancing the reliability and performance of your load balancer setup.

Stay tuned for future posts about using LUA & Rust with HAProxy. Happy testing!