A/B Testing and Feature Flags

Overview

Let’s start by defining the difference between A/B testing and feature flagging. There is a lot of overlap between the two because the definition of both includes the simultaneous presence of two (or more) different ways of presenting a user interface and/or accomplishing a task within an application. The only real difference is the purpose of that split. In A/B testing, the goal of the split is to measure the relative efficacy of the two implementations to determine which one provides the best outcomes. Feature flagging is less specific that this - it only refers to an implementation method for achieving the split. So you could say that A/B testing uses feature flags to split an application into two different implementations.

You can use Adobe Target to establish feature flags that can then dictate to an application which implementation to use. But Target’s feature set allows for a richer set of capabilities beyond just feature flagging. You can not only return a boolean indicating whether the feature is on or off, but include additional metadata to dictate more details that each implementation can use to refine the implementation for a given demographic. Moreover, Target provides tooling that can give you information about the impacts of each implementation through tight integration with Adobe Analytics.

Client vs. Server

Target supports making that determination either on the client or on the server.

Client

On the client, the flow, as the name suggests, happens in the user’s browser and there is no server involvement.

graph LR
  Browser --1. Request --> Server
	Server --2. Response--> Browser
	Browser --3. Request Offers--> AdobeTarget[Adobe Target]
	AdobeTarget --4. Offers--> Browser

Target refers to the possible options as “Offers” but for the purpose of this explanation we can think of them as features. But you can see from the diagram that after the payload is returned from the server and the browser renders the response, we ask Target for the offers which are returned and then based on the response, changes are made to the application.

The downside of this approach is that you are potentially altering the user interface after it’s already been rendered which is jarring for the user and not the best experience. You can hide the whole page until the offers are returned and the changes are made but then you are simply making it take longer for the user to see the page.

Use this approach when the changes being made are minimally invasive. For example, an update to a small portion of the page that is not seen immediately when the user arrives on the page. Changing a form or input field in a modal that is not visible when the user first arrives on the page might be a good use for this type of thing.

Server

The server approach is helpful for more significant features that alter large portions of the page or dictate a change in the behavior in the application as a whole. In the server flow, when a request comes in from a browser, the server makes a determination on how to process the request based on a response from Target. 

graph LR
  Browser --1. Request--> Middleware
  Middleware --2. Request Offers--> AdobeTarget[Adobe Target]
	AdobeTarget --3. Offers--> Middleware
	Middleware --4a. Option 1 --> Server
	Middleware --4b. Option 2 --> Server
	Server --5. Response --> Middleware
	Middleware --6. Response--> Browser

The server approach will slow down the requests every time because we make a request to Adobe Target but there are ways to mitigate that through path analysis and configuration. For more advanced implementations, there are ways to cache the Target logic in the middleware so that we don’t have to query Adobe for every request.

NextJS

With NextJS, we can take advantage of the middleware file to implement the call to Adobe Target. The way we have implemented it in our App Router-based application is to provide a mechanism to “register” a path with a specific A/B test. The registration process involves associating a pathname with an A/B test implementation so that during the middleware request, the code will check for any registered A/B tests for the current path and make a request to Target to get the offer to use for that test. It will then pass that offer into the handler for the test which will do something based on that data. The options for what to do include:

  1. Rewrite the URL
  2. Alter the headers on the request (to add a cookie, for example)
  3. Redirect the request
  4. Pass the request through and alter the response
  5. Stop any pass through to the server and, instead, return a response right there.

Rewriting the URL

You would rewrite the URL in cases where you are presenting a completely different implementation of a page. For example, in E-Commerce, let’s say you want to show the user a redesigned product detail page. This allows the engineering team to develop a different implementation of the page at a separate endpoint independent of the original. It’s a good approach because it allows that team to work independently of the work that’s being done one the original page.

Altering Cookies

This approach is similar to a client approach in that the actual work to change the page behavior or appearance is being done in the client through Javascript that is reading the injected cookie. This is a good approach in cases where you are making small modifications to an existing UI. It does involve branching logic in the rendering code which is not ideal but it reduces the weight of the A/B test.

Other Approaches

The other approaches are not ideal in that they are either fragile (altering the content returned from the server or bypassing the server altogether) or have noticeable user impacts (redirection).

Some Other Notes

When implementing this within a large React application, it’s a good idea to use a provider/hook to get information about the A/B test to colocate the logic that determines which experience to show the user. The hook can do the work of referencing cookies, local storage, query parameters, etc. to determine if there is any client-side work to be done. For example:

1
2// useFeatureFlags.ts
3function useFeatureFlags() {
4	const flags = cookies().get('feature-flags')?.split(',') || []
5	return flags.reduce((acc, f) => {acc[f] = true; return acc}, {})
6}
7
8// MyComponent.tsx
9function MyComponent () {
10	const flags = useFeatureFlags()
11	if (flags.myTest) {
12		// ...do something 
13	} else {
14		// ...do something else
15	}
16}

Note: When using the code above, the use of cookies will require that the component be a client component for obvious reasons. See more about this here.