Microservices automated testing and local development

Using Roxy in your backend for your automated testing and local development flow

This page describes the challenge and solution for dealing with feature flags in local development and automated testing in a microservice architecture. It shows how you can use Roxy, a Docker image that mocks CloudBees Feature Management storage and provides REST API to control flag behavior on local development or for automated testing (creating flag behavior fixture, controlling the flag values, etc…).

Understanding feature flags in the microservice architecture

What are microservices?

"In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."

— James Lewis and Martin Fowler

Contrary to common belief, feature flags should not be a standalone service in your environment.

Adding a feature flagging microservice does not follow the guidelines of microservice architecture and introduces the following architectural flaws:

The Feature Flags service is not a vertical service - choosing the flag service approach is more suitable for SOA and not microservices architecture
Feature flags evaluation latency is dependant on the network topography and feature flags service load
Other services are not independently deployable because of coupling with feature flags microservices

Some of these issues can be solved by an implementing a caching layer on the consumer microservices, but this solution introduces more complexity and fragility into the system. It is better to choose the right architecture, a distributed one.

In a distributed solution the Feature Flags SDK is installed on the relevant microservice, the SDK uses CloudBees Feature Management storage to get the flags configuration but a connection is not required when evaluating flags.

Here is a diagram that demonstrates this architecture:

Microservices architecture diagram

As you can see, the SDK is installed into the microservices and fetches the configuration from the CloudBees Feature Management storage. Calculating whether a flag is enabled/disabled is done in local memory with a distributed algorithm, avoiding the architectural flaws described above.

The issue with automated testing

Not Unit testing

This section does not describe how to unit test, it is focused on automated acceptance testing that is done on the entire microservice environment (or parts of it).

The above diagram shows the desired architecture for a feature flag enabled environment. As you can see the Feature Flag SDK runs on multiple instances of multiple microservices. Each microservice consumes different or shared flags.

When writing tests for this environment, you first need to set up the test fixtures. The test fixtures includes setting up various components, loading relevant data into the database and setting flag values.

When we set flag values we want to know as little as possible about the system, to eliminate any dependencies to implementation that will make the test fragile.

In order to do this, CloudBees Feature Management has a component called Roxy. Roxy is a Docker image that mocks CloudBees Feature Management storage and provides a REST API to control flags behavior in a non production environment.

The issue with local development

When developing a service, the developer is often required to set a flag value for their specific localhost environment, these flags can be consumed on the service they are developing or on other services in their environment.
It is not the developer’s concern to understand which flags are consumed by which microservice and how many instances each microservice is running. To hide these implementation details from the developer it is required to have a single point of abstraction to set up flags value across the environment

To allow the developer to control the flags values on their development environment, CloudBees Feature Management has released a component called Roxy. Roxy is a Docker image that mocks CloudBees Feature Management storage and provides a REST API to control flag behavior in a non-production environment.

Roxy architecture view

Roxy replaces CloudBees Feature Management storage and runs from inside your domain, in practice it supplies a mock service on top of CloudBees Feature Management software as a service solution. Here is how Roxy fits into a microservice architecture.

As can be seen above, the CloudBees Feature Management SDK that is running on each microservice fetchs configuration from Roxy instead of CloudBees Feature Management storage.

Running Roxy

Roxy is distributed as a Docker image. Roxy listens on port 3333
Here is how you run it with the Docker command line:

docker run -p 4444:3333 -d rollout/roxy:latest

This command will start Roxy inside the container and will expose port 4444 as Roxy port. The next step is to configure CloudBees Feature Management SDK to work with Roxy as its configuration source.

Redirecting the SDK to Roxy

Configure the CloudBees Feature Management SDK to work with Roxy as its configuration source:

RoxOptions options = new RoxOptions.Builder()
  .withRoxyURL(new URL("http://localhost:4444"))
  .build();
Rox.setup(<ROLLOUT_KEY>, options);

const options = {
  roxy: 'http://localhost:4444'
}

Rox.setup("<ROLLOUT_KEY>", options);

import {Rox} from 'rox-ssr';

const options = {
  roxy: 'http://localhost:4444'
}

Rox.setup('<ROLLOUT_KEY>', options);

var Options = new RoxOptions(new RoxOptions.RoxOptionsBuilder
{
    RoxyURL = new Uri("http://localhost:4444")
}

Rox.Setup("<ROLLOUT_KEY>", Options);

options = RoxOptions(
            roxy_url = 'http://localhost:4444/'
)

Rox.setup("<ROLLOUT_KEY>", options)

options := server.NewRoxOptions(server.RoxOptionsBuilder {
  RoxyURL = 'http://localhost:4444/' },
)

rox.setup("<ROLLOUT_KEY>", options)

option = Rox::Server::RoxOptions.new(
            roxy_url = 'http://localhost:4444/'
)

Rox::Server::RoxServer.setup("<ROLLOUT_KEY>", option).join

$roxOptionsBuilder = (new RoxOptionsBuilder())
    ->setRoxyURL("http://localhost:4444/");

Rox::setup("<ROLLOUT_KEY>",  new RoxOptions($roxOptionsBuilder));

int main(int argc, char **argv) {
    RoxOptions *options = rox_options_create();
    rox_options_set_roxy_url(options, "http://localhost:4444/");

    rox_setup("<ROLLOUT_KEY>", options);
    rox_shutdown();
}

int main(int argc, char **argv) {
    Rox::Options *options = Rox::OptionsBuilder()
        .SetRoxyUrl("http://localhost:4444/")
    .Build();

  Rox::Setup("<ROLLOUT_KEY>", options);
    Rox::Shutdown();
}

After setting this withRoxyUrl configuration the SDK will fetch its configuration from localhost:4444

Supported on SDK 3.2.0 and higher

This withRoxyUrl configuration is supported on Java SDK from version 3.2.0

Controlling flags via REST API

Roxy supports the following REST API for setting flags values:

GET /flags/<flagname> - get flag behavior
GET /flags/ - get all flags behavior
POST /flags/<flagname> - set flag behavior within body. To set a boolean flag to true send body: { "expression": "true" }, to set a boolean flag to false send body: { "expression": "false" }, to set a string flag to "example" send body: { "expression": "\"example\"" }
DELETE /flags/<flagname> - Reset flag behavior
DELETE /flags/ - Reset all flags behavior

The API is also available via a Swagger interface at http://localhost:4444/api-docs.

Example: Updating `newFeature` boolean flag to true

curl --header "Content-Type: application/json" \
     --request POST \
     --data '{ "expression": "true" }' \
     http://localhost:4444/flags/newFeature

Example: Updating `newFeatureName` string flag to "example"

curl --header "Content-Type: application/json" \
     --request POST \
     --data '{ "expression": "\"example\"" }' \
     http://localhost:4444/flags/newFeatureName