Running Condenser, Jussi and a new service locally + adding feature flags to Condenser

Intro

We're part of the team working on Steemit.com, and wanted to share some of our recent work getting the site ready for what will be a very busy 2018.

Benjamin Chodoroff (github | steemit)
Iain Maitland (github | steemit)

The work discussed here also covers contributions from other steem employees and the amazing community.

Abstract

The frontend for steemit.com is Condenser, a universal React app. This codebase gets updated all the time, with bugfixes and new features. Sometimes, these features are so complex that the team wants to only enable them for a small subset of users to see how they work and to allow users to give us valuable feedback into how the change works or does not work for them.

To do this, we needed to set up a "feature flag" system: for logged-in users, while the site is being rendered, the client asks the server which special features should be turned on, and the client will either enable or disable those features for that particular user.

On the server side, we added some Conveyor API endpoints to let users ask for their feature flags, and let developers or management define new feature flags as well as the probability that a flag is enabled for a user.

On the client side, we created a system where a new feature can be contained in a component and decide whether or not that component gets rendered based on the user's feature flag state, which is inserted in our Redux store when a user starts their session.

Getting all of this written & doing integration tests is a bit outside of our usual workflow, working mostly with Condenser -- we aren't often rolling out new backend API endpoints at the same time the corresponding frontend functionality is being defined. In cases like this, developers need to be able to run a local instance of the backend service as well as Condenser. Sitting in between Condenser and the new backend service is Jussi, our JSON-RPC call router, which can be configured to route most calls to the normal prod services (like steemd) and only route certain calls to our local Conveyor dev server.

Developers working with Condenser will occasionally run into circumstances when they need to run local services with Jussi, so we documented the steps here:

Goals

We're going to get Conveyor working on top of Jussi. This will involve having Condenser, Jussi and Conveyor all running simultaneously on our dev machine.

Jussi is a proxy for many JSON-RPC services related to the Steem blockchain.

Conveyor is a JSON-RPC micro service for Condenser configuration.

Condenser is the Steemit.com react app, with toggle-able components dictated by Conveyor.

Set up Jussi

Clone Jussi and apply this diff - this preps Jussi for local conveyor development:

git apply jussi_local_conveyor_config.diff

There is one environment specific edit that's not covered by this diff.
In PROD_UPSTREAM_CONFIG.json, for the config:

["conveyor","http://192.168.11.3:8090"]

You need to specify an IP that is accessible to the Jussi docker image, i.e. your wireless card's or docker vlan's address.

To discover this, in the terminal run ifconfig, in the case of Docker's Vlan, the IP is identified at dockerO under inet, something like 172.17.0.1.

If you run conveyor with make devserver it will default to port 8090, therefore in this example case we edit PROD_UPSTREAM_CONFIG.json:

["conveyor","http://172.17.0.1:8090"]

Build Jussi

Build Jussi's Docker Image -- this step takes a while!

docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" .

Run Jussi.

docker run -itp 9000:8080 "$USER/jussi:$(git rev-parse --abbrev-ref HEAD)"

Condenser usually claims port 8080, so we ask Docker to make the Jussi container port 8080 to 9000 on the host.

Set up Condenser

Apply this diff to Condenser:

git apply condenser_local_conveyor_config.diff

This diff will tell Condenser to talk to our local Jussi instead of api.steemit.com.

Run condenser

SDC_DATABASE_URL="mysql://root:[email protected]/steemit_dev" NODE_ENV=development node ./node_modules/babel-cli/bin/babel-node.js  ./webpack/dev-server.js

Your database connection string will probably be a bit different ;)

Set up Conveyor

Conveyor uses the node-config module; per the load order we can create a local.toml config like so:

cat>config/local.toml<<EOF
admin_role = 'foobarman'
rpc_node = 'http://localhost:9000/'
EOF

where foobarman is some existing steem username that you have the private posting key for, and the rpc_node is the api for the steem blockchain we will be authenticating that user against (our local Jussi).

Run Conveyor

make devserver

Set up some feature flags.

All being well, Jussi, Conveyor, and Condenser are all up and running locally. It's time to test out the feature-flags functionality we are interested in.

Download the rpcit command line tool for making rpc requests.

rpcit is a command line tool used to make API requests to a JSON-RPC server like Jussi.

yarn global add rpcit

To make privileged calls, like the ones needed to interact with Conveyor's feature flags endpoints, you must supply a private key & username to be used in signing the requests. You can find your private keys at https://steemit.com/@accountname/permissions - note the 'private' posting key needs to be toggled on to be displayed.

Add a feature flag

RPCIT_KEY=wif RPCIT_ACCOUNT=foobarman rpcit -a http://localhost:9000 -s conveyor.set_feature_flag_probability testflag :0.5

This creates a feature flag called 'testflag' with a 50% probablity that it will resolve to true for a given user.

Test it out

Navigate to the locally running instance of Condenser in your browser. The pertinent call to conveyor happens in the user saga. So to test we need to be a logged in user. As a logged in user, refreshing the page should trigger a call from Condenser to get currently enabled feature flags from Conveyor (via Jussi). We can watch the Conveyor logs in the terminal to determine if this is happening.

    rpc_req: {
      "id": 554497485567665,
      "method": "conveyor.get_feature_flags"
    }

Wrap a Condenser feature in a flag.

Now we can wrap front-end code in the <ConnectedFlag /> component to have it conditionally display depending on what the given flag resolves to. In the event that the flag resolves to false or if it is not found due to some error, a fallback component is rendered.

Demo usage:

// Conditionally render any number of wrapped Children
<ConnectedFlag
    flag="testflag"
    Fallback={<LoadingIndicator/>}
>
    <h1> Hello World </h1>
</ConnectedFlag>

// Explicitly Render a singular component.
<ConnectedFlag
    flag="testflag"
    FlagComponent={<Icon name="user" />}
    Fallback={<LoadingIndicator/>}
/>

// If flag is false or not present, render a fallback
<ConnectedFlag
    flag="testflag"
    FlagComponent={<Icon name="user" />}
    Fallback={<LoadingIndicator/>}
/>

This can be tested by applying this diff:

git apply condenser_demo.diff

and navigating to http://localhost:8080/faq.html where 'Hello World' will appear whenever the flag named testflag evaluates to true (this will be the case for 50% of users based on our RPC command issued above) and a loading indicator will appear when it does not.

Conclusion

Thanks for reading! If you have any questions leave a comment & we'll do our best to respond :)

Benjamin Chodoroff (github | steemit) & Iain Maitland (github | steemit)