Benchmarking a database YCSB has never heard of

JanusGraph on FoundationDB, and the binding I had to write from scratch before any measurement was possible.

I needed to benchmark JanusGraph running on a FoundationDB storage backend. YCSB is the standard tool for exactly this kind of measurement, and it ships bindings for a long list of databases. It does not ship one for JanusGraph. So the benchmark had to be built before any benchmarking could happen.

This was graph database research under Prof. Shahram Ghandeharizadeh, as part of an eBay and USC collaboration on transactional graph databases. The question was how JanusGraph behaves when its storage layer is FoundationDB rather than the usual suspects.

The stack, and why each layer is there

Four layers, and the binding in the middle is the piece that did not exist.

YCSB generates load and measures it. It talks to a database through a binding: a class implementing its datastore interface, mapping YCSB's five operations onto whatever the target actually understands. For a key-value store that mapping is nearly free. For a graph database, less so, since YCSB thinks in records and JanusGraph thinks in vertices, properties, and edges.

Getting the cluster to exist at all

Before any of that, the two systems had to actually form a cluster on multi-node CloudLab machines, and they were not keen.

The failures were the flavour that teaches you the most and feels the worst at the time: client library version mismatches, and network binding faults where a service was listening on a loopback interface and therefore invisible to the other nodes. Nothing crashed cleanly. Things just did not connect.

End-to-end validation went through Gremlin traversals over Apache TinkerPop, and only once those ran from a remote node did I trust the cluster enough to benchmark it.

Writing the binding

The binding turned out to be four separate pieces of work, and only the first is the one people expect:

A new Maven module, because YCSB expects each binding to live as its own module in the reactor build.
A client implementing the YCSB datastore interface, translating insert, read, update, delete, and scan into graph operations.
Binding registration, so the launcher can resolve the name you pass on the command line to the class you wrote.
Packaging changes, which is where most of the time actually went. Getting the launcher to find the module at runtime is a classpath and assembly problem, not a coding problem, and it fails in ways that look like your code is missing.

The scan operation is where the abstraction leaks hardest. In a key-value store, scan is a range read over ordered keys. In a graph, there is no inherent ordering to scan across, so the operation has to be defined rather than merely implemented, and whatever you choose shapes the numbers you get out.

The schema is not a formality

JanusGraph needs its schema declared: vertex labels, property keys, and crucially the indexes. I wrote these in Groovy, adapted from the BG benchmark schema.

schema.groovygroovy

mgmt = graph.openManagement()
userId = mgmt.makePropertyKey('userId').dataType(String.class).make()
user   = mgmt.makeVertexLabel('user').make()

// composite index: without this, a lookup by userId is a full scan,
// and the benchmark measures the wrong thing entirely
mgmt.buildIndex('byUserId', Vertex.class).addKey(userId).buildCompositeIndex()
mgmt.commit()

Index status transitions and reindexing are the part that catches people. A JanusGraph index is not usable the moment you create it. It moves through states, and you have to wait for it to become available and sometimes trigger a reindex over existing data. Benchmark before that completes and you will measure unindexed lookups while believing you measured indexed ones.

What I would tell anyone doing this

When a benchmark has no binding for your database, that absence is information. It usually means the mapping is genuinely awkward, and the awkward parts are where you have to make judgement calls that affect your results.
Write down the decisions the mapping forced on you. Anyone comparing your numbers to theirs needs to know how you defined scan.
Budget real time for packaging and classpath work. It is not the interesting part and it is frequently the majority of the calendar.
Verify the index is live before trusting a single measurement.

The binding lives in a public repo, linked from my projects. It is not a polished library, it is research tooling. But if you are pointing YCSB at JanusGraph, it is a starting point that did not exist before.