Skip to main content

Benchmark Methodology

How the JobMaster vs. Hangfire comparison is set up, so the numbers on that page can be read (and challenged) with full context.

Fair concurrency, not default concurrency​

JobMaster's per-worker execution concurrency comes from its bucket model: each bucket gets a fixed base capacity per priority level (Medium = 5 concurrent execution slots), multiplied by ParallelismFactor. Every JobMaster worker in the baseline run has 2 Medium-priority buckets, so each worker has 2 × 5 = 10 concurrent job-execution slots.

Hangfire has no equivalent bucket/priority concept. A single BackgroundJobServer just runs WorkerCount jobs concurrently, defaulting to Environment.ProcessorCount × 5 if left unset. That default has nothing to do with JobMaster's setting and would silently bias the comparison toward whichever number happened to come out higher. To keep the comparison honest, Hangfire's WorkerCount is set explicitly to match JobMaster's real per-worker capacity for each config tested (10 at the baseline, 20 when JobMaster is tuned up to 4 buckets/worker) rather than relying on either framework's own unrelated default.

Both frameworks also run the same number of worker containers (20), so both systems have the same total in-flight job capacity throughout the entire test.

Scope: same server for master and agent connections​

Every JobMaster config in this comparison points the master connection and every agent connection at the same database server (for example, one PostgreSQL server for both, or one RavenDB server for both). JobMaster supports routing master and agent connections to different servers, including different providers entirely (for example, a PostgreSQL master with NATS as the agent-side transport), but we deliberately kept it to a single server per config for this round: introducing a second moving part on top of the database swap would make it harder to attribute any result to one variable or the other, and this comparison is meant to be a straightforward apples-to-apples one first. Split-server topologies like PostgreSQL + NATS are planned for a follow-up benchmark, not included here.

Resource limits​

Every container, database and worker alike, has an explicit CPU/memory cap, not the host's full uncapped resources, so every framework and database engine competes for the same fixed budget:

  • Database container: 3 CPU / 6 GB
  • Worker container: 0.25 CPU / 0.5 GB

Test hardware​

All runs execute on AWS EC2 m5.4xlarge instances (16 vCPU / 64 GB RAM), launched from the same prebuilt AMI. Each full 3x-repeat cycle for a given config runs on a different instance than the previous cycle used for that config, rotating across a pool of 5 identical instances rather than always benchmarking the same framework on the same physical box. This avoids one instance's own quirks (noisy neighbor, degraded hardware, etc.) silently biasing one framework's numbers across the whole comparison.

Burst-mode test shape​

Each run fires a fixed number of jobs (25,000) all at once, split across all worker containers in parallel: 60 concurrent POST /schedule-now calls firing simultaneously, rather than a steady arrival rate. This isolates two distinct things instead of conflating them into one number:

  • Scheduling throughput: how fast the system accepts and durably persists the whole burst, measured by the slowest of the concurrent schedule calls to return (all of them start at once, so the phase ends when the last one finishes).
  • Execution throughput: how fast, once accepted, jobs are actually picked up and completed by workers, measured from run start to the last real completion, a separate and usually much longer phase than scheduling itself.

Repetition​

Every result is the average of 3 repeated runs, not a single run. Single-run comparisons for this kind of test turned out to carry more noise than expected during testing. In one case, a single run made a configuration look meaningfully slower than baseline, and a 3x-repeat average revealed that result was just noise, not a real effect. No result on this site is reported without repetition.

Bucket tuning ("baseline" / "2x execution")​

Beyond the fixed baseline config, a follow-up round doubles --buckets (how many Medium-priority buckets each worker runs) to see how it trades off against scheduling throughput. It's labeled "2x execution" rather than "tuned" because it isn't a universal improvement: it's a deliberate trade of scheduling throughput for execution throughput, not a strictly better configuration. The baseline uses 2 buckets (2 × 5 = 10 concurrent execution slots per worker, see above); doubling to 4 buckets doubles that to 20 slots per worker.

For a fair comparison at 4 buckets, Hangfire's WorkerCount is raised to 20 to match JobMaster's new 20-slot-per-worker capacity, the same principle as the baseline's WorkerCount 10 matching 2 buckets.

Planned follow-up: a more comfortable worker CPU allocation​

Exploratory local testing (not part of the published comparison above and not yet 3x-repeated) found that this page's 0.25 CPU/worker allocation is on the austere side for a .NET workload. This is a general .NET-on-containers behavior (the runtime sizes its thread pool and GC off the container's own CPU quota, not the host's full core count), not something specific to JobMaster or Hangfire, but it does mean 0.25 CPU under-represents how either framework performs with a comfortable amount of CPU actually available. Giving workers real headroom (2 CPU, based on internal testing so far) raised execution throughput several-fold in exploratory runs and scaled cleanly, with 5 workers running comfortably up to 150,000 jobs in a single burst.

A follow-up benchmark round at 2 CPU / 4 GB per worker (3 CPU / 6 GB database), scaled up to 20 workers on a dedicated machine, is planned with the same 3x-repeat rigor as the rest of this page, and will be published once complete. Nothing from this exploratory round is reflected in the published numbers on this site today.