Buckets & Self-Healing

This page covers how JobMaster partitions work across workers and recovers automatically when one crashes — useful once you're configuring or scaling a cluster. The system is designed to maximize concurrency while completely eliminating database deadlocks.

What is a Bucket?

A bucket is the fundamental logical unit of concurrency. Instead of every worker in your cluster competing for individual jobs (which causes massive database locking), workers create and own Buckets.

Atomic Locking: When a worker owns a bucket, it takes exclusive ownership of every job inside it for that cycle.

Parallelism: If you have 10 workers and 10 buckets, each worker takes one bucket, and they all work in perfect parallel without ever touching the same data.

Configuration (BucketQtyConfig): You define the number of buckets per priority.

High Volume: More buckets = Higher parallelism.

Heavy Jobs: Fewer buckets = Less resource strain per worker.

builder.Services.AddJobMasterCluster(config =>
{
    config.ClusterId("My-Cluster");
    ...
        
    config.AddWorker()
        .AgentConnName("MyPostgresAgent-1")
        .BucketQtyConfig(JobMasterPriority.VeryLow, 1)
        .BucketQtyConfig(JobMasterPriority.Low, 2)
        .BucketQtyConfig(JobMasterPriority.Medium, 3)
        .BucketQtyConfig(JobMasterPriority.High, 4)
        .BucketQtyConfig(JobMasterPriority.Critical, 5);
});

For how individual jobs move through their own state machine (PendingSave → OnMaster/InBucket → ... → Succeeded/Failed), see Job Lifecycle. The rest of this page covers the lifecycle of the buckets that host them.

Bucket Life Cycle

In JobMaster, Buckets are not static; they have their own life cycle governed by the health of the Agent Workers. This state machine ensures that even if a server crashes, no job is ever lost in the system.

Bucket Status Definitions

Status	Phase	Description
Active	Operation	The normal operating state. The bucket is owned by a healthy Agent Worker and is actively onboarding and processing new jobs.
Completing	Graceful Exit	Proactive Shutdown. The bucket stops onboarding new jobs but continues to execute current work and syncs all PendingSave items.
ReadyToDrain	Idle	All in-flight jobs have been returned to the Master DB and the bucket is idle, waiting for a drain runner to process any remaining OnMaster jobs.
Lost	Hard Crash	Reactive Recovery. An Agent Worker stopped heartbeating unexpectedly. The bucket is orphaned, and its contents must be rescued.
Draining	Recovery	A drain runner has claimed the bucket (either a ReadyToDrain bucket or a healthy worker adopting a Lost one) and is physically moving jobs back to the Master DB (OnMaster) for redistribution across the cluster.
ReadyToDelete	Finalization	The bucket is empty and all states are synced. It is now a "tombstone" awaiting permanent deletion from the Agent storage.

---

The "Graceful Exit" Flow (Completing)

When an Agent Worker is signaled to shut down (e.g., during a deployment), it moves its buckets into the Completing state. This is a critical feature for Zero-Downtime Architecture:

1. Block Intake: The bucket immediately stops pulling new jobs from the Master Database.
2. Finish Active Work: Jobs currently in Processing or Queued are allowed to complete naturally.
3. Flush Persistence: Every job marked as PendingSave (not yet in the Master DB) is prioritized for a final sync to ensure no data is lost.
4. Ready to Drain: Once idle, the bucket moves to ReadyToDrain until a drain runner processes any remaining OnMaster jobs.
5. Clean Retirement: Once fully drained and empty, the bucket becomes ReadyToDelete and the Agent Worker shuts down safely.

---

The "Orphan" Rescue Flow (Lost → Draining)

If an Agent Worker crashes or loses network connectivity, JobMaster heals the cluster automatically:

1. Detection: The system identifies a missing heartbeat and marks the affected buckets as Lost.
2. Takeover: A healthy Agent Worker claims the orphaned bucket.
3. Redistribution: The healthy worker enters Draining mode, moving all unfinished jobs back to the Master Database.
4. Re-Onboarding: These jobs return to the OnMaster status, where they will be naturally picked up by active buckets on other healthy workers.

---

The Fallback Bucket (Last-Resort Safety Valve)

If a job has no Active bucket to claim it — matching its priority/lane, owned by a live worker — for more than 2.5 minutes, the Coordinator creates a temporary fallback bucket so the job isn't starved indefinitely. This is almost always a configuration gap (nothing set up for that priority/lane, or every worker serving it is currently down), not a load spike. It's created at the highest possible priority, starting from Critical down to Medium (which can never be disabled), skipping any priority the cluster has disabled via DisablePriority.

The fallback bucket is a real bucket like any other: it's persisted to the Master DB through a dedicated, automatically-managed reserved agent connection, so jobs assigned to it survive a Coordinator restart instead of being lost. If the worker that created it goes offline, the fallback bucket is cleaned up automatically once it's confirmed orphaned — you don't need to manage it manually.

Seeing fallback buckets activate is a sign to check your BucketQtyConfig and worker/lane assignments — see Performance Tuning.

See: Workers Configuration