Keeping background jobs boring on a self-hosted server

I run more of my own servers than is sensible, and for a long time background jobs were the part that bit me most. Digests that didn't send, image resizing that stalled, cleanup tasks that quietly stopped running after an upgrade. Here's the setup that has kept things boring for the last few months.

One cron entry, checked

UNA runs its scheduled tasks from a single cron entry every minute. The mistake I kept making was assuming it was running. Now a tiny health check records the time of the last successful run on a private admin page, and if it's more than ten minutes old I get a notification on my phone.

Small jobs, not big ones

Our old digest job tried to build and send every email in one go. When it failed halfway, nobody knew who had already received theirs. We split it so each member's digest is its own small job in a queue. A failure now affects one person, not the whole network.

Retries with patience

• Retry a failed job three times, waiting a little longer each time.
• After the third failure, park it and tell a human.
• Never retry something that isn't safe to repeat (see my webhook post from this morning).

Upgrade days

Before an upgrade I pause the queue, let it drain, upgrade, then start it again and watch the first few runs. It adds ten minutes and has saved me from at least two half-finished jobs.

You don't need fancy tools for any of this. A small VPS, one cron line and the habit of checking that it actually ran will get a community a very long way.