There is no language-agnostic distributed background job system that’s easy to use. Please, someone build it!
The existing products out there that are pretty great:
The existing products out there that are pretty great:
- Celery (python)
- Sidekiq (Ruby)
- Jobrunr (Java)
- Quartz (Java)
- Forget what the C# options are
- Oban (elixir)
- Bullmq (nodejs)
- Faktory (agnostic) but poor client library support.
Faktory gets the closest, and truly only agnostic option I’ve seen, but there aren’t any complete client library for the JVM.
A good job system should do many things:
- Work in a distributed environment where there may be many concurrent workers.
- Persist jobs
- Retry jobs
- Acknowledge job completion or failure
- Have a GUI / web UI for easy actionable items
- Have the other niceties like scheduling, concurrency limits, idempotentcy checks.
- And last but not least, be easily deployable via docker or binary. Even better if I can just include the library in my app as code like Oban and Jobrunr does.
Now some may point: What about durable execution engines like Temporal? Too expensive to use their managed version, it’s $200. And do I really also want to pay the 4-5 services it takes to host Temporal? My friend Cole suggested I could take the temporal dev server to production - and that’s a valid suggestion, but I’m also not ready to really manage any service aside from my web server.
However, durable engines are the future imo. Maybe we’ll get a language agnostic https://flawless.dev one day:
So what did I pick on the JVM? None of the above.
I deployed LavinMQ on cloudampq because that’s easier than setting up SQS and Google pubsub AND I get to run LavinMQ in docker. I wrote some code to publish my tasks to a queue, consume the queue, and called it a day. I don’t need the fancy scheduling stuff atm. I still don’t have a nice pattern to distribute tasks, but it’s okay! I’m running a single worker instance apart from my web server, and in that worker, I handle rate limiting and concurrency in-memory. And lastly, ensure all jobs are idempotent for easy retries.
Dennis