Dennis Dang

July 13, 2024

Language-agnostic background job systems where?!

There is no language-agnostic distributed background job system that’s easy to use. Please, someone build it!

The existing products out there that are pretty great:
  1. Celery (python)
  2. Sidekiq (Ruby)
  3. Jobrunr (Java)
  4. Quartz (Java)
  5. Forget what the C# options are
  6. Oban (elixir) 
  7. Bullmq (nodejs)
  8. Faktory (agnostic) but poor client library support.

Faktory gets the closest, and truly only agnostic option I’ve seen, but there aren’t any complete client library for the JVM.

A good job system should do many things:
  1. Work in a distributed environment where there may be many concurrent workers.
  2. Persist jobs
  3. Retry jobs
  4. Acknowledge job completion or failure
  5. Have a GUI / web UI for easy actionable items
  6. Have the other niceties like scheduling, concurrency limits, idempotentcy checks.
  7. And last but not least, be easily deployable via docker or binary. Even better if I can just include the library in my app as code like Oban and Jobrunr does.

Now some may point: What about durable execution engines like Temporal? Too expensive to use their managed version, it’s $200. And do I really also want to pay the 4-5 services it takes to host Temporal? My friend Cole suggested I could take the temporal dev server to production - and that’s a valid suggestion, but I’m also not ready to really manage any service aside from my web server. 

However, durable engines are the future imo. Maybe we’ll get a language agnostic https://flawless.dev one day:

So what did I pick on the JVM? None of the above. 

I deployed LavinMQ on cloudampq because that’s easier than setting up SQS and Google pubsub AND I get to run LavinMQ in docker. I wrote some code to publish my tasks to a queue, consume the queue, and called it a day. I don’t need the fancy scheduling stuff atm. I still don’t have a nice pattern to distribute tasks, but it’s okay! I’m running a single worker instance apart from my web server, and in that worker, I handle rate limiting and concurrency in-memory. And lastly, ensure all jobs are idempotent for easy retries.

Dennis