Jan Tytgat
Friday, October 4, 2024
Developing a job scheduler in go
Table of contents
History
| Date | Changes |
|---|---|
| 2024/10/04 | Initial publication |
Introduction
For some tools I am working on related to Citrix and NetScaler, I needed the capability to schedule jobs or commands at a fixed time or at an interval. While technically possible to use “cron” on Linux/BSD-based operating systems, and “scheduled tasks” on Windows, I wanted to provide the capability through the tool itself when running as a daemon or windows service.
At the same time, I saw this as a great opportunity to dive more into concurrency with Go. This is supposed to be the first post in a series on the different packages in this module.
While I still consider myself a novice in Go, I am not sure splitting the different parts into separate packages is idiomatic Go. However, I consider them to be addressing separate concerns so it made sense for me. Specifically the cron and the task packages can be used independently of the others. Hit me up through GitHub if you think something might be improved, I am totally open to constructive feedback!
Source
The source code for this project is publicly available on GitHub with an open source license.
Documentation
The series of blog posts is intended to explain and reason about the decisions I’ve made during development, obstacles I’ve run into and so on. It’s not my intention to provide full documentation here, for that I will refer to the documentation in the repository.
Tests
At the time of writing, the project is still under heavy development. This means that when you go and look at the code in the repository right now, you might find a lack of tests and coverage reports. This will be addressed in due time, when I’ve been able to identify the actual design I’m working towards. Again, I do not consider myself to be a professional programmer, so cut me some slack if you feel the order in which I’m doing things is wrong according to test-driven desing. 😅
Packages
Overview
So far, the following packages have been identified:
| package | description |
|---|---|
| cron | Definition of a cron schedule and the functionality for triggering actions |
| task | Implementation of a task, the task handler and a handler pool for providing concurrency to handle tasks |
| job | A job (being a sequence of tasks), a catalog in memory to store the jobs |
| orchestrator | Combining the functionality in the other packages to embed in a real program |
These packages will be explored in detail in individual blog posts.
Cron
What makes cron job tick? 😉
More specifically, how do you define a cron schedule? Because the schedule for a cron job is vastly different compared to a scheduled task on Windows, we’ll provide some background on cron. From there, we can dive into the specification, what the different fields are and how we can tie it all together.
We’re going to explore the functionality of checking for a valid timestamp on the schedule, and how to get notified by a trigger.
Task
Obviously, I needed something more than just calling another executable present on the host system. If that was the only use case, I would happily settle for cron jobs and scheduled tasks.
However, my use case is geared towards “complex” jobs, involving multiple steps to be taken programmatically. While this would techically be possible using cron or scheduled tasks, it would be a very convulated way of implementing it. And I would still need individual executables to perform the task at hand.
We’ll explore what a task is, how we can handle that specific task and how we can make sure the same task (for different jobs) can run in parallel.
At the same time, there might be constraints on a specific task in terms of concurrency, so we need to take care of that as well.
As a last part, the goal is to keep the relationship between a task and its handler function as frictionless as possible.
Job
A job can be a simple task, or a sequence of tasks which need to be completed. This part of the project will define how we go about handling jobs in parallel, where we store the results and how we handle failures in between tasks.
Orchestrator
Finally, we need something to stitch it all together.
The orchestrator is the heart of everything, literally orchestrating all the “moving parts”, from keeping track of jobs in a catalog to scheduling, triggering, queueing and running the jobs at the right time. We’ll also explore how I can use all of it in a real application (which will also be the topic of another series of blog posts).
🤩 STAY TUNED !! 🤩