Scheduled Tasks: The Good Way (or: Why You Should Stop Using Crontab)
Managing scheduled tasks is crucial part for most of the advanced software projects out there. At Appwrite, we use background tasks for sending emails, cleaning cache, aggregating stats, creating backups, and more app crucial processes.
When we first discussed how we are going to implement background tasks in our app, we wanted to make sure our architecture is simple, reliable, and fault tolerance. Managing background tasks can be hard, and there are several reasons that made CRON server management not suitable for our needs.
TL;DR
- We found Crontab as not suitable for advanced production setups
- We used Redis with a Rescue scheduler to manage our tasks in a queue
- We integrated our solution in Appwrite to allow any developer to take advantage of it
- Monitoring — Monitoring background scripts is usually hard just by the fact they run in the background. There is no endpoint to ping, and most of the time, developers only monitor the scripts’ output log.
- Fault Tolerance — Managing failover for a CRON script is hard just by the fact a single-threaded process usually handles them.
- Logging — This one is pretty easy to implement, but as long as no one view or monitor theses logs, it isn’t very beneficial. Extracting the logs and alerting for problems requires extra work.
- Developer Accessibility — Crontab is a system-level process not an application process. It’s very challenging to give application developers access to the system level. We wanted our developers to have easy access to both add and review our application background jobs.
- Deployment — We needed to deploy code changes the same way we release new versions. Managing live code updates and having no version control system for our tasks is not an option for us.
- Race conditions — What happens when one task starts when previous one hasn’t finished running yet? We need to decide whether tasks can run asynchronously or synchronously.
Redis Resque for the Rescue!
We decided we want to use a queue server for managing our tasks. By using a queue messaging architecture, we can make sure a failed task will be retried, this way we can also scale up the number of workers we use to process jobs.
We chose to use Redis & Redis Rescue for managing our queue. We already used Redis for in-memory cache, and we felt comfortable with it. Using Rescue on top of Redis also gave us the power to schedule our queue jobs according to our scheduling needs easily. Another great things about Redis is that is combines flexibility and simplicity really well.
Another great thing about Redis is that it combines flexibility and simplicity well.
As with everything we do at Appwrite, we are not only trying to solve common and complex development problems, we are trying to simplify them and make our solutions accessible for all developers. This is why we added a new dashboard on the Appwrite console that allows us to manage our tasks with a simple UI. This way, every developer using Appwrite as a backend server can also add his own background timed tasks with literally — a click of a button.
All we needed to create a new task is to add a new protected endpoint to our API. Using this method, we can manage our entire app and tasks codebase from a single place and take advantage of our existing development workflow, version control, tests, CI, CD, and container orchestration.
Managing tasks as an integral part of our code repository helps us write high-quality background tasks that are treated as a first-class citizen in our apps and projects.
Anyone Can Use Our Task Manager(!)
Appwrite is an open-source backend server packaged as a set of docker containers that you can install on any host or cloud service you want with a single command. Appwrite offers several services to help developers build apps faster, and our tasks manager is only one of them.
To use our task manager and other backend APIs, all you need to do is run our docker-compose installation script:
This script will set up all the required docker container Appwrite requires to run, and in 2 minutes you will have Appwrite dashboard and all of its service ready to run.
Once ready, you can just go to the tasks module on the Appwrite dashboard and define your new tasks to run at any recurring schedule you need.
To set a new task, all you need to do is define an HTTP endpoint, HTTP method, and optionally a basic HTTP authentication to protect your API endpoint from unauthorized access.
Appwrite task manager supports a standard CRON syntax to define your task schedule. If you’re not familiar with the CRON syntax, this tutorial is a good place to understand how to define a CRON schedule quickly.
Appwrite tasks also help us log and count our task errors. We can also use our error count to alert or stop our tasks execution at a predefined threshold and avoid our server from exploding with errors.
Summary
Using tasks management with a well-architected infrastructure and a well defined modern coding process helped us create a much more stable and fault tolerance applications. Our new process is really easy to understand, and any developer in our team can easily adopt it.
Learn More
Appwrite is an open-source project that is 100% driven by its community. You can learn more about Appwrite features, API, and integration by visiting Appwrite official website or Github repository. You can also follow Appwrite news and announcements on our Twitter or Facebook accounts. You can also chat with us on our Discord server.
To learn more about how to use Appwrite tasks you can also use the Appwrite’s official docs on that matter.
About Me
I am Eldad Fux, a Software Architect, CTO, open-source enthusiastic, and the creator of appwrite.io. You can follow me on my Twitter account: https://twitter.com/eldadfux or connect with me via LinkedIn: https://www.linkedin.com/in/eldadfux/.
