How Akuko uses Temporal to manage critical background jobs
by John Mwashuma
At Akuko, we pride ourselves on providing a seamless experience for our users, whether they're exploring data visualizations, interacting with maps, or performing complex analyses. Behind the scenes, a lot of processes ensure that the data is fresh, the maps are responsive, and everything runs smoothly. To manage these critical background tasks, we use a powerful tool called Temporal.
What is Temporal?
Temporal is a robust workflow orchestration engine that helps us manage long-running processes, background jobs, and complex workflows with ease. Think of it as a conductor in an orchestra, ensuring that all the instruments (in this case, tasks) play in harmony, even if one or more instruments need to pause or restart. Whether a task needs to be retried, delayed, or run concurrently with others, Temporal.io handles it gracefully.
Why We Chose Temporal.io
At Akuko, we deal with a variety of tasks that are crucial to the performance and reliability of our platform. For example:
Uploading and Processing Sources: One of the most critical operations we perform is uploading and processing sources. These sources could be large datasets, geographical information, or any other form of data that powers the visualizations and insights on Akuko.io. Temporal handles the complexity of uploading, validating, and processing these sources, ensuring that they are integrated into our platform smoothly and efficiently.
Baking Tiles: This involves generating and updating map tiles that are served to users when they interact with our maps. It's a resource-intensive process that needs to be managed carefully to ensure that maps load quickly and accurately.
Clearing Cache: We maintain caches of data to speed up responses to users. However, these caches need to be cleared and updated regularly to ensure that users always have access to the most up-to-date information.
These tasks need to be done reliably and efficiently, without causing disruptions to the user experience. Temporal provides the perfect solution by allowing us to define, schedule, and manage these tasks with high reliability and visibility.
How It Works
Here's a simplified breakdown of how we use Temporal:
Workflows: Temporal allows us to define workflows, which are like detailed instructions for completing a task. For example, a workflow for baking tiles might involve fetching the latest data, generating the tiles, and then updating the map service with the new tiles. For uploading sources, a workflow might include steps like receiving the uploaded data, performing validation checks, processing the data, and then making it available for use.
Activities: Within each workflow, there are activities—discrete units of work that need to be performed. These activities can be things like retrieving data from a database, processing it, or sending a notification when the task is complete.
Resilience: One of the standout features of Temporal is its resilience. If an activity fails—say, due to a network issue —Temporal automatically retries it. If a job needs to be paused and resumed later, Temporal can handle that too. This ensures that our background tasks are completed successfully, even in the face of temporary setbacks.
Visibility: Temporal also gives us detailed visibility into the state of our workflows. This means we can monitor tasks in real-time, see where they might be getting stuck, and take action if needed. This transparency helps us keep everything running smoothly and ensures that issues are resolved quickly.
Benefits for Our Users
For our users, the use of Temporal translates to a more reliable and responsive platform. Maps load faster, data is always up-to-date, and everything just works, even if there are challenges behind the scenes. This reliability is critical for users who depend on Akuko.io for accurate data and insights.
Conclusion
Temporal is an essential part of our infrastructure at Akuko.io, enabling us to manage complex workflows and background tasks with ease. Whether we're baking map tiles, clearing caches, uploading and processing sources, or handling any other critical process, Temporal ensures that everything runs smoothly and efficiently. This, in turn, allows us to deliver the best possible experience to our users.
We believe that the technology we use should serve our users, not complicate their lives. With Temporal, we're able to keep that promise, providing a platform that's both powerful and dependable.
