A challenge often faced in multi-tenant SaaS is ensuring that each tenant gets a fair share of a platform's resources. At Salsify, we had to address this in our Delayed Job-based background task execution infrastructure. Because our customers have different use cases, they tend to run tasks of varying complexity and size. Over time we developed tenant-fairness job reservation strategies that made scaling our job system difficult if not impossible. In this post, I discuss how we managed to extend Delayed Job to solve for tenant fairness in a scalable manner.
One of the blessings and curses of Rails development is the ability to use Rails console for debugging issues and inspecting data. The console is oftentimes used in a production environment, as it is the quickest method to glean information about any problems. With great power comes great responsibility: A command that attempts to reset a local developer environment by deleting all records of a model could easily be input into a production console. In addition, with potentially unknown clipboard data (thanks "copy-on-select"), any valid Ruby code with line breaks will automatically be executed if pasted. Any user who is deleting models, queuing up events, and updating records accidentally or intentionally should be made aware of the implications. At Salsify, we've solved this problem using a combination of open-source and home-brewed improvements and rolled them into a handy gem. Read on to learn more!
But despite all of the advantages of running background jobs, under real world usage you can still run into challenging situations that require thoughtful handling. One general class of problems that can arise is achieving fairness in resource usage across users.