But despite all of the advantages of running background jobs, under real world usage you can still run into challenging situations that require thoughtful handling. One general class of problems that can arise is achieving fairness in resource usage across users.
Many web interfaces let a user effortlessly page through large sets of data. Implementing database queries that fetch these pages is also effortless for the programmer, usually requiring an
LIMIT in the case of SQL and a
SIZE in the case of Elasticsearch. Although this method is easy on the user and programmer, pagination queries of this type have a high hidden cost for SQL, Elasticsearch and other database engines.
At Salsify, we encountered this problem when implementing a feature to allow a user to step through records in a large, heavily filtered and sorted set. We had to implement an efficient pagination solution that would work in both our SQL and ES datastores. In this post we’ll look at an interesting technique to make pagination efficient for the database with large datasets. Specifically, we’ll look at implementation in SQL as well as how to translate this method to Elasticsearch.
Using vim has been one of the greatest productivity boosters for my development life. I got into vim as a lowly system administrator because it seemed to be the tool of the trade. From there, my knowledge grew and now it is my editor of choice for almost all projects. This post will go through my current setup for Ruby on Rails development. I'd like to give a huge shout out to @tpope who has created a plethora of amazing plugins all worth making a part of your daily workflow.
By Matthew Cross on Nov 5, 2015 8:15:00 AM
Here at Salsify, many of our customers regularly import large amounts of tabular product data into the system. This data needs to be sorted prior to being handled by different parts of the import process. Since we are running on Heroku, memory is a scarce resource. Sorting these arbitrarily large tabular data files requires great care. Reading all of the data into memory at once can result in extremely long execution times due to increased pressure on the Ruby garbage collector and can starve other processes on the same system of memory.
We needed a way to sort large files using a predictable amount of memory. Big data technologies like map reduce were overkill for our data scale. Creating an offline-sort gem to do this turned out to be quite the adventure and forced me to dig deeper into how Ruby manages memory, ultimately requiring a specialized heap implementation.
At Salsify we run most of our Ruby on Rails based infrastructure in Heroku. We recently switched our Puma web workers over from Heroku 2X dynos to more powerful Heroku PX dynos and saw dramatic performance improvements: a 51% reduction in mean response time and a 59% reduction in 99th percentile response time. Based on this we did some experiments running our more memory/CPU intensive Delayed Jobs on PX dynos and saw a similarly encouraging 43% reduction in mean job execution time and 66% reduction in 99th percentile job execution time. This was great for a proof of concept, but only one of the eight cores on the PX dyno was being used. In order to take this to production in a cost-effective manner, we had to figure out how to utilize all of the cores on the dyno.
At Salsify, we manage the flow of product information between manufacturers, and distributors through to large retail outlets such as Walmart and Google Shopping. Our flexible schema allows users to describe their products however they see fit. We store product attributes in a single table, but each product attribute has a potentially different data type and associated application-specific logic.
When debugging software, particularly if it's code you didn't write and aren't familiar with, one of the hardest parts can be just determining where to start. Without the right context, you can end up chasing red herrings all over before finally tracking down the underlying problem. Well-placed log messages can help provide that context, saving that time and frustration. In this post, we'll look at adapting a logging technique employed by many Node tools on the command line for use in an Ember.js app in the browser.
By Randy Burkes on Oct 28, 2014 2:18:28 PM
Salsify is a multi-tenant SaaS product information exchange platform hosted in Heroku. We utilize delayed job extensively for long running tasks like import/export/search indexing/etc. One of the features we've grown to love about delayed job is its extensibility via plugins (see our other posts). Recently, an increasing number of job workers have been exceeding their dyno memory quota**, and consequently suffering serious performance degradation. While we are always working to improve our code performance, we thought it would be nice to have a mechanism for euthanizing workers that are unrecoverably exhausted. Enter the DelayedJob::YouthInAsiaPlugin.