Many web interfaces let a user effortlessly page through large sets of data. Implementing database queries that fetch these pages is also effortless for the programmer, usually requiring an OFFSET and LIMIT in the case of SQL and a FROM and SIZE in the case of Elasticsearch. Although this method is easy on the user and programmer, pagination queries of this type have a high hidden cost for SQL, Elasticsearch and other database engines.

At Salsify, we encountered this problem when implementing a feature to allow a user to step through records in a large, heavily filtered and sorted set. We had to implement an efficient pagination solution that would work in both our SQL and ES datastores. In this post we’ll look at an interesting technique to make pagination efficient for the database with large datasets. Specifically, we’ll look at implementation in SQL as well as how to translate this method to Elasticsearch.

Have you ever noticed that no one writes "How we name ourRuby variables at Company X" blog posts? No one's making the Hacker News front page with "I combined these two strategies for method naming and suddenly my JavaScript is maintainable!" And yet when it comes to CSS, developers are all about naming strategies. We sing the praises of BEM, SMACSS, OOCSS, SUIT, or whatever other set of capital letters is popular this week. Why is that?

Introduction

Using vim has been one of the greatest productivity boosters for my development life. I got into vim as a lowly system administrator because it seemed to be the tool of the trade. From there, my knowledge grew and now it is my editor of choice for almost all projects. This post will go through my current setup for Ruby on Rails development. I'd like to give a huge shout out to @tpope who has created a plethora of amazing plugins all worth making a part of your daily workflow.

Here at Salsify, many of our customers regularly import large amounts of tabular product data into the system. This data needs to be sorted prior to being handled by different parts of the import process. Since we are running on Heroku, memory is a scarce resource. Sorting these arbitrarily large tabular data files requires great care. Reading all of the data into memory at once can result in extremely long execution times due to increased pressure on the Ruby garbage collector and can starve other processes on the same system of memory.

We needed a way to sort large files using a predictable amount of memory. Big data technologies like map reduce were overkill for our data scale. Creating an offline-sort gem to do this turned out to be quite the adventure and forced me to dig deeper into how Ruby manages memory, ultimately requiring a specialized heap implementation.

At Salsify we run most of our Ruby on Rails based infrastructure in Heroku. We recently switched our Puma web workers over from Heroku 2X dynos to more powerful Heroku PX dynos and saw dramatic performance improvements: a 51% reduction in mean response time and a 59% reduction in 99th percentile response time. Based on this we did some experiments running our more memory/CPU intensive Delayed Jobs on PX dynos and saw a similarly encouraging 43% reduction in mean job execution time and 66% reduction in 99th percentile job execution time. This was great for a proof of concept, but only one of the eight cores on the PX dyno was being used. In order to take this to production in a cost-effective manner, we had to figure out how to utilize all of the cores on the dyno.

When debugging software, particularly if it's code you didn't write and aren't familiar with, one of the hardest parts can be just determining where to start. Without the right context, you can end up chasing red herrings all over before finally tracking down the underlying problem. Well-placed log messages can help provide that context, saving that time and frustration. In this post, we'll look at adapting a logging technique employed by many Node tools on the command line for use in an Ember.js app in the browser.

Salsify is a multi-tenant SaaS product information exchange platform hosted in Heroku. We utilize delayed job extensively for long running tasks like import/export/search indexing/etc. One of the features we've grown to love about delayed job is its extensibility via plugins (see our other posts). Recently, an increasing number of job workers have been exceeding their dyno memory quota**, and consequently suffering serious performance degradation. While we are always working to improve our code performance, we thought it would be nice to have a mechanism for euthanizing workers that are unrecoverably exhausted. Enter the DelayedJob::YouthInAsiaPlugin.