<img height="1" width="1" src="https://www.facebook.com/tr?id=973228766067554&amp;ev=PageView &amp;noscript=1">
Screen_Shot_2014-05-30_at_1.37.33_PM

THE SALSIFY SMARTER ENGINEERING BLOG

Fancy trees with botanist

By Keith Williams on Jun 14, 2017 2:21:28 PM

Have you ever wanted to build your own calculator, query language, or even web browser? Parsers and Transformers are tremendously useful for these applications and many more, and thanks to tools like Salsify's own Botanist you don't need to be an expert in compiler design to work with them.

Read More →

Adventures in Avro

By Tim Perkins on Jun 13, 2016 8:11:57 AM
As part of our microservices architecture we recently adopted Avro as a data serialization format. In the process of incorporating Avro we created a Ruby DSL for defining Avro schemas, developed a gem for generating models from Avro schemas, and reimplemented an Avro schema registry in Rails. Here’s how we got there ... 
 
Here’s a situation that may be familiar: at Salsify we are moving towards a microservices architecture. Have you done that too? Are you thinking about doing it? This is a pretty common progression for startups that built a monolithic application first, found great market fit and then need to scale both the application and the team. At Salsify, we already have quite a few services running outside of the original monolith, but several months ago we started to define an architecture for how we should chip away at the monolith and structure new core services.
 
Naturally part of the architecture we are defining is how services should communicate with each other. For synchronous communication, we decided to stick with HTTP REST APIs that speak JSON. For asynchronous communication, we selected Apache Kafka.
 
We evaluated several data serialization formats to use with Kafka. The main contenders were Apache Avro, JSON, Protocol Buffers and Apache Thrift. For asynchronous communication we wanted a data format that was more compact and faster to process. Asynchronous data may stick around longer and the same message may be processed by multiple systems so version handling was important. A serialization system should also provide additional benefits like validation and strong typing.
 
At this point, I should inject that we’re primarily a Ruby shop. We love our expressive, dynamic language of choice, so a big factor in the selection process was how well the framework integrates with Ruby. Based on the title of this post, it's not going to be any surprise which option was the winner ...

Read More →

Delayed Job Queue Fairness

By Robert Kaufman on May 23, 2016 1:06:13 PM
 
At Salsify, we use Delayed Job extensively to handle asynchronous tasks. This works well for us, as it means we can finish web requests faster, resulting in a more responsive web app, while offloading non-urgent tasks to background jobs. For the most part, Delayed Job (and similar job queuing mechanisms like Resque, Celery, etc.) provide a simple and highly effective approach for running background work, making sure that it gets done, and providing a framework to scale your compute resources to handle expected workloads. Even beyond that, these frameworks create straightforward opportunities for dynamically scaling resources to handle spikes in workload. For example, we use an excellent service called HireFire to dynamically scale our Delayed Job worker pools based on queued work. Meaning, we can meet the needs of changing workload while keeping our hosting costs reasonable.

But despite all of the advantages of running background jobs, under real world usage you can still run into challenging situations that require thoughtful handling. One general class of problems that can arise is achieving fairness in resource usage across users. 

Read More →

Recent Posts