A Two Sigma engineer explains how how Ibis works with BigQuery to provide an elegant and flexible Python interface for composing SQL queries.
The author walks through how to build a metrics system for a high performance data platform, taking a look at some of the important factors to consider when choosing what open source offerings to use.
Distributed transactions suffer from poor performance due to two major limiting factors. First, distributed transactions suffer from high latency because each of their accesses to remote data incurs a long network delay. Second, this high latency increases the likelihood of contention among distributed transactions, leading to high abort rates and low performance. The authors present Sundial, an in-memory distributed optimistic concurrency control protocol that addresses these two limitations.
Two Sigma’s Co-Founder discusses algorithmic investing, self-driving cars, blockchain, education, and more.
Two Sigma security researcher Trammell Hudson presents his work on the new LinuxBoot firmware—now an official Linux Foundation project.
The NERF and Heads projects bring Linux back to the cloud servers' boot ROMs by replacing nearly all of the vendor firmware with a reproducible built Linux runtime that acts as a fast, flexible, and measured boot loader.
The authors provide a solution for the Efficient Server Audit Problem based on several new techniques, including simultaneous replay and efficient verification of concurrent executions, implemented for PHP web applications.
The author presents CelFS, Two Sigma’s geo-distributed file system. Although CelFS has scaled to serve tens of petabytes of data, it uses physical partitioning to provide quality of service guarantees, it has a high replication overhead, and cannot take advantage of outsourced cold storage The talk further describes our response to these limitations in Jaks, a new storage system to reduce the TCO of CelFS and serve as the backend for other systems at Two Sigma.
This presentation discusses the design and implementation of Smooth at Two Sigma, our experience running it over the past two years, ongoing challenges, and future directions.