Responsive and Scalable Real-time Data Analytics

Posted on November 2, 2017

Author: Cecilia Ye

Presented at: 2017 SHPE Conference

AbstractThe ability to make sense of large amounts of data from disparate sources in real-time is valuable to diverse use cases in many fields. Designing a system that can extract immediate insights from large amounts of data in real-time requires a special way of thinking. Real-time data applications must handle data in motion and extract information as soon as data arrives, and have different design concerns than traditional data platforms designed for manipulating data at rest.

How exactly should you design a production-grade real-time analytics application capable of providing timely insights? Reactive computing systems, in which data flows respond automatically to propagating changes, is ideally suited for use cases requiring real-time analytics and continuous computation.

This talk describes how reactive programming principles and the actor model provide an effective framework for building streaming analytics platforms. Specific topics covered by the talk include:

  • Introduction to the concept of reactive programming;
  • Description of how the actor model supports building reactive data applications that are responsive, resilient, scalable, and message-driven;
  • Presentation of our real-word experience with building a reactive data application. In particular, the presentation highlights the architecture of a high-performance analytics engine that continuously calculates statistics and metrics on-the-fly from live streams of data;
  • Considerations and challenges when applying a reactive framework to a concrete implementation; solutions that can help address those challenges.

This article is not an endorsement by Two Sigma of the papers discussed, their viewpoints or the companies discussed. The views expressed above reflect those of the authors and are not necessarily the views of Two Sigma Investments, LP or any of its affiliates (collectively, “Two Sigma”). The information presented above is only for informational and educational purposes and is not an offer to sell or the solicitation of an offer to buy any securities or other instruments. Additionally, the above information is not intended to provide, and should not be relied upon for investment, accounting, legal or tax advice. Two Sigma makes no representations, express or implied, regarding the accuracy or completeness of this information, and the reader accepts all risks in relying on the above information for any purpose whatsoever. Click here for other important disclaimers and disclosures.