Author: Cecilia Ye
Presented at: 2017 SHPE Conference
Abstract: The ability to make sense of large amounts of data from disparate sources in real-time is valuable to diverse use cases in many fields. Designing a system that can extract immediate insights from large amounts of data in real-time requires a special way of thinking. Real-time data applications must handle data in motion and extract information as soon as data arrives, and have different design concerns than traditional data platforms designed for manipulating data at rest.
How exactly should you design a production-grade real-time analytics application capable of providing timely insights? Reactive computing systems, in which data flows respond automatically to propagating changes, is ideally suited for use cases requiring real-time analytics and continuous computation.
This talk describes how reactive programming principles and the actor model provide an effective framework for building streaming analytics platforms. Specific topics covered by the talk include:
- Introduction to the concept of reactive programming;
- Description of how the actor model supports building reactive data applications that are responsive, resilient, scalable, and message-driven;
- Presentation of our real-word experience with building a reactive data application. In particular, the presentation highlights the architecture of a high-performance analytics engine that continuously calculates statistics and metrics on-the-fly from live streams of data;
- Considerations and challenges when applying a reactive framework to a concrete implementation; solutions that can help address those challenges.
