Introducing Flint: A Time-Series Library for Apache Spark
An overview of Flint, an open source library Two Sigma developed to enhance Spark’s functionality for time series analysis.
Insights: python
Ibis and BigQuery: Scalable Analytics with the Comfort of Python
A Two Sigma engineer explains how how Ibis works with BigQuery to provide an elegant and flexible Python interface for composing SQL queries.
Best Practices from Building a Machine Learning Bot for Halite
An overview of best practices derived from building a machine-learning based starter bot for Halite, Two Sigma’s public artificial intelligence programming challenge.
The Future of Pandas
Architecture overview for the future of the Python Pandas data analytics library.
BeakerX (for PyData NYC)
An overview of BeakerX, a collection of kernels and extensions to the Jupyter interactive computing platform.
Introducing Pandas UDFs for PySpark
A Two Sigma researcher introduces the Pandas UDFs feature in the upcoming Apache Spark 2.3 release, which substantially improves the performance and usability of user-defined functions (UDFs) in Python.
Improving Python and Spark Performance and Interoperability with Apache Arrow
Apache Arrow-based interconnection between various big data tools (SQL, UDFs, machine learning, big data frameworks, etc.) enables you to use them together seamlessly and efficiently,
A Workaround for Non-Determinism in TensorFlow
Speed and repeatability are crucial in machine learning, but the latter is not guaranteed in TensorFlow. A Two Sigma researcher demonstrates a workaround to attain repeatable results.