はてなブックマーク

Dask and Pandas and XGBoost Playing nicely between distributed systems This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation Summary This post talks about distributing Pandas Dataframes with Dask and then handing them over to distributed XGBoost for training. More generally it discusses the value of launching multiple dis

This work is supported by Continuum Analytics the XDATA Program and the Data Driven Discovery Initiative from the Moore Foundation Summary Dask Dataframe extends the popular Pandas library to operate on big data-sets on a distributed cluster. We show its capabilities by running through common dataframe operations on a common dataset. We break up these computations into the following sections: Intr

This post compares two Python distributed task processing systems, Dask.distributed and Celery. Disclaimer: technical comparisons are hard to do well. I am biased towards Dask and ignorant of correct Celery practices. Please keep this in mind. Critical feedback by Celery experts is welcome. Celery is a distributed task queue built in Python and heavily used by the Python community for task-based w

This work is supported by Continuum Analytics and the XDATA Program as part of the Blaze Project In this post we analyze weather data across a cluster using NumPy in parallel with dask.array. We focus on the following: How to set up the distributed scheduler with a job scheduler like Sun GridEngine. How to load NetCDF data from a network file system (NFS) into distributed RAM How to manipulate dat