The Future of the Modern Data Stack in 2022
Photo by Mike Kononov on UnsplashThe 2023 version of this article is out! Read it here. As the data world slowed down for the holidays, I got some downtime to step back and think about the last year. And I can’t help but think, wow, what a year it’s been! Is it just me, or did data go through five years’ worth of change in 2021? It’s partially COVID time, where a month feels like a day and a year
The importance of layered thinking in data engineering
Looks a bit like a data lake right? (Tangled wires by Cory Doctorow on Flickr (CC BY-SA 2.0) )Who is this for?Are you a data scientist or data engineer keen to build sustainable and robust data pipelines? Then this article is for you! We’ll walk through a real-world example and by the end of this article you’ll understand why you need a layered data engineering convention to avoid the mistakes we
Beyond Pandas: Spark, Dask, Vaex and other big data technologies battling head to head
WhyWhen confronting a new data science problem, one of the first questions to ask is which technology to use. There is hype; there are standard tools; there are bleeding-edge technologies, entire platforms and off-the-shelf solutions. Over the last few years, I’ve been building proof of concepts and solutions with any technology I could get my hands on. If there is a new platform, I register for t
Advanced SQL Templates In Python with JinjaSql
In A Simple Approach To Templated SQL Queries In Python, I introduced the basics of SQL templates in Python using JinjaSql. This post further demonstrates the power of Jinja2 within JinjaSql templates using presets, loops, and custom functions. Let’s consider an everyday use case when we have a table with some dimensions and some numerical values, and we want to find some metrics for a given dimen
5 lesser-known pandas tricks
Pandas provides high-performance, easy-to-use data structures and data analysis tools for the Pythonpandas needs no introduction as it became the de facto tool for data analysis in Python. As a Data Scientist, I use pandas daily and I am always amazed by how many functionalities it has. In this post, I am going to show you 5 pandas tricks that I learned recently and using them helps me to be more
Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store
TLDR; Most machine learning models are trained using data from files. This post is a guide to the popular file formats used in open source frameworks for machine learning in Python, including TensorFlow/Keras, PyTorch, Scikit-Learn, and PySpark. We will also describe how a Feature Store can make the Data Scientist’s life easier by generating training/test data in a file format of choice on a file
ML Ops: Machine Learning as an Engineering Discipline
So, your company decided to invest in machine learning. You have a talented team of Data Scientists churning out models to solve important problems that were out of reach just a few years ago. All performance metrics are looking great, the demos cause jaws to drop and executives to ask how soon you can have a model in production. It should be pretty quick, you think. After all, you already solved
Apache Airflow Tips and Best Practices
Photo by JJ Ying on UnsplashWhen I first started building ETL pipelines with Airflow, I had so many memorable “aha” moments after figuring out why my pipelines didn’t run. As the tech documentation never covers everything, I tend to learn much more about a new tool from making mistakes and reading source code from good developers. In the blog post, I will share many tips and best practices for Air
How to analyse 100s of GBs of data on your laptop with Python
Many organizations are trying to gather and utilise as much data as possible to improve on how they run their business, increase revenue, or how they impact the world around them. Therefore it is becoming increasingly common for data scientists to face 50GB or even 500GB sized datasets. Now, these kind of datasets are a bit… uncomfortable to use. They are small enough to fit into the hard-drive of
How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions
How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions When comes to machine learning, data is certainly the new oil. The processes for managing the lifecycle of datasets are some of the most challenging elements of large scale machine learning solutions. Data ingestion, indexing, search, annotation, discovery are some of the aspects r
Data Science in daily life
(Source)I usually make sure I am not the “Late Person.” Even sometimes I couldn’t get to a meetup ahead of time; I get there on time. Like most of the people, I assume, I check the google map beforehand, planning well when to start. However, you can never avoid all kinds of the incident that get you stuck on the way. Lately, I just added a new event on my list. Just moved to Seattle for summer, fo
Scalable Python Code with Pandas UDFs: A Data Science Application
PySpark is a really powerful tool, because it enables writing Python code that can scale from a single machine to a large cluster. While libraries such as MLlib provide good coverage of the standard tasks that a data scientists may want to perform in this environment, there’s a breadth of functionality provided by Python libraries that is not set up to work in this distributed environment. While l
Best Python libraries for Machine Learning and Data Science : (Part -1)
Python is becoming popular day by day and has started to replace many popular languages in the industry. The main reason for Python’s popularity is because of the following reasons: Python is popularly known as Beginner’s language because of it’s simplicity.Python helps developers to be more productive from development to deployment and maintenance.Python’s syntax is very simple and high level whe
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
NumPy Basics Cheat Sheet (2021), Python for Data Science
How to visualize hidden relationships in data with Python — analysing NBA assists
Python ETL Tools: Best 8 Options
How to Boost your Python Apps using Httpx and Asynchronous Calls
Flying high with Vaex: analysis of over 30 years of flight data in Python
Speeding up Python Code: Fast Filtering and Slow Loops
Pandas for Football Analysis