はてなブックマーク

Introduction We are standing at the intersection of language and machines. I’m fascinated by this topic. Can a machine write as well as Shakespeare? What if a machine could improve my own writing skills? Could a robot interpret a sarcastic remark? I’m sure you’ve asked these questions before. Natural Language Processing (NLP) also aims to answer these questions, and I must say, there has been grou

XGBoost is a machine learning algorithm that belongs to the ensemble learning category, specifically the gradient boosting framework. It utilizes decision trees as base learners and employs regularization techniques to enhance model generalization. XGBoost is famous for its computational efficiency, offering efficient processing, insightful feature importance analysis, and seamless handling of mis

Introduction If you have spent some time in machine learning and data science, you would have definitely come across imbalanced class distribution. This is a scenario where the number of observations belonging to one class is significantly lower than those belonging to the other classes. This problem is predominant in scenarios where anomaly detection is crucial like electricity pilferage, fraudul

One of the most fundamental questions in the field of reinforcement learning for scientists across the globe has been – “How to learn a new skill?”. The desire to understand the answer is obvious – if we can understand this, we can enable human species to do things we might not have thought before. Alternately, we can train machines using reinforcement learning to do more “human” tasks and create

Introduction One of the most common questions we get on Analytics Vidhya is, How much maths do I need to learn to be a data scientist? Even though the question sounds simple, there is no simple answer to the the question. Usually, we say that you need to know basic descriptive and inferential statistics to start. That is good to start. But, once you have covered the basic concepts in machine learn

Introduction ‘Time’ is the most important factor which ensures success in a business. It’s difficult to keep up with the pace of time.  But, technology has developed some powerful methods using which we can ‘see things’ ahead of time. Don’t worry, I am not talking about Time Machine. Let’s be realistic here! I’m talking about the methods of prediction & forecasting. One such method, which deals wi

Introduction Imagine you get a dataset with hundreds of features (variables) and have little understanding about the domain the data belongs to. You are expected to identify hidden patterns in the data, explore and analyze the dataset. And not just that, you have to find out if there is a pattern in the data – is it signal or is it just noise? In this scenario, employing techniques like t-SNE coul

Introduction The power of artificial intelligence is beyond our imagination. We all know robots have already reached a testing phase in some of the powerful countries of the world. Governments, large companies are spending billions in developing this ultra-intelligence creature. The recent existence of robots have gained attention of many research houses across the world. Does it excite you as wel

Importance of Regular Expressions In last few years, there has been a dramatic shift in usage of general purpose programming languages for data science and machine learning. This was not always the case – a decade back this thought would have met a lot of skeptic eyes! This means that more people / organizations are using tools like Python / JavaScript for solving their data needs. This is where R

Creating accurate predictive models is a fundamental task in data analysis. It involves splitting data into training and test sets and applying statistical models or machine learning algorithms. Linear regression is a popular choice, but it often faces the challenge of overfitting, especially with a high number of parameters. This is where ridge and lasso regression comes in, offering practical so

Overview Deep dive into the concept of recommendation engine in python Building a recommendation system in python using the graphlab library Explanation of the different types of recommendation engines Introduction This could help you in building your first project! Be it a fresher or an experienced professional in data science, doing voluntary projects always adds to one’s candidature. My sole re

Introduction If things don’t go your way in predictive modeling, use XGboost. XGBoost algorithm has become the ultimate weapon of many data scientists. It’s a highly sophisticated algorithm, powerful enough to deal with all sorts of irregularities of data. It uses parallel computation in which multiple decision trees are trained in parallel to find the final prediction. This article is best suited

Complete Machine Learning Guide to Parameter Tuning in Gradient Boosting (GBM) in Python Overview Learn parameter tuning in gradient boosting algorithm using Python Understand how to adjust bias-variance trade-off in machine learning for gradient boosting Introduction If you have been using GBM as a ‘black box’ till now, maybe it’s time for you to open it and see, how it actually works! This artic

Introduction Time Series (referred as TS from now) is considered to be one of the less known skills in the data science space (Even I had little clue about it a couple of days back). I set myself on a journey to learn the basic steps for solving a Time Series problem and here I am sharing the same with you. These will definitely help you get a decent model in any future project you take up! Before

One of the most common question, which gets asked at various data science forums is: What is the difference between Machine Learning and Statistical modeling? I have been doing research for the past 2 years. Generally, it takes me not more than a day to get clear answer to the topic I am researching for. However, this was definitely one of the harder nuts to crack. When I came across this question

Mastering Python’s Set Difference: A Game-Changer for Data Wrangling

Introduction One of the most interesting and challenging things about data science hackathons is getting a high score on both public and private leaderboards. I have closely monitored the series of data science hackathons and found an interesting trend. cross validation using python and R trend is based on participant rankings on the public and private leaderboards. One thing that stood out was th

Mastering Python’s Set Difference: A Game-Changer for Data Wrangling

Overview Learn web scraping in Python using the BeautifulSoup library Web Scraping is a useful technique to convert unstructured data on the web to structured data BeautifulSoup is an efficient library available in Python to perform web scraping other than urllib A basic knowledge of HTML and HTML tags is necessary to do web scraping in Python Introduction The need and importance of extracting dat

Introduction In his famous book – Think and Grow Rich, Napolean Hill narrates story of Darby, who after digging for a gold vein for a few years walks away from it when he was three feet away from it. Now, I don’t know whether the story is true or false. But, I surely know of a few Data Darby around me. These people understand the purpose of machine learning, its execution and use just a set 2 – 3

Introduction Tree based algorithms are considered to be one of the best and mostly used supervised learning methods. Tree based algorithms empower predictive models with high accuracy, stability and ease of interpretation. Unlike linear models, they map non-linear relationships quite well. They are adaptable at solving any kind of problem at hand (classification or regression). Methods like tree m

Google’s self-driving cars and robots get a lot of press, but the company’s real future is in machine learning, the technology that enables computers to get smarter and more personal. Eric Schmidt (Google Chairman) We are probably living in the most defining period of human history. The period when computing moved from large mainframes to PCs to the cloud. But what makes it defining is not what ha

Introduction This was in my first year of engineering degree. A hungry, home-food sick student (me) was treated (by a college senior) with a lavish buffet in one of the best five star hotels in Mumbai! You get served with so many dishes that you struggle to decide where to start, what to taste and what to eat! Why is this relevant here? Well, I had a similar feeling when I looked at the videos fro

Analytics Vidhya is the leading community of Analytics, Data Science and AI professionals. We are building the next generation of AI professionals. Get the latest data science, machine learning, and AI courses, news, blogs, tutorials, and resources.

Introduction Machine learning algorithms often require tuning to achieve optimal performance. This article focuses on the importance of tuning Random Forest and understanding the key random forest parameters, a popular ensemble learning method. The author shares a personal experience of significantly improving their Kaggle competition ranking through random forest tuning parameters. Random Forest,

Introduction The beauty of art lies in the message it conveys. At times, reality is not what we see or perceive. The endless efforts from the likes of Vinci and Picasso have tried to bring people closer to the reality using their exceptional artworks on a certain topic/matter. Data scientists are no less than artists. They make paintings in form of digital visualization (of data) with a motive of

Introduction PyCon(s) carry a benevolent motive of helping the Python community worldwide by providing extensive knowledge resources. I started following PyCon conferences from 2013. My first learning experience from PyCon tutorials & workshops inspired me to follow it back in the year 2014 and this craze continued in 2015 as well. You can check out the training recommendation for tutorials of Pyc

Introduction It happened a few years back. After working on SAS for more than 5 years, I decided to move out of my comfort zone. Being a data scientist, my hunt for other useful tools was ON! Fortunately, it didn’t take me long to decide – Python was my appetizer. I always had an inclination for coding. This was the time to do what I really loved. Code. Turned out, coding was actually quite easy!