Solved: getting the number of missing values in pandas

Pandas is a widely used open-source data manipulation library for Python. It provides data structures and functions needed to effectively manipulate and analyze large datasets. One common problem data scientists and analysts encounter while using pandas is handling missing values in the dataset. In this article, we will explore how to count the number of missing values in a pandas DataFrame using various techniques, step-by-step explanations of the code, and delve deeper into some of the libraries and functions involved in solving this problem.

Counting Missing Values in Pandas

To begin, we need to first import the pandas library. If you haven’t installed it yet, simply run the command `pip install pandas` in your terminal or command prompt.

import pandas as pd

Once we have imported the pandas library, let’s create a sample DataFrame with missing values, which we’ll use throughout this article to demonstrate different techniques of counting missing values.

data = {
    'Name': ['Anna', 'Ben', 'Carla', None, 'Eva'],
    'Age': [25, None, 30, 35, None],
    'City': ['NY', 'LA', None, 'SF', 'LA']
}

df = pd.DataFrame(data)

In this example, we have a DataFrame with three columns: Name, Age, and City. There are some missing values, which we will find and count in the next section.

Finding and Counting Missing Values using isnull() and sum()

The first method to count missing values in a pandas DataFrame is by using the isnull() function. This function returns a DataFrame of the same shape as the original, but with True or False values indicating whether the corresponding entry is missing (i.e., contains None or NaN) or not.

missing_values = df.isnull()

Now we have a DataFrame of the same shape, with True values indicating missing entries. To count these missing values, we can simply use the sum() function. By using it over the DataFrame, we can get the number of missing values for each column.

count_missing_values = df.isnull().sum()

This will give us a pandas Series with the number of missing values for each column in our DataFrame.

Alternative Approach: Using isna() and sum()

Another approach to count missing values in a pandas DataFrame is by using the isna() function. It’s an alias for isnull() and works in the same way.

count_missing_values = df.isna().sum()

This will give the same result as the previous approach, counting the number of missing values for each column in our DataFrame.

Counting Missing Values in the Entire DataFrame

If we want to find the total number of missing values in the entire DataFrame, we can simply chain another sum() function after the first sum() function.

total_missing_values = df.isnull().sum().sum()

This will return the total number of missing values in the entire DataFrame.

In summary, handling missing values in pandas is a crucial step in the data cleaning and pre-processing phase. By using the isnull() or isna() functions, in combination with the sum() function, we can efficiently count the number of missing values in our DataFrame, making it easier to address and manage missing data issues in our analysis.