You can use the pandas notnull() function to test whether or not elements in a pandas DataFrame are null.
If an element is equal to NaN or None, then the function will return False.
Otherwise, the function will return True.
Here are several common ways to use this function in practice:
Method 1: Filter for Rows with No Null Values in Any Column
df[df.notnull().all(1)]
Method 2: Filter for Rows with No Null Values in Specific Column
df[df[['this_column']].notnull().all(1)]
Method 3: Count Number of Non-Null Values in Each Column
df.notnull().sum()
Method 4: Count Number of Non-Null Values in Entire DataFrame
df.notnull().sum().sum()
The following examples show how to use each method in practice with the following pandas DataFrame:
import pandas as pd import numpy as np #create DataFrame df = pd.DataFrame({'team': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'], 'points': [18, 22, 19, 14, 14, 11, 20, np.nan], 'assists': [5, np.nan, 7, 9, 12, 9, 9, np.nan], 'rebounds': [11, 8, 10, 6, 6, 5, np.nan, 12]}) #view DataFrame print(df) team points assists rebounds 0 A 18.0 5.0 11.0 1 B 22.0 NaN 8.0 2 C 19.0 7.0 10.0 3 D 14.0 9.0 6.0 4 E 14.0 12.0 6.0 5 F 11.0 9.0 5.0 6 G 20.0 9.0 NaN 7 H NaN NaN 12.0
Example 1: Filter for Rows with No Null Values in Any Column
The following code shows how to filter the DataFrame to only show rows with no null values in any column:
#filter for rows with no null values in any column
df[df.notnull().all(1)]
team points assists rebounds
0 A 18.0 5.0 11.0
2 C 19.0 7.0 10.0
3 D 14.0 9.0 6.0
4 E 14.0 12.0 6.0
5 F 11.0 9.0 5.0
Notice that each of the rows in this filtered DataFrame have no null values in any column.
Example 2: Filter for Rows with No Null Values in Specific Column
The following code shows how to filter the DataFrame to only show rows with no null values in the assists column:
#filter for rows with no null values in the 'assists' column
df[df[['assists']].notnull().all(1)]
team points assists rebounds
0 A 18.0 5.0 11.0
2 C 19.0 7.0 10.0
3 D 14.0 9.0 6.0
4 E 14.0 12.0 6.0
5 F 11.0 9.0 5.0
6 G 20.0 9.0 NaN
Notice that each of the rows in this filtered DataFrame have no null values in the assists column.
Example 3: Count Number of Non-Null Values in Each Column
The following code shows how to count the number of non-null values in each column of the DataFrame:
#count number of non-null values in each column
df.notnull().sum()
team 8
points 7
assists 6
rebounds 7
dtype: int64
From the output we can see:
- The team column has 8 non-null values.
- The points column has 7 non-null values.
- The assists column has 6 non-null values.
- The rebounds column has 7 non-null values.
Example 4: Count Number of Non-Null Values in Entire DataFrame
The following code shows how to count the number of non-null values in the entire DataFrame:
#count number of non-null values in entire DataFrame
df.notnull().sum().sum()
28
From the output we can see there are 28 non-null values in the entire DataFrame.
Additional Resources
The following tutorials explain how to perform other common filtering operations in pandas:
How to Filter a Pandas DataFrame by Column Values
How to Filter for “Not Contains” in Pandas
How to Filter a Pandas DataFrame on Multiple Conditions