Use “Not Equal” Operator in PySpark (With Examples)

---

There are two common ways to filter a PySpark DataFrame by using a “Not Equal” operator:

Method 1: Filter Using One “Not Equal” Operator

#filter DataFrame where team is not equal to 'A'
df.filter(df.team!='A').show()

Method 2: Filter Using Multiple “Not Equal” Operators

#filter DataFrame where team is not equal to 'A' and points is not equal to 5
df.filter((df.team!='A') & (df.points!=5)).show()

The following examples show how to use each method in practice with the following PySpark DataFrame:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

#define data
data = [['A', 'East', 11, 4], 
        ['A', 'East', 8, 9], 
        ['A', 'East', 10, 3], 
        ['B', 'West', 6, 12], 
        ['B', 'West', 6, 4], 
        ['C', 'East', 5, 2]] 
  
#define column names
columns = ['team', 'conference', 'points', 'assists'] 
  
#create dataframe using data and column names
df = spark.createDataFrame(data, columns) 
  
#view dataframe
df.show()

+----+----------+------+-------+
|team|conference|points|assists|
+----+----------+------+-------+
|   A|      East|    11|      4|
|   A|      East|     8|      9|
|   A|      East|    10|      3|
|   B|      West|     6|     12|
|   B|      West|     6|      4|
|   C|      East|     5|      2|
+----+----------+------+-------+

Example 1: Filter Using One “Not Equal” Operator

We can use the following syntax to filter the DataFrame to only contain rows where the team column is not equal to A:

#filter DataFrame where team is not equal to 'A'
df.filter(df.team!='A').show()

+----+----------+------+-------+
|team|conference|points|assists|
+----+----------+------+-------+
|   B|      West|     6|     12|
|   B|      West|     6|      4|
|   C|      East|     5|      2|
+----+----------+------+-------+

Notice that each of the rows in the resulting DataFrame contain a value in the team column that is not equal to A.

Example 2: Filter Using Multiple “Not Equal” Operators

We can use the following syntax to filter the DataFrame to only contain rows where the team column is not equal to A and the value in the points column is not equal to 5:

#filter DataFrame where team is not equal to 'A' and points is not equal to 5
df.filter((df.team!='A') & (df.points!=5)).show()

+----+----------+------+-------+
|team|conference|points|assists|
+----+----------+------+-------+
|   B|      West|     6|     12|
|   B|      West|     6|      4|
+----+----------+------+-------+

Notice that each of the rows in the resulting DataFrame contain a value in the team column that is not equal to A and a value in the points column that is not equal to 5.