Table of Contents
To compare strings between two columns in PySpark, the following steps can be followed:
1. First, import the necessary libraries for PySpark and initialize a Spark session.
2. Load the dataset containing the two columns of strings into a dataframe.
3. Use the “select” function to select the two columns that need to be compared.
4. Apply the “when” function to create a new column which will contain the comparison result.
5. Use the “like” operator to specify the condition for the comparison between the two columns.
6. Assign a value to the new column based on the result of the comparison.
7. Finally, use the “show” function to display the dataframe with the new column containing the comparison result.
In summary, PySpark offers a simple and efficient way to compare strings between two columns by using the “when” and “like” functions. This allows for easy analysis and manipulation of string data within a dataframe.
PySpark: Compare Strings Between Two Columns
You can use the following syntax to compare strings between two columns in a PySpark DataFrame:
Method 1: Compare Strings Between Two Columns (Case-Sensitive)
df_new = df.withColumn('equal', df.team1==df.team2) This particular example compares the strings between columns team1 and team2 and returns either True or False to indicate if the strings are the same or not.
Method 2: Compare Strings Between Two Columns (Case-Insensitive)
from pyspark.sql.functions import lower
df_new = df.withColumn('equal', lower(df.team1)==lower(df.team2)
This particular example performs a case-insensitive comparison between the strings in columns team1 and team2.
The following example shows how to use each method in practice with the following DataFrame that contains two columns of basketball team names:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
#define data
data = [['Mavs', 'Mavs'],
['Nets', 'nets'],
['Lakers', 'Lakers'],
['Kings', 'Jazz'],
['Hawks', 'HAWKS'],
['Wizards', 'Wizards']]
#define column names
columns = ['team1', 'team2']
#create dataframe using data and column names
df = spark.createDataFrame(data, columns)
#view dataframe
df.show()
+-------+-------+
| team1| team2|
+-------+-------+
| Mavs| Mavs|
| Nets| nets|
| Lakers| Lakers|
| Kings| Jazz|
| Hawks| HAWKS|
|Wizards|Wizards|
+-------+-------+ Example 1: Compare Strings Between Two Columns (Case-Sensitive)
We can use the following syntax to compare the strings (case-sensitive) between the team1 and team2 columns:
#compare strings between team1 and team2 columns
df_new = df.withColumn('equal', df.team1==df.team2)
#view new DataFrame
df_new.show()
+-------+-------+-----+
| team1| team2|equal|
+-------+-------+-----+
| Mavs| Mavs| true|
| Nets| nets|false|
| Lakers| Lakers| true|
| Kings| Jazz|false|
| Hawks| HAWKS|false|
|Wizards|Wizards| true|
+-------+-------+-----+
The new column named equal returns True if the strings match (including the case of the strings) between the two columns or False otherwise.
Example 2: Compare Strings Between Two Columns (Case-Insensitive)
We can use the following syntax to compare the strings (case-insensitive) between the team1 and team2 columns:
from pyspark.sql.functions import lower
#compare strings between team1 and team2 columns
df_new = df.withColumn('equal', lower(df.team1)==lower(df.team2))
#view new DataFrame
df_new.show()
+-------+-------+-----+
| team1| team2|equal|
+-------+-------+-----+
| Mavs| Mavs| true|
| Nets| nets| true|
| Lakers| Lakers| true|
| Kings| Jazz|false|
| Hawks| HAWKS| true|
|Wizards|Wizards| true|
+-------+-------+-----+
The new column named equal returns True if the strings match (regardless of case) between the two columns or False otherwise.
Note #2: You can find the complete documentation for the PySpark withColumn function .
Additional Resources
The following tutorials explain how to perform other common tasks in PySpark: