PySpark: Replace String in Column

---

You can use the following syntax to replace a specific string in a column of a PySpark DataFrame:

from pyspark.sql.functions import *

#replace 'Guard' with 'Gd' in position column
df_new = df.withColumn('position', regexp_replace('position', 'Guard', 'Gd'))

This particular example replaces the string “Guard” with the new string “Gd” in the position column of the DataFrame.

The following examples show how to use this syntax in practice.

Example: Replace String in Column of PySpark DataFrame

Suppose we have the following PySpark DataFrame that contains information about various basketball players:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

#define data
data = [['A', 'Guard', 11], 
        ['A', 'Guard', 8], 
        ['A', 'Forward', 22], 
        ['A', 'Forward', 22], 
        ['B', 'Guard', 14], 
        ['B', 'Guard', 14],
        ['B', 'Guard', 13],
        ['B', 'Forward', 7],
        ['C', 'Guard', 8],
        ['C', 'Forward', 5]] 
  
#define column names
columns = ['team', 'position', 'points'] 
  
#create dataframe using data and column names
df = spark.createDataFrame(data, columns) 
  
#view dataframe
df.show()

+----+--------+------+
|team|position|points|
+----+--------+------+
|   A|   Guard|    11|
|   A|   Guard|     8|
|   A| Forward|    22|
|   A| Forward|    22|
|   B|   Guard|    14|
|   B|   Guard|    14|
|   B|   Guard|    13|
|   B| Forward|     7|
|   C|   Guard|     8|
|   C| Forward|     5|
+----+--------+------+

We can use the following syntax to replace the string “Guard” with the new string “Gd” in the position column of the DataFrame:

from pyspark.sql.functions import *

#replace 'Guard' with 'Gd' in position column
df_new = df.withColumn('position', regexp_replace('position', 'Guard', 'Gd'))

#view new DataFrame
df_new.show()

+----+--------+------+
|team|position|points|
+----+--------+------+
|   A|      Gd|    11|
|   A|      Gd|     8|
|   A| Forward|    22|
|   A| Forward|    22|
|   B|      Gd|    14|
|   B|      Gd|    14|
|   B|      Gd|    13|
|   B| Forward|     7|
|   C|      Gd|     8|
|   C| Forward|     5|
+----+--------+------+

From the output we can see that each occurrence of “Guard” has been replaced with “Gd” in the position column of the DataFrame.

Note #1: The regexp_replace function is case-sensitive.

Note #2: You can find the complete documentation for the PySpark regexp_replace function .