How to Extract the Year from a Date in PySpark

You can use the following standard syntax to extract the year from a date column within a PySpark DataFrame:

from pyspark.sql.functions import year

df_new = df.withColumn('year', year(df['date']))

This powerful and common example creates a new column called year that efficiently extracts the integer year component from the date values contained in the date column of the original DataFrame. This transformation is the foundation upon which more complex temporal analyses and aggregations are subsequently built in PySpark.

Practical Implementation: Defining the Sample Data

To demonstrate the year extraction process effectively and provide a clear, executable example, we will work with a sample DataFrame representing sales records over several years. This scenario is highly representative of common use cases in real-world data science projects, where raw transactional data needs to be structured temporally for analysis. Our initial task is to initialize a SparkSession and populate a DataFrame with synthetic data containing a ‘date’ column and a corresponding ‘sales’ metric.

We utilize the SparkSession.builder.getOrCreate() method to initialize the environment, ensuring Spark is available and ready for processing. The data itself is defined as a list of lists, where the first element is the date string (in the standard ‘YYYY-MM-DD’ format) and the second element is the corresponding sales quantity. Defining the column names explicitly as ‘date’ and ‘sales’ allows us to use spark.createDataFrame to instantiate the DataFrame object successfully, maintaining control over the initial data schema.

The code block below illustrates the concise steps required for creating and initially displaying the sample DataFrame. While the ‘date’ column appears highly structured, its data type depends on Spark’s schema inference during creation. Although the year() function is robust enough to often handle common date string formats automatically, relying on explicit type casting via to_date() function remains the safest practice for handling production data diversity. For this educational example, we proceed directly to the extraction assuming standard format recognition.

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# Define sample data for sales transactions across multiple years
data = [['2021-04-11', 22],
        ['2021-04-15', 14],
        ['2021-04-17', 12],
        ['2022-05-21', 15],
        ['2022-05-23', 30],
        ['2023-10-26', 45],
        ['2023-10-28', 32],
        ['2023-10-29', 47]]
  
# Define the column names for the DataFrame
columns = ['date', 'sales']
  
# Create the DataFrame using the data and defined columns
df = spark.createDataFrame(data, columns) 
  
# Display the initial DataFrame structure and content
df.show()

+----------+-----+
|      date|sales|
+----------+-----+
|2021-04-11|   22|
|2021-04-15|   14|
|2021-04-17|   12|
|2022-05-21|   15|
|2022-05-23|   30|
|2023-10-26|   45|
|2023-10-28|   32|
|2023-10-29|   47|
+----------+-----+

Executing the Year Extraction Operation

Our objective is now to take the df we created and derive a new column containing only the integer year value corresponding to each entry in the ‘date’ column. This transformation is pivotal for tasks like grouping sales data by year to identify long-term trends or preparing features for advanced time-series models that rely on clean, numerical temporal components. We will use the crucial combination of the year() function and the withColumn() function to achieve this modification efficiently and scalably.

First, we ensure the year function is imported from pyspark.sql.functions. We then chain the withColumn() transformation directly to the original DataFrame (`df`). We specify the new column name as ‘year’. Critically, inside withColumn(), we apply the year() function, pointing it specifically to df['date']. Because of Spark’s architecture, this operation is executed in parallel across the cluster partitions, resulting in high throughput even for petabyte-scale datasets. The resulting DataFrame, df_new, maintains all the original columns while including the newly calculated ‘year’ column.

Viewing the resulting DataFrame (df_new) confirms that the extraction was successful and instantaneous. The newly generated ‘year’ column accurately reflects the year component of the corresponding dates (2021, 2022, and 2023). This derived column is now immediately ready to be used in powerful aggregation functions (such as df_new.groupBy('year').sum('sales')) or precise filtering operations, drastically simplifying subsequent analysis steps that rely on annual segmentation and reporting.

from pyspark.sql.functions import year

# Extract year from the existing 'date' column and store it in a new column named 'year'
df_new = df.withColumn('year', year(df['date']))

# View the resulting DataFrame with the newly added 'year' column
df_new.show()

+----------+-----+----+
|      date|sales|year|
+----------+-----+----+
|2021-04-11|   22|2021|
|2021-04-15|   14|2021|
|2021-04-17|   12|2021|
|2022-05-21|   15|2022|
|2022-05-23|   30|2022|
|2023-10-26|   45|2023|
|2023-10-28|   32|2023|
|2023-10-29|   47|2023|
+----------+-----+----+

As shown in the output above, the new year column successfully contains the integer representation of the year for each corresponding date in the date column, confirming the robust functionality of the year() function combined with the immutable transformation capabilities of the DataFrame API.

Deep Dive into withColumn() for DataFrame Transformations

The withColumn() function is arguably one of the most fundamental and frequently used methods for PySpark DataFrame manipulation. Its primary role is to enable the modification or creation of columns based on arbitrary expressions defined by the user. When used for date extraction, as we have demonstrated, it ensures that the original DataFrame remains immutable—a core principle of Spark and functional programming—by generating a new DataFrame object that incorporates the calculated column.

The function signature requires two primary parameters: the name of the column to be added or replaced, and the Column expression that defines the logic for calculating the new values. This expression is precisely where we embed our specific SQL functions, such as year(), to_date(), or even complex conditional statements involving multiple columns. Because Spark utilizes lazy evaluation, the expression defined within withColumn() is not actually computed until an action (like show(), count(), or write()) is called, allowing the Catalyst optimizer to analyze and potentially rearrange the execution plan for maximum efficiency.

While one could technically achieve a similar result using the `select()` function by aliasing the expression, withColumn() is generally the preferred method when the intent is to retain all existing columns while adding a new, derived column. For instance, if you were only interested in the date and the newly extracted year, select(df['date'], year(df['date']).alias('year')) would work. However, for complex, iterative transformation workflows where preserving the full context of the original data is crucial, withColumn() provides a cleaner, more readable, and semantically appropriate approach, making the code easier to maintain and debug.

Note that we used the withColumn function to add a new column called year to the DataFrame while keeping all existing columns the same.

Note: You can find the complete documentation for the PySpark withColumn function here.

Handling Complex Date Formats using to_date()

In real-world data pipelines, dates are rarely delivered in a perfectly clean, ISO standard format (‘YYYY-MM-DD’). Sometimes they arrive as arbitrary strings like ‘DD/MM/YYYY’, ‘MM-DD-YY’, or even include extraneous time zone information within a non-standard string structure. If PySpark cannot automatically infer the schema or if the format is non-standard, applying the year() function directly will typically result in null values or execution errors because the underlying type is not a valid DateType. This situation dictates that an explicit data type conversion step must be integrated into the workflow.

The to_date() function is the essential solution for this common data quality challenge. It requires two mandatory arguments: the column containing the date strings and the format pattern string that accurately corresponds to the structure of the input data. For example, if your raw data column contained ’26/10/2023′, the necessary format pattern string would be ‘dd/MM/yyyy’. This conversion function is then nested within our withColumn transformation, ensuring the data type is corrected before the specialized functions like `year()` are applied.

If our DataFrame contained a string column named ‘raw_date’ in the format ‘MM/dd/yyyy’, the comprehensive, multi-step extraction syntax would look like the following code example. This approach emphasizes safety and correctness by first casting the potentially ambiguous string data into a reliable DateType using to_date(), and then performing the extraction using year(), ensuring maximum stability when dealing with diverse and variable data sources.

from pyspark.sql.functions import year, to_date

df_converted = df.withColumn('date_converted', to_date(df['raw_date'], 'MM/dd/yyyy'))
df_final = df_converted.withColumn('extracted_year', year(df_converted['date_converted']))

Related PySpark Data Transformation Tutorials

The following tutorials explain how to perform other common data transformation and manipulation tasks in PySpark, building upon the foundational knowledge of column modification and function application demonstrated in this guide:

• Extracting the month, day, or quarter from a date column using similar built-in functions.

• Calculating the difference in days or months between two date columns using datediff().

• Converting PySpark DataFrames to Pandas for localized analysis on smaller, filtered subsets.

• Using date_format() for customized date string outputs that include year, month, and day in specific formats.

• Working with Timestamp data types and handling time zone conversions effectively.