Photon 並非適用於所有 Spark 查詢的神奇加速器

Why this matters: Photon is a vectorized query engine that replaces portions of the standard Spark JVM-based execution with C++. Understanding when it triggers and where it fails prevents wasted compute spend on workloads that gain zero performance benefit from the engine.

The most persistent myth in the Databricks ecosystem is that switching to a Photon-enabled runtime automatically makes every Spark job run faster. It is easy to see why: "Photon" sounds like a performance upgrade, and the marketing literature promises significant speedups for ETL and SQL. However, Photon is not a universal accelerator. It is a specialized engine designed for specific types of data processing. When your workload falls outside its supported surface area, your cluster spends time overhead-mapping Spark operations to Photon kernels for no gain, or worse, falling back to the standard JVM execution path mid-query.

Prerequisites

To follow this tutorial, you need:

• A Databricks workspace with a cluster running Databricks Runtime (DBR) 11.3 LTS or higher.
• The databricks-runtime provides Photon by default on supported instance types.
• Access to the Spark UI to inspect query plans.
• pyspark installed locally if you wish to mock the data, though the query profile analysis must be performed on a cluster with Photon enabled.

Photo by Lightsaber Collection on Unsplash

Step 1: Generating a workload that Photon loves

Photon shines brightest when processing large volumes of data using SQL-like operations: joins, aggregations, and scanning columnar formats like Parquet or Delta. It is designed to minimize the overhead of the Java object model by processing data in a vectorized, columnar format in native memory.

Consider this illustrative scenario where we join two large tables. Photon will attempt to vectorize the join and the subsequent aggregation.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum

# Create a local SparkSession for testing
spark = SparkSession.builder.appName("PhotonTest").getOrCreate()

# Create hypothetical data
df_sales = spark.range(1000000).selectExpr("id as sales_id", "cast(rand()*100 as int) as region_id", "cast(rand()*1000 as double) as amount")
df_regions = spark.range(100).selectExpr("id as region_id", "concat('Region_', cast(id as string)) as region_name")

# A classic join and aggregate query
result = df_sales.join(df_regions, "region_id") \
    .groupBy("region_name") \
    .agg(sum("amount").alias("total_sales"))

result.explain()

Step 2: Inspecting the Query Profile

The explain() output in your notebook provides a glimpse, but the Spark UI is where you confirm Photon’s activity. When you run the above code on a Photon-enabled cluster, open the Spark UI and navigate to the SQL / DataFrame tab.

Click on the query description. You are looking for the "Photon" indicators:

1. Operator Names: Look for operators prefixed with "Photon" (e.g., PhotonScan, PhotonHashAggregate, PhotonShuffleExchange). If you see these, the engine is actively executing that node.
2. Physical Plan: If you see standard Spark operators (like HashAggregate instead of PhotonHashAggregate), Photon is not handling that specific operation.

If your query is entirely standard Spark, the "Photon" column in the SQL tab will be blank or indicate 0% usage. This is your primary diagnostic tool: if your expensive jobs show 0% Photon usage, you are paying for the Photon-enabled instance type without getting the benefit.

Step 3: Identifying the fallback mechanism

Photon does not support every expression or data type in the Spark ecosystem. When it encounters an unsupported operation (like a custom UDF that requires the JVM), it performs a "fallback."

In the Spark UI, look for the transition points. If a large portion of your DAG is "Photon" but a specific stage reverts to standard JVM operators, you have found a bottleneck. Often, this is caused by:

• Python UDFs: Photon cannot run Python code directly.
• Complex Types: Certain nested types or specific custom serialization formats.
• Unsupported SQL functions: Rare, but specific legacy Spark functions may not have a native C++ implementation yet.

Step 4: Testing for "Photon-hostile" code

To see Photon get bypassed, run a query that relies heavily on a Python UDF.

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

# A Python UDF forces a transition to the JVM/Python process
@udf(returnType=StringType())
def expensive_py_udf(val):
    return f"processed_{val}"

df = spark.range(100000).select(expensive_py_udf(col("id")).alias("transformed"))
df.collect()

When you view the query profile for this, you will notice the plan lacks the Photon prefix on the projection operator. By introducing the UDF, you effectively "de-vectorized" that part of the pipeline.

Photo by Markus Spiske on Unsplash

Common pitfalls

• Instance Type Misalignment: Photon is not available on all instance types. If you select an instance type that is not optimized for it, the cluster will start, but Photon will be disabled. Always check the Databricks documentation for your specific cloud provider's Photon-supported instance types.
• Over-reliance on UDFs: If your entire ETL pipeline is wrapped in Python UDFs to perform standard transformations, you have effectively neutered Photon. Where possible, rewrite UDFs using native Spark SQL functions (pyspark.sql.functions). Native functions are heavily optimized for Photon.
• Small Data overhead: For very small datasets, the overhead of moving data into the Photon native memory space can sometimes make the query slightly slower than the standard JVM engine. Do not optimize for Photon if your data fits in a single partition and runs in milliseconds.
• Assuming 100% coverage: You will rarely see 100% Photon usage in complex pipelines. Aim for "Photon-heavy" segments in your largest data-shuffling stages.

Conclusion

Photon is a powerful tool for large-scale data processing, but it is not a "set it and forget it" feature. It requires you to write "Photon-friendly" code—avoiding custom UDFs in favor of native SQL expressions and ensuring your data joins and aggregations are well-structured.

Use the Spark UI as your source of truth. If you see the "Photon" prefix in your query plan, you are getting the speedup. If you see the standard JVM operators, investigate the node and determine if a UDF or an unsupported function is forcing the engine to fall back.

Try it: Take one of your existing, high-latency SQL pipelines, run it on a Photon-enabled cluster, and inspect the Spark UI. Identify the stages that are not running as Photon, and attempt to replace any UDFs in those stages with native Spark SQL functions. Observe the change in the Spark UI execution plan.

Cover photo by Trevor Kay on Unsplash.

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.