DATABRICKS-SPARK Question 111

Community Discussion

Written by other learners. Comments can disagree with the marked answer — check the official documentation when in doubt.

• Chose D0

  - Comprehensive and Detailed Explanation From Exact Extract: When converting a standard PySpark UDF to a Pandas UDF for performance optimization, the function must operate on a Pandas Series as input and return a Pandas Series as output. In this case, the original function signature: def shake_256(raw: str) -> str is scalar - not compatible with Pandas UDFs. According to the official Spark documentation: "Pandas UDFs operate onpandas.Seriesand returnpandas.Series. The function definition should be: def my_udf(s: pd.Series) -> pd.Series: and it must be registered usingpandas_udf(...)." Therefore, to fix the error: The function should be updated to: def shake_256(df: pd.Series) -> pd.Series: return df.apply(lambda x: hashlib.shake_256(x.encode()).hexdigest(20)) This will allow Spark to efficiently execute the Pandas UDF in vectorized form, improving performance compared to standard UDFs. Reference: Apache Spark 3.5 Documentation # User-Defined Functions # Pandas UDFs