A data scientist at a financial services company is working with a Spark DataFrame containing transaction records. The DataFrame has millions of rows and includes columns fortransaction_id,account_number, transaction_amount, andtimestamp. Due to an issue with the source system, some transactions were accidentally recorded multiple times with identical information across all fields. The data scientist needs to remove rows with duplicates across all fields to ensure accurate financial reporting. Which approach should the data scientist use to deduplicate the orders using PySpark?
Select one answer.
Purchase Full Access to view answer options & explanation
The first 118 questions in this bank are completely free. Unlock Full Access on the exam page to view all answer choices, verified answers, explanations, and community discussions for all 392 questions.
