feat: Support dynamic partition overwrite mode in comet

[coderfender]

Which issue does this PR close?

Closes ##2970

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[coderfender]

@comphead this is the PR to support overwrite mode in comet. Please take a look whenever you get a chance

[comphead]

Thanks @coderfender I think it would be also needed to support partititioned mode. Otherwise this approach will delete entire user data

The flow is like

User calls: df.write.mode("overwrite").save(path)
    ↓
DataFrameWriter.saveInternal() 
    ↓
InsertIntoHadoopFsRelationCommand.run()
    ↓
[Line 131] deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
    ↓
[Line 238] committer.deleteWithJob(fs, staticPrefixPath, true)
    ↓
[Line 183] fs.delete(path, recursive=true)  ← ACTUAL DELETION HAPPENS HERE

This Spark implementation now

    val doInsertion = if (mode == SaveMode.Append) {
      true
    } else {
      val pathExists = fs.exists(qualifiedOutputPath)
      (mode, pathExists) match {
        case (SaveMode.ErrorIfExists, true) =>
          throw QueryCompilationErrors.outputPathAlreadyExistsError(qualifiedOutputPath)
        case (SaveMode.Overwrite, true) =>
          if (ifPartitionNotExists && matchingPartitions.nonEmpty) {
            false
          } else if (dynamicPartitionOverwrite) {
            // For dynamic partition overwrite, do not delete partition directories ahead.
            true
          } else {
            deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
            true
          }
        case (SaveMode.Overwrite, _) | (SaveMode.ErrorIfExists, false) =>
          true
        case (SaveMode.Ignore, exists) =>
          !exists
        case (s, exists) =>
          throw QueryExecutionErrors.saveModeUnsupportedError(s, exists)
      }
    }

[coderfender]

Thank you @comphead . Let me work on updating the code to not delete entire user data and just clear out the effected partitions

[comphead]

Thank you @comphead . Let me work on updating the code to not delete entire user data and just clear out the effected partitions

Thanks @coderfender Ideally if we can reuse Spark code

[coderfender]

Sure .Thank you. Let me try and port spark code to not have duplication