release-26.1: bulkmerge, importer: implement SST merging and optionally use it in the importer

[mw5h]

bulkmerge: implement SST merging in bulk merge processor

Previously, the bulkMergeProcessor was a stub that returned placeholder
output URIs without performing any actual SST merging. This made the
distributed merge infrastructure non-functional for real workloads.

This commit implements the core merging logic in the processor by:

1. Adding mergeSSTs method that identifies overlapping SSTs for each
   task's key range, creates an iterator over the external SST files,
   and writes merged output while respecting configurable size limits.

2. Introducing a new Merge function that creates and executes the
   DistSQL merge flow, collects results from all processors, and sorts
   the output SSTs by their start key for ingestion.

3. Adding comprehensive tests for both single-node and multi-node
   distributed merge scenarios, including test infrastructure for
   creating and verifying SST merge results.

The implementation now produces correctly merged, non-overlapping SST
files that can be ingested into ranges, making the distributed bulk
merge functionality operational.

Fixes: #156658
Release note: None

Co-Authored-By: Annie Pompa annie@cockroachlabs.com

sql: integrate distributed merge into DSC index backfill

Previously, the declarative schema changer (DSC) index backfiller only
partially supported the new distributed merge pipeline. It would generate SSTs,
but then didn't have the rest of the pipeline in place. Now that all components
of the pipeline are available, this completes the entire flow. Note, this
applies to the DSC only. A separate change will do something similar for the
legacy schema changer.

This change completes the integration by:

• Introducing a registration bridge (bulk_bridge.go) to break import cycles
  between pkg/sql, pkg/sql/bulkmerge, and pkg/sql/bulkingest
• Implementing runDistributedMerge and runDistributedIngest to coordinate
  the merge and ingest phases
• Registering the Merge and IngestFiles implementations via init() functions
• Updating logic tests to validate the new distributed merge flow
• Removing placeholder tests that previously expected failures. The part that
  verified mixed version was specifically added to a mixed-version only test.

Informs: #158378
Epic: CRDB-48845

Release note: none

bulkmerge: prevent loopback processor hang on task failure

Prior to this change, the loopback processor in the distributed merge
pipeline could hang if a worker failed during execution. This commit
ensures that merge_coordinator closes the loopback as soon as it sees
meta.Err, preventing such hangs.

Also adds testing hooks via BulkMergeTestingKnobs, enabling injection of
errors for validation purposes.

Informs #156658
Epic: CRDB-48845
Release note: none

bulkmerge: fix SST boundary handling for column families

Previously, the merge processor would split SSTs at arbitrary byte
positions when reaching the target SST size, which could result in
splits within column families. Additionally, the split picker was
validating that SST boundaries were already at safe split points and
rejecting SSTs that weren't, which was both overly strict (rejecting
valid splits) and ineffective (the check itself was flawed).

This commit fixes both issues:

1. In the merge processor, SST endpoints are now selected using
   EnsureSafeSplitKey() to find the next safe split point (row
   boundary) after reaching the target size, then using PrefixEnd()
   to properly bound the SST. This ensures we never split within
   a column family.

2. In the split picker, the incorrect validation check has been
   removed. The merge processor now ensures SST boundaries are
   correct at write time, making validation at split-picking time
   redundant.

3. In combine_file_info, merge span boundaries are now validated
   using EnsureSafeSplitKey() to ensure they start at safe split
   points.

Informs: #156658

Release note: none

Epic: CRDB-48845

bulksst: add utility to append one sst file set to another

Previously, there was no way to combine multiple SSTFiles structures,
which is needed when aggregating SST manifests from multiple processors
in distributed operations.

This commit adds an Append method to SSTFiles that combines SST slices,
row samples, and sums the total sizes. This utility will be used by the
distributed merge pipeline for IMPORT operations to collect SST metadata
from multiple import processors.

Informs: #156660
Release note: None

importer: optionally use distributed merge for IMPORT jobs

Previously, IMPORT jobs always used the legacy direct-ingest approach
where each processor directly ingested SSTs into the storage layer. This
approach doesn't work well for large imports as it can create excessive
L0 files and doesn't allow for merge-time optimizations.

This commit adds support for the distributed merge pipeline to IMPORT
operations, following the same pattern used by index backfills. The
implementation:

1. Adds a new cluster setting bulkio.import.distributed_merge.mode to
   control whether distributed merge is used for imports (defaults to
   false for backward compatibility).

2. Determines the merge mode at IMPORT planning time based on the
   cluster setting value and stores it in the job payload
   (ImportDetails.use_distributed_merge). This ensures the decision
   persists across job resumptions and isn't affected by setting changes
   during execution.

3. Adds a version gate at V26_1 to ensure all nodes in the cluster
   support the new proto fields before allowing distributed merge to be
   enabled.

4. When distributed merge is enabled, import processors write SSTs to
   node-local storage and emit file metadata instead of directly
   ingesting. The coordinator collects these SSTs and uses the bulkmerge
   package to merge and ingest them.

This harmonizes the IMPORT implementation with the index backfill
approach and provides a common foundation for future optimizations.

Fixes: #156660

Co-Authored-By: Jeff Swenson jeffswenson@betterthannull.com
Co-Authored-By: Faizan Qazi faizan@cockroachlabs.com

Release note (sql change): Added a new cluster setting
bulkio.import.distributed_merge.mode to enable distributed merge
support for IMPORT operations. When enabled (default: false), IMPORT
jobs will use a two-phase approach where import processors first write
SST files to local storage, then a coordinator merges and ingests them.
This can improve performance for large imports by reducing L0 file
counts and enabling merge-time optimizations. Note: This feature
requires all nodes to be running version 26.1 or later.

[blathers-crl]

Thanks for opening a backport.

Before merging, please confirm that the change does not break backwards compatibility and otherwise complies with the backport policy. Include a brief release justification in the PR description explaining why the backport is appropriate. All backports must be reviewed by the TL for the owning area. While the stricter LTS policy does not yet apply, please exercise judgment and consider gating non-critical changes behind a disabled-by-default feature flag when appropriate.

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is