Skip to content

Conversation

@lyne7-sc
Copy link

Which issue does this PR close?

  • Closes #.

Rationale for this change

This PR improves performance by:

  • Using exact Utf8View byte size (sum of data buffers) instead of row-based approximation.
  • Building results via .concat()/.join(sep) on a pre-allocated Vec<&str> to avoid String reallocations.

Benchmark

Case Before After Change
concat_ws_scalar/8 299.18 ns 233.18 ns -21.93%
concat_ws_scalar/32 327.53 ns 251.44 ns -23.23%
concat_ws_scalar/128 405.80 ns 271.27 ns -33.15%
concat_ws_scalar/4096 976.02 ns 791.33 ns -18.92%
concat_scalar/8 248.71 ns 221.24 ns -11.05%
concat_scalar/32 284.26 ns 240.53 ns -15.39%
concat_scalar/128 301.91 ns 257.61 ns -14.67%
concat_scalar/4096 916.68 ns 805.33 ns -12.15%

What changes are included in this PR?

Performance optimization for concat and concat_ws functions scalar path.

Are these changes tested?

  • Existing unit and integration tests pass.
  • New benchmarks added to verify performance improvement.

Are there any user-facing changes?

No. It's a pure performance optimization.

@github-actions github-actions bot added the functions Changes to functions implementation label Dec 29, 2025
@Omega359
Copy link
Contributor

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux fedora 6.17.12-300.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Dec 13 05:06:24 UTC 2025 x86_64 GNU/Linux
Comparing perf/concat (301ab2e) to 83ed192 diff
BENCH_NAME=concat
BENCH_COMMAND=cargo bench --bench concat
BENCH_FILTER=
BENCH_BRANCH_NAME=perf_concat
Results will be posted here when complete

1 similar comment
@Omega359
Copy link
Contributor

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux fedora 6.17.12-300.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Dec 13 05:06:24 UTC 2025 x86_64 GNU/Linux
Comparing perf/concat (301ab2e) to 83ed192 diff
BENCH_NAME=concat
BENCH_COMMAND=cargo bench --bench concat
BENCH_FILTER=
BENCH_BRANCH_NAME=perf_concat
Results will be posted here when complete

@Omega359
Copy link
Contributor

🤖: Benchmark completed

Details

group                            main                                   perf_concat
-----                            ----                                   -----------
concat function/concat/1024      1.03      9.3±0.85µs        ? ?/sec    1.00      9.0±0.29µs        ? ?/sec
concat function/concat/4096      1.00     35.1±0.71µs        ? ?/sec    1.00     35.1±1.01µs        ? ?/sec
concat function/concat/8192      1.01     69.9±3.74µs        ? ?/sec    1.00     69.6±0.95µs        ? ?/sec
concat function/concat/scalar                                           1.00     32.5±0.13µs        ? ?/sec

@andygrove andygrove added the performance Make DataFusion faster label Dec 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation performance Make DataFusion faster

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants