Audio: STFT Process: Add new SOF module #10306

singalsu · 2025-10-15T16:13:59Z

No description provided.

src/arch/host/CMakeLists.txt

src/audio/stft_process/stft_process.c

src/audio/stft_process/stft_process_common.c

src/audio/stft_process/stft_process_setup.c

src/arch/host/CMakeLists.txt

src/audio/stft_process/stft_process_common.c

src/audio/stft_process/stft_process_setup.c

singalsu · 2025-10-21T17:02:19Z

Improved audio quality with 32 bit window functions and switch to Hann window type.

lgirdwood · 2025-10-22T15:51:35Z

@singalsu can you also add a Readme.md to the module directory describing usage/tuning. Thanks !

test/cmocka/src/math/window/window.c

src/include/sof/math/window.h

singalsu · 2025-12-02T12:45:12Z

src/include/sof/audio/coefficients/fft/twiddle_3072_32.h

+#define FFT_MULTI_TWIDDLE_SIZE	2048
+
+/* in Q1.31, generated from cos(i * 2 * pi / FFT_SIZE_MAX) */
+const int32_t multi_twiddle_real_32[FFT_MULTI_TWIDDLE_SIZE] = {


Make this cold data with needed parts (depends on FFT size) copy to fast RAM.

src/math/fft/fft_common.c

src/math/fft/fft_32.c

singalsu · 2025-12-05T13:34:26Z

Update: The audio quality is now good, transparent for 16 bit audio like the CD format. This is the output from running test process_test('stft_process_1536_240_', 32, 32, 48000, 1, 1);

Copilot

Pull request overview

This PR adds a new STFT (Short-Time Fourier Transform) processing module for SOF, including comprehensive test infrastructure for FFT operations and configuration files for audio topology.

Adds UUID registration and module initialization for the new stft_process component
Implements topology configuration files for STFT processing with different window sizes (1024x256 and 1536x240)
Adds extensive FFT/IFFT test infrastructure with reference data for multiple FFT sizes
Updates build configuration to include the new STFT process module

Reviewed changes

Copilot reviewed 56 out of 59 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
uuid-registry.txt	Registers UUIDs for `math_fft` and `stft_process` modules
tools/topology/topology2/include/components/stft_process/*.conf	Configuration files for STFT processing with Hann window parameters
tools/topology/topology2/include/bench/stft_process*.conf	Benchmark configuration files for STFT processing with different sample formats
tools/topology/topology2/development/tplg-targets-bench.cmake	Build system updates to include STFT process module targets
tools/topology/topology2/cavs-benchmark-*.conf	Integration of STFT process into benchmark topologies
tools/testbench/utils_ipc4.c	Adds STFT process module initialization
tools/rimage/config/tgl*.toml	Module configuration for STFT process component
test/cmocka/src/math/window/window.c	Updates window function constant name
test/cmocka/src/math/fft/ref_*.h	Reference test data for FFT/IFFT operations across multiple sizes
test/cmocka/src/math/fft/*.c	Test implementations for FFT, IFFT, and DFT3 operations
test/cmocka/src/math/fft/*.m	MATLAB/Octave scripts for generating FFT test reference data
src/math/fft/tune/README.md	Documentation for generating twiddle factors

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tools/topology/topology2/development/tplg-targets-bench.cmake

src/audio/stft_process/tune/setup_stft_process.m

singalsu · 2025-12-05T15:36:52Z

src/audio/stft_process/tune/setup_stft_process.m

+    cfg.common_path = [cfg.tools_path 'tune/common'];
+    cfg.tplg_ver = 2;
+    cfg.ipc_ver = 4;
+    cfg.channel = 0;


Add description of channel, e.g. 0 for process all channels, 1 .. max channels, select a mono channel

Update, currently this is ignored but could be supported later if e.g. want to process one selected channel as mono from stereo stream.

might be worth a comment, unless you can have all the documentation in place for next release. Can be done incrementally.

Yes, I will update the configuration blobs to support the formats the frequency domain models need. This likely still changes. Also I'm not sure if the models are multi-channel. It may need also stereo to mono conversion and maybe some double mono output option.

src/math/fft/fft_common.c

lyakh · 2025-12-08T08:34:58Z

src/math/fft/fft_common.c

+
+	/* set up the bit reverse index */
+	for (i = 1; i < size; ++i)
+		bit_reverse_idx[i] = (bit_reverse_idx[i >> 1] >> 1) | ((i & 1) << (len - 1));


not sure what this is meant to do, but it isn't using initial bit_reverse_idx[] values from the second half - above len / 2 - is that correct?

It reverses order of bits, e.g. 0b10100000 would become 0b00000101. I didn't invent that, it was in original code and I have better code targets to improve for performance. This isn't a hot code part.

sof/src/math/fft/fft_common.c

Line 61 in 0855423

plan->bit_reverse_idx[i] = (plan->bit_reverse_idx[i >> 1] >> 1) |

What code snippet do you suggest to replace this? I can test if you propose something improved.

all good now, thanks for an offline explanation! A comment, explaining that bit_reverse_idx[i] contains i with its bits reversed would help, but maybe it's obvious to most

Yep, true it doesn't hurt to add it.

lyakh · 2025-12-08T09:12:58Z

src/audio/stft_process/stft_process.h

+	int fft_padded_size;
+	int fft_hop_size;
+	int fft_buf_size;
+	int half_fft_size;


unsigned int or size_t if in bytes?

These are not count of bytes but count of samples. Could use "length" as well.

I don't get the C experts' point of favoring unsigned when possible. The int numbers range is sufficient and it's always safe to combine with arithmetic while unsigned types aren't. The computer ALU is signed 2's complement and there is no native unsigned computation in them. Unsigned is good for bitfields in my opinion. If incrementing, decrementing, comparing (=subtract), the CPU computes it as signed.

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1428r0.pdf

@singalsu hm, interesting, thanks. Being obviously very respectful of Mr. Stroustrup, I see some of his points, but I still couldn't quite understand all of them. He's saying, that mixing signed and unsigned types is error-prone. Of course it is. That's why it either shouldn't be done or only while carefully checking values. E.g. he's giving an example of auto a = area(height1-height2, length1-length2); which can bite if height1 < height2. Of course it can. But that would happen regardless of the type. You have to check ranges regardless of the type. Your check would just look different and with unsigned I find it would look more natural. E.g. if you know that your valid values are nonnegative and below a MAX_VAL, with unsigned you just check x < MAX_VAL while with signed you also have to check for nonnegative. He's emphasising that "unsigned is nonnegative with modular arithmetic". Sure, but isn't signed int the same and maybe even more confusingly so? Where you can add two large positive numbers and get a negative one? So how are signed integers "always safe to combine with arithmeti?" And no, not "when possible," but when makes sense. Also - I think more importantly - using unsigned for variables which should never be negative also serves as documentation.
This isn't critical, I'm certainly not even attempting to make this a blocker issue, having a variable of a signed type tells me that -1 is a valid value for it, while in your use cases it supposedly isn't.

I will think more the config blob format, there the e.g. channels might be better a channels mask as unsigned to select a specific mic channel for mono processing. I'm right now adding multiple channels processing to this. It could go to this PR or later as incremental. But these FFT sizes (math literature talks about size, but should I say length?) are used in equations, so int is natural for them.

lyakh · 2025-12-08T09:13:37Z

src/audio/stft_process/stft_process.h

+	int32_t *window; /**< fft_size */
+	int source_channel;
+	int prev_data_size;
+	int sample_rate;


also unsigned?

lyakh · 2025-12-08T09:14:05Z

src/audio/stft_process/stft_process.h

+	size_t frame_bytes;
+	int source_channel;
+	int max_frames;
+	int channels;


src/audio/stft_process/stft_process_common.c

src/audio/stft_process/stft_process_setup.c

kv2019i

Looks very good @singalsu ! Lot of helpful comments in the code, making this easier to follow. Nothing major in the review. It seems you have some todos left in code and commits, so I gather you will be updating this series still.

kv2019i · 2025-12-08T14:12:21Z

src/math/fft/fft_common.c

+	//for (j = 0; j < plan->num_ffts; j++)
+	//	for (i = 0; i < plan->fft_size; i++)
+	//		fprintf(fh1, "%d %d\n",
+	//			plan->tmp_o32[j][i].real, plan->tmp_o32[j][i].imag);


I think adding a "#ifdef DEBUG_DUMP_TO_FILE" or some such and keeping these in code is ok (and better than just having these commented out).

src/math/fft/fft_common.c

src/audio/stft_process/Kconfig

src/audio/stft_process/stft_process.c

singalsu · 2025-12-12T15:52:53Z

New version, with stereo processing. It was dual mono for 2ch. The review comments are addressed abd I hope I didn't miss essential. I don't agree with unsigned int usage preference for data sizes (as frames or samples) and channels count, so I didn't change those.

singalsu · 2025-12-12T15:54:51Z

Looks very good @singalsu ! Lot of helpful comments in the code, making this easier to follow. Nothing major in the review. It seems you have some todos left in code and commits, so I gather you will be updating this series still.

Still missing from this as functionality the log magnitude spectrum domain between FFT and IFFT. And the all the twiddle factors can be made cold. It can be incremental if this is merged, or I can keep updating this.

lgirdwood · 2025-12-17T13:24:13Z

@singalsu can you check CI, one build failure for plugin. Thanks !

singalsu · 2025-12-17T15:32:17Z

@singalsu can you check CI, one build failure for plugin. Thanks !

There's a mess with 16 bit FFT build triggered by MFCC but needing 32 bit FFT parts. I need to separate better the 16 bit and 32 bit FFT stuff for build success.

singalsu · 2025-12-17T17:39:00Z

@singalsu can you check CI, one build failure for plugin. Thanks !

There's a mess with 16 bit FFT build triggered by MFCC but needing 32 bit FFT parts. I need to separate better the 16 bit and 32 bit FFT stuff for build success.

Need to fix the cmocka unit test fail that I caused with the kconfig changes.

singalsu · 2025-12-18T10:52:57Z

src/math/fft/fft_32_hifi3.c


-	/* step 1: re-arrange input in bit reverse order, and shrink the level to avoid overflow */
-	inu = AE_LA64_PP(inx);
-	for (i = 1; i < size; ++i) {


Reminder to myself: Need to fix this also for 16 bit HiFi FFT. Currently there is no test for it. Need to check translating the tests to ztest with xt-xcc or xt-clang build.

singalsu · 2025-12-18T10:55:39Z

src/math/fft/fft_common.c

+
+	/* set up the bit reverse index */
+	for (i = 1; i < size; ++i)
+		bit_reverse_idx[i] = (bit_reverse_idx[i >> 1] >> 1) | ((i & 1) << (len - 1));


Yep, true it doesn't hurt to add it.

singalsu · 2025-12-18T11:00:27Z

src/audio/stft_process/tune/setup_stft_process.m

+    cfg.common_path = [cfg.tools_path 'tune/common'];
+    cfg.tplg_ver = 2;
+    cfg.ipc_ver = 4;
+    cfg.channel = 0;


Yes, I will update the configuration blobs to support the formats the frequency domain models need. This likely still changes. Also I'm not sure if the models are multi-channel. It may need also stereo to mono conversion and maybe some double mono output option.

singalsu · 2025-12-18T11:02:12Z

src/audio/stft_process/stft_process.h

+	STFT_PAD_START = 2,
+};
+
+enum sof_stft_process_fft_window_type {


Yep, the blob that is embedded to topology sets all the STFT parameters like window type and length.

singalsu · 2025-12-18T11:07:58Z

src/audio/stft_process/stft_process.h

+	int16_t frame_shift; /**< samples, e.g. 160 for 10 ms @ 16 kHz */
+	int16_t reserved_16;
+	enum sof_stft_process_fft_pad_type pad; /**< Use PAD_END, PAD_CENTER, PAD_START */
+	enum sof_stft_process_fft_window_type window; /**< Use RECTANGULAR_WINDOW, etc. */


Yes, it's an ABI. Enum is same size as int. Should I use e.g. int16_t and type cast it to the nicer looking enums when applied? I'd like to do it incrementally since these parameters will still change/add/delete when the models to be used with this are known better.

src/audio/stft_process/stft_process.toml

src/audio/stft_process/llext/llext.toml.h

The module versions of the functions mod_fft_plan_new() and mod_fft_plan_free() should be used instead. Since all previous usage of the functions has been updated these are safe to remove. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

The bit reverse order for index zero is the same but the scaling and copy can't be bypassed for inb[0]. The for loop needs to start from index zero. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

src/math/fft/fft_32_hifi3.c

This patch contains fixes and improvements for the FFT implementation. The aligned loads and stores can't be used for generic pointer arithmetic jumps, the pointer increment/decrement must be done by the instruction. Therefore the aligned operations are converted to non-aligned with requirement to align for 64 bits the FFT input and output buffers. The bit reverse ordering of data must start from zero, the start from one looks like a mistake. The change improves the accuracy when compared to reference FFT in Octave. In IFFT the output needs to be made complex conjugate. It was missing from the last scaling loop. These fixes also saved about 2 MCPS in STFT module usage. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch updates the headers document style to preferred style in SOF. The similar partial documentation comments are removed from window.c. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds 32 bit Q1.31 format window functions for better precision in audio processing. The supported window types are rectangular, Blackman and Hamming. The constant define for Blackman is suffixed with Q15 to help avoid using it with new 32 bit window and it's similar constant. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds the Hann window. It provides higher attenuation of side lobes vs. similar family Hamming window. Hann window is suitable for use in STFT overlap-add procedure. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds function mod_fft_multi_plan_new() and fft_multi_execute_32() to execute other than a power of two FFT size. The FFT sizes such 1536 that is 3*512 or 3072 that is 3*1024 are supported by this change. The procedure to compute this consists of doing three power of two FFTs, multiply the outputs with twiddle factors and doing DFT of size for the outputs. The produce is wrapped by the new functions those are used similarly as previous FFT functions. The fft-multi function can be also used for power of two FFTs. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds tests for the new functions with tests fft_multi and dft3. The reference test vectors data is created with Octave scripts ref_fft_multi.m and ref_dft3.m. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds script to create configuration blobs for STFT process module. A few blobs are created to be used with topology to test STFT. This is WIP. The blob format is not yet final. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds build of topologies to test the STFT process module. The topologies initialize the processing for 512, 768, 1024 and 1536 size FFTs. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This module provides analysis and synthesis filters for frequency domain processing. The used technique is short term Fourier transform (STFT) and inverse STFT. The FFT length, hop, and window type are configured with the bytes control configuration blob. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu · 2025-12-22T15:46:34Z

Added topology & blob with 1 ms FFT hop for LL pipeline processing example.

lgirdwood · 2025-12-22T21:19:19Z

@singalsu I've merged since functional CI has passed but pls do fix the doxygen warnings in incremental PR

[0/2] cd /home/runner/work/sof/sof/docbuild && doxygen sof.doxygen
/home/runner/work/sof/sof/src/include/sof/math/fft.h:114: warning: Found unknown command '@x'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:115: warning: Found unknown command '@y'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:114: warning: Found unknown command '@x'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:115: warning: Found unknown command '@y'

singalsu · 2025-12-23T09:43:36Z

@singalsu I've merged since functional CI has passed but pls do fix the doxygen warnings in incremental PR

[0/2] cd /home/runner/work/sof/sof/docbuild && doxygen sof.doxygen
/home/runner/work/sof/sof/src/include/sof/math/fft.h:114: warning: Found unknown command '@x'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:115: warning: Found unknown command '@y'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:114: warning: Found unknown command '@x'
/home/runner/work/sof/sof/src/include/sof/math/fft.h:115: warning: Found unknown command '@y'

Yep, I'll do right now a small PR for this. It takes still a while before the next cold twiddle factors data PR is ready.

lyakh reviewed Oct 16, 2025

View reviewed changes

singalsu commented Oct 16, 2025

View reviewed changes

singalsu force-pushed the stft_process branch 2 times, most recently from ae6f2f9 to 1b25adc Compare October 21, 2025 17:00

singalsu force-pushed the stft_process branch from 1b25adc to 355f15e Compare December 1, 2025 17:55

singalsu commented Dec 2, 2025

View reviewed changes

singalsu force-pushed the stft_process branch from 355f15e to 8ad9eeb Compare December 5, 2025 12:22

singalsu force-pushed the stft_process branch from 8ad9eeb to a7407a0 Compare December 5, 2025 13:36

singalsu marked this pull request as ready for review December 5, 2025 13:37

singalsu requested a review from ranj063 as a code owner December 5, 2025 13:37

Copilot AI review requested due to automatic review settings December 5, 2025 13:37

singalsu requested review from dbaluta, jsarha, kv2019i, lbetlej, lgirdwood, mmaka1 and plbossart as code owners December 5, 2025 13:37

Copilot AI reviewed Dec 5, 2025

View reviewed changes

singalsu force-pushed the stft_process branch from a7407a0 to 9f6246f Compare December 5, 2025 14:59

singalsu commented Dec 5, 2025

View reviewed changes

tools/topology/topology2/development/tplg-targets-bench.cmake Outdated Show resolved Hide resolved

singalsu commented Dec 5, 2025

View reviewed changes

lyakh reviewed Dec 8, 2025

View reviewed changes

kv2019i reviewed Dec 8, 2025

View reviewed changes

singalsu force-pushed the stft_process branch from 9f6246f to 151059e Compare December 12, 2025 15:49

kv2019i approved these changes Dec 17, 2025

View reviewed changes

singalsu force-pushed the stft_process branch from 151059e to a4c6d32 Compare December 17, 2025 17:23

singalsu force-pushed the stft_process branch from a4c6d32 to 38c085e Compare December 18, 2025 09:58

singalsu commented Dec 18, 2025

View reviewed changes

src/audio/stft_process/stft_process.toml Outdated Show resolved Hide resolved

src/audio/stft_process/llext/llext.toml.h Outdated Show resolved Hide resolved

singalsu force-pushed the stft_process branch from 38c085e to de72839 Compare December 18, 2025 16:36

singalsu added 2 commits December 18, 2025 18:38

Math: FFT: Fix a mistake in input data scaling and copy

1506e16

The bit reverse order for index zero is the same but the scaling and copy can't be bypassed for inb[0]. The for loop needs to start from index zero. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu force-pushed the stft_process branch from de72839 to fea4ba4 Compare December 18, 2025 16:49

singalsu commented Dec 19, 2025

View reviewed changes

src/math/fft/fft_32_hifi3.c Outdated Show resolved Hide resolved

singalsu added 7 commits December 19, 2025 19:41

Math: FFT: Add UUID to be able to trace errors

4bfcd38

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Math: Window: Update comments and prepare for 32 bit add

d9db7dc

This patch updates the headers document style to preferred style in SOF. The similar partial documentation comments are removed from window.c. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Math: Window: Add Hann window functions

a3a1925

This patch adds the Hann window. It provides higher attenuation of side lobes vs. similar family Hamming window. Hann window is suitable for use in STFT overlap-add procedure. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Math: Add cmocka tests for dft3() and fft_multi()

82fbca5

This patch adds tests for the new functions with tests fft_multi and dft3. The reference test vectors data is created with Octave scripts ref_fft_multi.m and ref_dft3.m. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu force-pushed the stft_process branch from fea4ba4 to 5243312 Compare December 19, 2025 17:49

singalsu added 3 commits December 22, 2025 17:33

Tools: Topology: Add STFT process test topology

c76d112

This patch adds build of topologies to test the STFT process module. The topologies initialize the processing for 512, 768, 1024 and 1536 size FFTs. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu force-pushed the stft_process branch from 5243312 to bb5e20a Compare December 22, 2025 15:45

lgirdwood approved these changes Dec 22, 2025

View reviewed changes

lgirdwood merged commit 02ff2c7 into thesofproject:main Dec 22, 2025
41 of 45 checks passed

Audio: STFT Process: Add new SOF module #10306

Audio: STFT Process: Add new SOF module #10306

Uh oh!

Conversation

singalsu commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singalsu commented Oct 21, 2025

Uh oh!

lgirdwood commented Oct 22, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

singalsu commented Dec 5, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kv2019i left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singalsu commented Oct 15, 2025 •

edited

Loading