Proposal for a revamped equalizer

[DarthRainbows]

I've spent a few days looking into the equalizer filter, and I came across some flaws that are baked into its implementation:

1. EQ band count is fixed at 8
2. frequency centers of EQ bands are unknown (and even reverse-engineering them has proven to be a challenge)
3. the FFT size is fixed at a rather small 256

A fixed count of EQ bands limits the use cases for the EQ. Allowing for custom band counts would be expand those.

Not knowing the frequency centers, however, impacts the usability of the equalizer. It becomes an exercise of guess and check to see if the result is what you want.

A small FFT size is good for performance, but it severely restricts frequency resolution. A 44.1kHz sample rate would result in 172Hz bins. That's a pretty large resolution; you can't build an equalizer with a band around 50Hz and another band around 120Hz, they're both in the same bin. As I understand it, 1024 bins is generally considered the minimum for an EQ, and 2048 or even 4096 for the professional grade stuff.

My C++ skills are 25 years out of date, so it would be nice to get some feedback from an expert on the feasibility of these suggestions.

FFTFilterInstance

• change STFT_WINDOW_SIZE, STFT_WINDOW_HALF, STFT_WINDOW_TWICE to class members
  • default to FFT size 256
• add a setFftWindowSize(unsigned int fftSize) method

Question: are there other places in the code base where an FFT size of 256 is assumed?

EqFilterInstance

I suspect that modifying this in-place while maintaining backwards compatibility might be more trouble than its worth. Perhaps it would be better to create an EqFilterV2Instance or some such.

• add a data structure to define an EQ band by:
  • center frequency
  • band width/Q-factor
  • gain
• do we need a compatible data structure to define the "wet" band? I don't think so, this is just a double in the range [0, 1] for the wet/dry mix
• add a new method setEqBands to set the bands for the EQ
  • this is the part that is hard for backwards compatibility
  • maybe we could set the default EQ bands in the constructor? That would require reverse-engineering the existing bands, which as stated above is challenging to do.
• add a new getEqBands method to retrieve the bands used so users can retrieve the frequency center, Q-factor, and gain of each band. This is especially important for users who use the default bands
• adjust other methods to account for variable EQ bands
  • bands would be numbered in ascending order by center frequency for methods that care about order
• adjust fftFilterChannel to use the defined frequency centers of the bands to determine how to adjust bins

consider: setEqBands could be the primary (only?) means of controlling the EQ state. Change the gain on one band, update them all. This is how I'd handle it in a higher-level language, but maybe the standards in C++ are different, particularly if there's a performance or other cost that makes this not a good idea.

It might even be beneficial to have multiple EQ filter types, or at least some way of configuring the EQ filter, as there are different algorithms for mapping the EQ bands to the bins:

• Catmull-Rom (this is used now)
• triangular
• gaussian
• biquad

There are probably more. Each has its own benefits and drawbacks in quality and performance.

[alnitak]

Hi @DarthRainbows, I'm glad of this discussion!

First of all I am not a great audio engeener and I am not always developed with C/C++ in the last few decades :)

The SoLoud library used in this plugin is meant to be as light as possible to be used mainly for games and it uses only code really open and free to use/modify.

Lately I come across the same doubt as your for the current EQ. I finally started an app using this plugin and I looked at the EQ filter too because of its very low quality. As you noticed, it uses a 256 FFT size which is quite low for an EQ filter, but it uses also an algorithm that is based on pitch shift bins which is not ideal.

The FFT is the key for this filter as you know, and the FFT code is not the best for performances compared to KissFFT or FFTW, but the author of SoLoud C++ lib wanted a really permissive license for all his code.

So, I tried to implement a new EQ filter already! It is still in progress and maybe you come in time to help me :). It is in the eq3 branch.
By now, it uses just 3 bands but it can be modified to have for example a method to set its number. It uses also a frequency, Q factor and the gain as you wished for each bands.
The FFT size used here is still 256, I didn't tried yet to use a bigger one, but SoLoud can handle an FFT size of 1024 and maybe others. I didn't understand well how the FFT data is stored when calling fft1024, because it seems to still use 256 as for the analyzer.cpp I used to get FFT data (and I admit that something should be reviewed at my side).

Question: are there other places in the code base where an FFT size of 256 is assumed?

Yes, for example when the SoLoud.setVisualizationEnabled is active and maybe in other SoLoud filters. So when you need (from Dart) the FFT data and you are using a filter that uses FFT, the FFT is computed 2 times I think.

Thanks for this discussion, and if you want to join experimenting or improving whatever you think, you are very welcomed!

[DarthRainbows]

The FFT is the key for this filter as you know, and the FFT code is not the best for performances compared to KissFFT or FFTW, but the author of SoLoud C++ lib wanted a really permissive license for all his code.

Ooura doesn't have support for any SIMD vector hardware acceleration, so it is rather slow. A little digging shows that the muFFT library is highly regarded for performance - even better than FFTW for FFTs size 2ⁿ - and is MIT licensed, so it would work well with flutter_soloud, even if the main soloud library wouldn't want to use it. KissFFT is slow like Ooura, AFAICT it doesn't have the SIMD support either. Do we want to go to the effort of changing the FFT engine out? Might be taking on too much for the immediate goal of making a more useful equalizer. A potential 100x speedup is hard to turn down tho.

Yes, for example when the SoLoud.setVisualizationEnabled is active and maybe in other SoLoud filters. So when you need (from Dart) the FFT data and you are using a filter that uses FFT, the FFT is computed 2 times I think.

I was thinking more in terms of potentially breaking functionality by making the FFT no longer restricted to 256. Like, does the visualizer assume a size of 256? We'd need to be on the lookout for those things while testing.

---

Just for a first-order approximation of effort, I asked Gemini to analyze the complexity of replacing the Oouma FFT algorithm with the muFFT package. Assuming it didn't hallucinate anything here, the task doesn't seem so bad. Still probably worth making into its own project, IMO, especially if we don't have a C++ expert on hand to do the work.

Analysis of the Current FFT Implementation

The current FFT functionality is encapsulated within the SoLoud::FFT namespace, with its interface defined in src\soloud\include\soloud_fft.h and implementation in src\soloud\src\core\soloud_fft.cpp.

The public API consists of:

• fft(float *aBuffer, unsigned int aBufferLength)
• ifft(float *aBuffer, unsigned int aBufferLength)
• Specific-size wrappers: fft1024, fft256, ifft256.
  The implementation uses Takuya Ooura's FFT code, which performs a real-to-complex FFT. For an input buffer of N real-valued floats, it produces an in-place complex-valued output in a packed format:

• aBuffer[0]: Real part of DC component (frequency 0).
• aBuffer[1]: Real part of Nyquist component (frequency N/2).
• aBuffer[2*k] and aBuffer[2*k+1]: Real and imaginary parts for frequency k (for 1 <= k < N/2).
  The inverse FFT (ifft) expects this packed format and transforms it back into N real-valued floats.

Usage in the Codebase

The primary user of SoLoud::FFT is the FFTFilter and its derived classes (like EqFilter, BassboostFilter, etc.). The core logic is in src\soloud\src\filter\soloud_fftfilter.cpp.

The process in FFTFilterInstance::filterChannel is as follows:

1. A real-valued signal of STFT_WINDOW_SIZE (256) samples is prepared in a temporary buffer.
2. FFT::fft() is called on this buffer.
3. The packed complex data is then processed by fftFilterChannel(). This often involves converting to magnitude/phase representation using helper functions like comp2MagPhase().
4. After processing, the data is converted back to the packed complex format (e.g., with magPhase2Comp()).
5. FFT::ifft() is called to get the real-valued signal back.

Analysis of muFFT Library

I will perform a search to understand the muFFT library's API.

Based on the search results, muFFT is a single-precision FFT library optimized for SIMD architectures (SSE, AVX, NEON). It supports real-to-complex and complex-to-real transforms. Its data layout for complex numbers is an interleaved [real, imag, real, imag, ...] format, which is exactly what the current SoLoud FFT implementation uses.

Complexity Estimation

Replacing the existing FFT implementation involves the following steps:

1. Remove Old Code: Delete the fftimpl namespace and the contents of src\soloud\src\core\soloud_fft_lut.cpp
2. Add muFFT Source: Add the muFFT source files to the SoLoud project/build system.
3. Modify SoLoud::FFT Implementation: Update the functions in src\soloud\src\core\soloud_fft.cpp

• fft(float *aBuffer, unsigned int aBufferLength): This function would need to be changed to call the muFFT real-to-complex transform function. This will likely involve:
  • Creating a muFFT plan/handle for the given aBufferLength. This should probably be cached for performance if the same size is used repeatedly (which it is: 256).
  • Calling the muFFT execution function.
  • muFFT's output for a real-to-complex transform of size N is N/2 + 1 complex points. The current implementation produces N/2 complex points in a packed format. The output format of muFFT would need to be adapted to match what the rest of the FFTFilter code expects, or the FFTFilter code would need to be adapted to muFFT's output. Given the packed format, some minor data shuffling might be necessary.
• ifft(float *aBuffer, unsigned int aBufferLength): This would be replaced with a call to muFFT's complex-to-real transform.
  Similar to the forward FFT, this would require creating and executing a muFFT plan.
  The input buffer aBuffer would need to be arranged in the format muFFT expects for its complex-to-real transform. This might require shuffling the packed data from SoLoud's format into a standard interleaved complex format.
  The ifft in SoLoud also includes a normalization step (aBuffer[i] *= 1.0f / float(aBufferLength / 2)). muFFT might handle normalization differently, so this would need to be adjusted or removed.

Evaluation

• API Compatibility: The data layout (interleaved float array for complex data) is compatible. Both are single-precision. This is a major advantage.
• Functionality: muFFT provides the necessary real-to-complex and complex-to-real transforms.
• In-place vs. Out-of-place: The current implementation is in-place. muFFT is an out-of-place implementation, which means a separate output buffer would be required. This adds a minor complexity, as the result would need to be copied back into the original buffer to maintain the current API contract.
• Packed Format: The biggest hurdle is the difference in handling the packed format for real FFTs. SoLoud's current implementation has a specific packing for DC and Nyquist frequencies. muFFT produces N/2 + 1 complex values. The code in FFTFilter that processes the FFT output (e.g., comp2MagPhase) would need to be adjusted to handle muFFT's output format, or a conversion layer would need to be written. Given that comp2MagPhase and its counterparts are used throughout the filter implementations, modifying them seems more direct than trying to force muFFT's output into the old format.
• Build System: Integrating a new library requires modifying the build files (CMake, etc.), which is a standard, albeit sometimes tedious, part of software development.
• Performance: muFFT is optimized for SIMD and is likely to be faster than the current implementation, which is the main motivation for such a change. It also includes runtime detection of CPU features.

Complexity Score

On a scale of 1 to 10:

• 1-2 (Trivial to Easy): If it were a drop-in replacement with an identical API and data format. This is not the case.
• 3-4 (Moderately Easy): This seems to be the most accurate rating. The core concepts are the same, and the data types match. The main work involves:
  1. Writing wrapper functions.
  2. Handling the out-of-place nature of muFFT.
  3. Adjusting the FFTFilter helper functions (comp2MagPhase, etc.) to work with muFFT's standard N/2 + 1 complex output format instead of the current custom packed format. This is the most significant part of the work but is localized to soloud_fftfilter.cpp.
• 5-7 (Moderate): This would be the case if data types were different (e.g., double vs. float) or if the data layout was fundamentally incompatible (e.g., split real/imaginary planes vs. interleaved).
• 8-10 (Hard to "Scrap everything and start over"): This would apply if the fundamental approach to filtering was incompatible with muFFT, requiring a rewrite of the entire FFTFilter logic. This is not the case.
  The changes are mostly confined to soloud_fft.cpp and soloud_fftfilter.cpp. The rest of the codebase, including the public-facing API in soloud.dart, would not be affected. The effort is localized and doesn't have wide-ranging consequences.

Therefore, the complexity is low-to-moderate.

Final Complexity Score: 4

It's more than a trivial drop-in replacement due to the differences in the packed real FFT format and the in-place vs. out-of-place nature, but it does not require architectural changes and the modifications are confined to a small number of files. An experienced C++ developer should be able to perform this change without major difficulties.

[alnitak]

A potential 100x speedup is hard to turn down tho.

I completely agree!! So much so that I'd like to start thinking using muFFT.

I was thinking more in terms of potentially breaking functionality by making the FFT no longer restricted to 256. Like, does the visualizer assume a size of 256? We'd need to be on the lookout for those things while testing.

if I remember well, when the Soloud::ENABLE_VISUALIZATION flag is enabled, the mixer, on each new audio data to process, will fill a 256 array of floats with the current audio buffer regardless of its size (a 2048 buffer will be shrinked to 256). This data can then be get using getWave(). When needed, the getFFT() function will calculate the FFT using the current 256 array of audio data. This is not the best solution IMHO, but it's what we have.
In light of this, I really think that the FFT is used only for visualization when calling getFFT and in the soloud_fftfilter.cpp which is only used by soloud_eqfilter.cpp and soloud_bassboostfilter.cpp.

So, there could be few or not big changes in the SoLoud core and only those 2 filters should be changed a lilttle. Even the way the FFT is obtained for visualization purposes could probably be updated without too much trouble.

I'll be taking a look at this soon, and if you'd like to join, you're welcome!

[DarthRainbows]

I'm looking. This C++ is quite different to what it was like when I was writing homework assignments in college 🤣

[alnitak]

I looked at muFFT and unfortunately it seems that it doesn't support arm64 architectures. Looking at the issues it's an old wish that come up with a fork.

We could try that fork or maybe try to find something else like FFT-implementation-in-C.

[alnitak]

I also come to your conclusions. Last week, I tried to use the original pffft just because it has less hassle to configure it since it has just a .c and a .h :)

I will try, hopefully soon, the marton78/pffft, but I haven't looked at how to use cmake on Mac and iOS yet.

I have modified the parametric_eq3_filter to use pffft.

I have also done some tests (don't know if this is the correct way to do this) using the SoLoud Ooura FFT and pffft performing 100000 iterations using random samples:

Samples	Linux AMD 5950x	Samsumg S22	Mac Mini M4
256	SoLoud: 75 ms pffft: 58 ms 21.93% faster	SoLoud: 90 ms pffft: 86 ms 4.70% faster	SoLoud: 52 ms pffft: 36 ms 30.85% faster
512	SoLoud: 155 ms pffft: 130 ms 15.68% faster	SoLoud: 193 ms pffft: 189 ms 1.82% faster	SoLoud: 83 ms pffft: 80 ms 3.51% faster
1024	SoLoud: 361 ms pffft: 271 ms 24.72% faster	SoLoud: 464 ms pffft: 398 ms 14.23% faster	SoLoud: 201 ms pffft: 167 ms 17.22% faster
2048	SoLoud: 748 ms pffft: 643 ms 14.07% faster	SoLoud: 975 ms pffft: 885 ms 9.16% faster	SoLoud: 431 ms pffft: 372 ms 13.73% faster
4096	SoLoud: 1755 ms pffft: 1330 ms 24.18% faster	SoLoud: 2269 ms pffft: 1897 ms 16.41% faster	SoLoud: 1007 ms pffft: 784 ms 22.18% faster

pffft seems better, but not as much as I was expecting! I think that marton78/pffft will be much more performant.

[DarthRainbows]

Those results don't seem right, like PFFFT isn't using the vector hardware.

I know a guy who is way better at c++ than me, ad he's volunteered to look into replacing the FFT.

[alnitak]

I've tried with the "original" pffft and I think it is using at least SSE extensions.

Also consider that the ms in that table I posted are for 100000 batches, resulting in a 0.29ms for a Dual-channel 44.1kHz audio (4096 samples on Linux, 1330/100000*22).

All this only if I understand what I did :). Did I miss something?

[DarthRainbows]

The performance you are seeing from the Samsung S22 is comparable to the benchmarks for the Raspberry Pi 3b+, which leads me to suspect there's something funny going on. The S22 should leave the RPi3b+ in the dust, even if you ran the benchmarks exclusively on the S22's low-power Cortex-A510 cores.

One thing to consider for benchmarking, are you using the real-valued, half-complex FFT or the complex-valued FFT? The real-value FFTs have half as many calculations (2.5Nlog2(N) vs 5Nlog2(N)).

[alnitak]

The test, I posted above, has the only purpose to see how much faster pffft is compared to the one used by SoLoud in a real situation (also don't know where to start doing a benchmark like the one you see around). So, to test a use case for an equalizer, I did an FFT followed by an IFFT using Ooura and the same using pffft.

I just wanted to know how much it's worth to spend some time trying marton78/pffft.