Audio Analyzer

Pipeline Overview

src/managers/audioanalyzer.{h,cpp} pre-computes FFT visualization data for an entire audio track (up to 3 hours) on a worker thread, then serves time-indexed levels to QML based on the playback position. Local-file playback uses this precomputed data; mirrored-phone audio uses the separate live PCM pipeline. Both paths use 96 bars and 100 ms chunks and produce levels on a 0-8 integer scale.

 Audio File (.mp3/.flac/.m4a/...)
       |
       v
 +--------------------------------------+
 | Decoder (streamed, buffer by buffer) |  QAudioDecoder in the file's native
 |  QAudioDecoder                       |  format, one buffer at a time
 +--------------------------------------+
       |  each buffer on arrival
       v
 +------------------+
 | Preprocessing    |  Average channels to mono
 |                  |  Decimate to ~8 kHz (every Nth sample of the whole
 |                  |  track; phase carried across buffers)
 +------------------+
       |
       v
 +------------------+
 | FFT Analysis     |  100 ms chunks, each FFT'd as soon as it fills
 |  radix-2         |  zero-padded to 1024, Hann window, magnitudes, DC dropped
 |                  |  96 log-spaced bands (each >= 1 bin), mean per band
 |                  |  Only band magnitudes kept (~4 KB per second)
 +------------------+
       |
       v
 +------------------+
 | Normalization    |  95th percentile of all band values = 1.0
 | & Quantization   |  Power curve (^0.6)
 |                  |  int(norm * 8) -> 0..8
 +------------------+
       |
       v
 +------------------+
 | Stored as        |  chunks x 96 levels, one byte each,
 | m_fftData        |  flat std::vector<quint8>
 |                  |  10 chunks per second of audio
 +------------------+
       |
       v (each MediaManager position report, 100 ms, while shown + playing)
 +------------------+
 | update_position  |  chunk = int(seconds * 10); zeros past the end
 |                  |  Emit fftDataChanged only if levels changed
 +------------------+
       |
       v
 +------------------+
 | QML Visualizer   |  96 Rectangles, height = level / 8
 | (MediaRoom.qml)  |  Behavior animations smooth the steps
 +------------------+

Implementation

It is registered as the context property audioAnalyzer in src/main.cpp and consumed only by the waveform visualizer in frontend/MediaRoom.qml.

Bar Count and the Single Toggle

The visualizer renders a fixed 96 frequency bars (m_numBars = 96) and MediaRoom polls the position every 16 ms (about 60 updates per second). Earlier versions had tiered quality settings; commit 118f484 collapsed them into the single showWaveformVisualizer setting (default true, see Settings Reference). The MediaRoom container hard-codes what used to be the top tier: 96 bars, 16 ms timer, 120 ms InOutSine bar animations and pill-shaped "fancy" bars. There are no quality tiers left in the analyzer or the QML.

Audio Decoding

QAudioDecoder in the native format

The analyzer runs Qt Multimedia's QAudioDecoder in the file's native format — no setAudioFormat() — and does the rest itself. QAudioDecoder uses the platform backend: MediaCodec on Android, FFmpeg or GStreamer on Linux, Windows Media Foundation on Windows, AVFoundation on macOS, so supported inputs depend on that backend.

In the browser demo

Qt has no QAudioDecoder backend for WebAssembly, so there (OCTAVE_WASM) analyzeAudio() asks the page instead: WebMusicLibrary::decodeMono() (src/platform/wasm/weblibrary.{h,cpp}) queues a decode on the main thread, where the visitor's File is decoded by Web Audio in an OfflineAudioContext at 8 kHz (it resamples to its own rate) and mixed to mono, while the analysis worker waits (at most 60 s). The samples go into the same ChunkStream (factor 1 at 8 kHz), so the FFT, bands and levels below are the same as on the device. Only songs of a folder picked in the browser can be decoded.

Preprocessing

1. Mix to mono - appendMono() averages the channels of each QAudioBuffer into a float vector holding just that buffer. Float, Int16, Int32 and UInt8 sources are handled (integer samples are scaled to -1..1); nothing at the native rate is kept beyond the current buffer.
2. Decimate - a ChunkStream (an anonymous-namespace struct in audioanalyzer.cpp), initialised from the first buffer's rate, keeps every Nth sample with N = nativeRate / 8000, so a 44.1 kHz file becomes 8820 Hz and a 48 kHz file 8000 Hz (effective rate nativeRate / N). The decimation phase is carried across buffers (position counts native samples so far), so the result is every Nth sample of the whole track. There is no anti-aliasing filter; for a visualizer that is acceptable and it cuts the FFT work roughly five-fold.
3. Cap - once kMaxAnalysisSeconds (3 h) of decimated audio is reached, the drain stops, the decode loop quits early and an info line is logged.

bool init(int nativeRate)
{
    factor = std::max(1, nativeRate / 8000);
    const int sampleRate = nativeRate / factor;
    chunkSize = static_cast<int>(sampleRate * chunkDuration);
    if (chunkSize < 2)
        return false;
    fftSize = nextPow2(chunkSize);          // 1024 for 800 or 882 samples
    fftLen = fftSize / 2;
    window = AudioAnalyzer::hannWindow(chunkSize);
    edges = logBandEdges(fftLen, numBars);
    maxSamples = static_cast<long long>(kMaxAnalysisSeconds) * sampleRate;
    ...
}

There is no peak normalisation of the samples. Scaling the samples scales every band magnitude and the 95th-percentile reference by the same factor (see Global Normalization), so the 0-8 levels do not depend on sample scale.

Driving the decoder on a worker thread

Because the worker has no event loop of its own, the decode runs a local QEventLoop that quits on finished or error, with a 30 s watchdog so a stalled codec cannot hang the worker forever; any buffers still queued when finished fires are drained before the decoder is stopped. A decoder error, a timeout or no usable audio fails the analysis (bars stay at zero), and each case is logged with its own Qt warning message, including how many samples the watchdog case had managed to decode.

Why native format: the analyzer used to ask the decoder for 8 kHz mono Int16 output directly. On Qt 6.11's FFmpeg backend that request yields no buffers and no error for MP3 sources (WAV converted fine, which hid it), so every analysis ran into the 30 s watchdog, logged Analysis failed, and the visualizer stayed dark on the shipped binary. Decoding natively removes the dependency on the backend's format converter, and on Android it also sidesteps MediaCodec's format quirks. Analysis time depends on the track, device and Qt Multimedia backend.

QAudioDecoder decoder;                       // no setAudioFormat(): native rate/channels/format
decoder.setSource(QUrl::fromLocalFile(filePath));

ChunkStream stream(numBars, chunkDuration);
std::vector<float> mono; int nativeRate = 0;   // mono: one buffer only
QEventLoop loop;
auto drain = [&]() {
    while (decoder.bufferAvailable() && !stream.full() && !badRate) {
        appendMono(decoder.read(), mono, nativeRate);   // average channels -> float
        if (!stream.ready && !stream.init(nativeRate)) { badRate = true; break; }
        stream.add(mono);                                  // decimate, FFT each full chunk
    }
    if (stream.full() || badRate) loop.quit();       // 3 h cap reached
};
QObject::connect(&decoder, &QAudioDecoder::bufferReady, &decoder, drain);
QObject::connect(&decoder, &QAudioDecoder::finished, &loop, &QEventLoop::quit);
watchdog.start(30000);                          // sets timedOut, quits the loop
decoder.start();
loop.exec();
drain();
// stream.raw: numBars magnitudes per finished chunk -> global normalisation

FFT Pipeline

The preprocessed audio is split into 100 ms chunks (m_chunkDuration = 0.1, 10 chunks per second). chunkSize = int(effectiveRate * 0.1) is 882 samples for a 44.1 kHz source and 800 for 48 kHz; the chunk is zero-padded to the next power of two (1024) because the radix-2 FFT needs a power-of-two length. Each chunk is analyzed independently in ChunkStream::finishChunk():

Per-Chunk Processing

1. Window function - a Hann window (hannWindow(chunkSize), computed once per file) reduces spectral leakage at chunk boundaries; the windowed samples fill the first chunkSize slots of the zeroed complex buffer.
2. FFT and magnitude - fft_radix2() transforms the 1024-point buffer in place. The magnitudes of bins 1 .. fftSize/2 are used (DC is skipped), which at the 8 kHz effective rate is 0-4 kHz: 512 bins at about 7.8 Hz per bin for 8000 Hz audio, or 8.6 Hz for 8820 Hz audio.
3. Logarithmic frequency bands - the bins are grouped into 96 bands with logarithmic spacing, giving more resolution to low frequencies. The edges are computed once per file by logBandEdges(fftLen, numBars), since they depend only on the bin count:

   std::vector<int> edges(numBars + 1);
   edges[0] = 0;
   for (int i = 1; i <= numBars; ++i) {
       int idx = static_cast<int>(std::pow(double(fftLen), double(i) / numBars));
       idx = std::max(idx, edges[i - 1] + 1);    // every band at least one bin wide
       edges[i] = std::min(idx, fftLen);
   }
   edges[numBars] = fftLen;

   Edge i is N^(i/96) truncated, but forced to be strictly greater than the previous edge, so the first bands are one bin each (linear) until the log curve overtakes them.
4. Band averaging - each band's value is the mean magnitude of its bins (std::abs(buf[i + 1]), the +1 skipping DC); because the edges are strictly increasing, every band has at least one bin

Global Normalization (Two-Pass)

After the raw band magnitudes of every chunk are known (the only per-chunk data kept while decoding), a second pass normalizes and quantizes:

// 95th percentile of every non-zero band value in the track
std::vector<float> allValues;
for (float v : raw)
    if (v > 0.0f) allValues.push_back(v);

float globalMax = 1.0f;
if (!allValues.empty()) {
    const size_t idx95 = std::min(size_t(allValues.size() * 0.95), allValues.size() - 1);
    std::nth_element(allValues.begin(), allValues.begin() + idx95, allValues.end());
    globalMax = allValues[idx95];
}

for (size_t i = 0; i < raw.size(); ++i) {
    float norm = std::min(1.0f, raw[i] / globalMax);
    norm = std::pow(norm, 0.6f);                    // power curve for visual balance
    result.fftData[i] = static_cast<quint8>(norm * 8.0f);   // 0-8 integer levels
}

The reference is the element at index 0.95 × n found with std::nth_element (no interpolation), which is accurate enough for a visualizer.

Because the reference is a percentile of the magnitudes themselves, multiplying the input samples by any constant leaves the levels unchanged. That is why the earlier peak normalisation of the samples was dropped: with it removed, the output was verified byte-identical to the old code for MP3, AAC and WAV.

Frequency Band Distribution

Linear spacing would dedicate most bars to high frequencies, which are less musically interesting; logarithmic spacing mirrors how pitch is perceived. Boundary i of 96 sits at bin N^(i/96) (truncated to an integer, then bumped to at least the previous boundary + 1), where N = 512 is the number of magnitude bins up to the effective Nyquist frequency (4 kHz at 8000 Hz, about 7.8 Hz/bin; 4.41 kHz at 8820 Hz, about 8.6 Hz/bin). Approximate boundaries for the 8 kHz case:

Data Flow

Once analysis is complete, the playback position is mapped to a chunk on every call:

void AudioAnalyzer::update_position(float positionSeconds)
{
    if (m_fftData.empty() || !m_isActive)
        return;

    const int chunkIndex = std::max(0, static_cast<int>(positionSeconds * 10.0f));   // 10 chunks per second

    QVariantList varList;
    if (size_t(chunkIndex) >= chunkCount()) {
        for (int b = 0; b < m_numBars; ++b) varList.append(0);   // past the analysed part: flat bars
    } else {
        const quint8 *levels = m_fftData.data() + size_t(chunkIndex) * m_numBars;
        for (int b = 0; b < m_numBars; ++b) varList.append(int(levels[b]));
    }

    if (varList != m_currentLevels) {           // only emit on change
        m_currentLevels = varList;
        emit fftDataChanged(m_currentLevels);
    }
}

The change detection prevents a signal (and 96 QML property writes) when the position is still inside the same chunk. A position past the last analysed chunk (beyond the 3 h cap, or the sub-chunk tail of a track) yields 96 zeros rather than holding the last chunk. set_active(false) emits a list of 96 zeros to clear the display; clear() drops the data and remembered file.

Live Phone Audio

src/main.cpp connects phoneMirrorManager.audioPlayed (forwarded from ScrcpyClient) to feed_live_pcm(pcm, bufferedMs). The input is signed 16-bit little-endian PCM at 48 kHz, stereo, just written to the phone-audio sink; bufferedMs is how much the sink still holds ahead of it. set_live(true) selects this mode (main.cpp sets it from nowPlaying.source); the analyzer runs update_live() itself every kLiveFrameMs (50 ms) while a view is active and the phone plays. The window is 100 ms, so faster frames would mostly recompute the same samples; this used to be MediaRoom's 16 ms timer (60 FFTs a second on the GUI thread).

The feed is mixed to mono, decimated to 8 kHz and kept in a one-second ring buffer. Each update analyses a 100 ms window ending near the sample currently being heard, using the sink backlog and elapsed time since the last feed. It uses the same Hann window, frequency-band layout and 0–8 quantization as file analysis, but an adaptive reference: the frame's 95th percentile, a one-second exponential release, a 0.85 scale and a floor of 1.0. These operations run on the GUI thread. No PCM for more than 250 ms, or insufficient samples, produces zeros.

Only changed levels emit fftDataChanged. Switching live mode clears the displayed levels; set_active(false) also clears them. is_analyzed() reports file-analysis availability, not whether the live ring has audio.

Signals and Slots

Signals

Properties

Slots

Threading Model

Decoding and FFT of a whole track take seconds and must not block the Qt event loop, so analyzeAudio() runs off the GUI thread and only onAnalysisDone() touches member state, on the GUI thread afterwards.

QFuture<AnalysisResult> future = QtConcurrent::run(
    [filePath, bars, chunkDur]() {
        return AudioAnalyzer::analyzeAudio(filePath, bars, chunkDur);   // static, no member access
    });
m_watcher.setFuture(future);        // finished -> onAnalysisDone() on the GUI thread

analyzeAudio() is a static function that receives everything it needs as arguments; QFutureWatcher::finished delivers the result to onAnalysisDone() on the GUI thread. The destructor cancels and waits for the watcher so the app can quit mid-analysis.

Guard Conditions

• Missing file: an empty or non-existent path is rejected with a warning
• Skip if already analyzed: if the same path is requested and data exists, nothing happens
• Busy: if an analysis is running, only the latest requested file is remembered (m_pendingFile). When the worker finishes, a superseded result is discarded and the pending file is analysed next, so a fast skip sequence does not leave the visualizer on the old track's data (or blank if the old track was never analyzed). A request that lands during the chained analysis replaces the pending file again.
• Previous data cleared: starting a new analysis clears m_fftData, so the bars are not driven by the previous track meanwhile

QML Integration

The display consumer is the waveform container in frontend/MediaRoom.qml, visible when settingsManager.showWaveformVisualizer is on and playlistEmpty is false. The analyzer follows what is playing on its own: src/main.cpp feeds it nowPlaying's source (live for the phone; also for Android Auto, whose audio is not fed to it yet, so the bars stay still) and play state, and MediaManager's track and position. The page only reports whether its bars are on screen and draws the levels:

// On screen and not mid card animation (the decode of a new track waits for it)
readonly property bool _barsOn: visible && !mediaRoom._cardAnimBusy
on_BarsOnChanged: audioAnalyzer.setViewActive(_barsOn)

// 96 bars, one Rectangle each; the Row is cached as a single texture while the
// card animation runs (layer.live is off then) to collapse 96 draw calls into one blit
Row {
    layer.enabled: true
    layer.live: !mediaRoom._cardAnimBusy
    property int numBars: audioAnalyzer ? audioAnalyzer.numBars : 96
    Repeater { model: waveformBars.numBars
        Rectangle { height: Math.max(2, (barLevel / 8) * waveformContainer.height)
                    Behavior on height { NumberAnimation { duration: 120; easing.type: Easing.InOutSine } } }
    }
}

// Levels arrive by signal and are written straight into the bars (one list
// conversion per update; a binding per bar would convert it 96 times)
Connections {
    target: audioAnalyzer
    enabled: waveformContainer.visible && !mediaRoom._cardAnimBusy
    function onFftDataChanged(levels) {
        for (var i = 0; i < waveformRepeater.count; i++)
            waveformRepeater.itemAt(i).barLevel = (levels && levels[i]) || 0
    }
}

Track Changes

On a track change MediaRoom snaps every bar to zero (dropBars(), with the Behavior animations disabled while _cardAnimBusy is true so the drop is instant) and leaves the visualizer visible. The same flag holds the analyzer's view inactive, so the new track is decoded once the carousel settles; the bars sit at zero until the analysis lands and then animate up on the next position report. An earlier version faded the whole visualizer out for the duration of the analysis, which could leave the visualizer missing for several seconds while decoding.

Performance Considerations

Analysis Time

Working at 8 kHz is the primary optimization. A 3-minute song at 44.1 kHz has about 8 million samples; at 8 kHz that is about 1.4 million, split into roughly 1800 chunks of 800-882 samples, each a 1024-point O(N log N) FFT. The analyzer decodes at the file's native rate and decimates each buffer as it arrives, so the full-rate audio is never held beyond one decoded buffer.

Memory Usage

While decoding, only the per-chunk band magnitudes are kept: 96 floats per 100 ms chunk, about 4 KB per second of audio. The previous design decoded the whole track before analysing it, with a peak around 1.2 GB for a 1-hour file. Measured after the change: a 1-hour file went from 1260 MB to 330 MB process peak (the Qt baseline) and from 6.3 s to 4.1 s. Analysis is capped at 3 hours of audio (kMaxAnalysisSeconds).

The stored result is one byte per bar per chunk: a 3-minute song is about 1800 × 96 = 172,800 bytes (~170 KB) in a flat std::vector<quint8>. Only the current track's data is kept; a new analysis replaces it.

QML Rendering Impact

Each of the 96 bars is a separate Rectangle with an animated height, updated about 10 times per second by local-file data (up to the visualizer timer rate for live audio) but re-laid out every animation frame. The Row is rendered through a cached layer (layer.enabled), which is frozen (layer.live: false) during the card animation so the 96 gradient draws collapse into one texture blit. On low-powered devices the remaining cost is the bar animations themselves; disable the visualizer with showWaveformVisualizer if a target cannot keep up.

Signal Emission Throttling

Local playback is read at each MediaManager::positionChanged (its 100 ms position tick, which runs only while playing), which is exactly the chunk rate, and update_position() only emits when the levels differ, so local-file playback produces at most 10 updates per second with no wasted lookups (MediaRoom used to poll 60 times a second, 50 of them for nothing). Seeks select another chunk on the next report; live phone audio is analysed every 50 ms (20 per second).

Dependencies and Failure Handling

The analyzer needs only Qt Multimedia (QAudioDecoder, already linked for QMediaPlayer) and Qt Concurrent; the FFT and window are its own code. Which formats decode depends on the platform's Qt Multimedia backend.

A file that cannot be analysed degrades to "no visualizer" rather than an error: a decoder error, the 30 s watchdog, no usable audio or a track shorter than one chunk each leave m_fftData empty, log a warning (Analysis failed for … in the octave.audioanalyzer category) and skip analysisComplete, so is_analyzed() stays false and the bars stay at zero.

Design Decisions