[C++] The model is not detecting voice / Low Threshold Issues

[s1lviu]

Hello, folks!

Thanks to the authors for this contribution to the VAD space.

I'm trying to implement Silero into the SvxLink open source project in order to prevent kerchunking (when the people transmit noises over the radio network).

1. Here is the Silero VAD method
2. here is how I used it
3. and here is how I appended and extracted the necessary audio data

My main issue is that the model is very far away from detecting any voice. I think it comes from how I managed the audio processing.

As I have a limited C++ knowledge, but I really think I'm close to achieve that for the benefit of radio amateurs, and also think Silero would be the best fit for this use case. I hope someone can help me with advice about what I'm doing wrong in the current implementation.

Best regards,
Silviu

[s1lviu]

As an addition, when I speak in Romanian, the threshold values are positive and when I switch to English goes far away down.
@snakers4 or @adamnsandle any idea about that?
Many thanks in advance!

[chenqianhe]

Thank you very much for reaching out and for your enthusiasm in contributing to the open-source community, as well as for utilizing silero-vad in your project. I'm thrilled to see your engagement and initiative.

However, due to current time constraints, I'm unable to fully read and comprehend your code at this moment. I did take a preliminary look at your implementation, and I suspect the issue you're encountering might stem from how the model is being called and inference is conducted.

From what I gathered, when you have an audio clip that needs to be processed by the model for inference, it is crucial to segment this audio into smaller chunks as required by the model. Then, these segments should be fed into the model in a loop to obtain the final results. This approach is similar to the code snippet showcased in the screenshot I provided.

I hope this initial insight proves helpful. When time allows, I'll dive deeper into your code for a more thorough review. In the meantime, I'd recommend revisiting the model documentation regarding audio processing and segmentation to ensure alignment with the model's requirements.

Should you have any more questions or if there's anything else I can assist with, please feel free to reach out.

[s1lviu]

Thanks for your response, @chenqianhe!

The audio is segmented as you can find here and here the audio chunks are passed to the model.

All those are done according to the documentation, the model accepts them, but the results are problematic. Hope you can find some time to run the project (a made a run.sh script in the project root to make this easier).

Thanks,
Silviu

[pengzhendong]

Please save your audio frames to file, and use the official code to infer the speech probs.

[s1lviu]

Hi, @pengzhendong
I've tried to do that, but I cannot manage to obtain a valid audio file, even I try to save it as PCM or decoding from OPUS:

void Reflector::writeAudioToWavFile(const std::vector<uint8_t>& opusData, int sampleRate, const std::string& callsign) {
    const int numChannels = 1; // Mono
    const int bitsPerSample = 16;
    const int frame_size = 320;

    // Initialize the Opus decoder
    opus::Decoder decoder(16000, numChannels);
    if (!decoder.valid()) {
        std::cerr << "Failed to initialize Opus decoder." << std::endl;
        return;
    }

    // Decode the Opus data to PCM format
    auto pcmData = decoder.Decode({opusData}, frame_size, false);
    if (pcmData.empty()) {
        std::cerr << "Decoding failed or no data was decoded." << std::endl;
        return;
    }

    // Convert pcmData to uint8_t for WAV writing, considering the 16 bits per sample
    std::vector<uint8_t> wavData(pcmData.size() * sizeof(opus_int16));
    memcpy(wavData.data(), pcmData.data(), wavData.size());

    // Construct the output file path
    std::string desiredPath = "/home/silviu/Desktop/audio/";
    std::string filename = desiredPath + callsign + ".wav";

    // Try to open the file for reading to check if it exists
    std::fstream file(filename, std::ios::binary | std::ios::in | std::ios::out);

    // File does not exist, create a new file
    if (!file.is_open()) {
        file.open(filename, std::ios::binary | std::ios::out);
        if (!file.is_open()) {
            std::cerr << "Error opening output file: " << filename << std::endl;
            return;
        }

        // Corrected to use wavData for size calculation
        int fileSize = 36 + wavData.size(); // Correct calculation using the size of the decoded PCM data

        // Write the WAV header for a new file
        file.seekp(0, std::ios::beg); // Ensure starting at the beginning
        file.put('R').put('I').put('F').put('F');
        file.write(reinterpret_cast<const char*>(&fileSize), sizeof(fileSize));
        file.put('W').put('A').put('V').put('E');
        file.put('f').put('m').put('t').put(' ');

        int subChunk1Size = 16; // For PCM
        file.write(reinterpret_cast<const char*>(&subChunk1Size), sizeof(subChunk1Size));
        short audioFormat = 1; // PCM
        file.write(reinterpret_cast<const char*>(&audioFormat), sizeof(audioFormat));
        file.write(reinterpret_cast<const char*>(&numChannels), sizeof(numChannels));
        file.write(reinterpret_cast<const char*>(&sampleRate), sizeof(sampleRate));

        int byteRate = sampleRate * numChannels * bitsPerSample / 8;
        file.write(reinterpret_cast<const char*>(&byteRate), sizeof(byteRate));
        short blockAlign = numChannels * bitsPerSample / 8;
        file.write(reinterpret_cast<const char*>(&blockAlign), sizeof(blockAlign));
        file.write(reinterpret_cast<const char*>(&bitsPerSample), sizeof(bitsPerSample));

        file.put('d').put('a').put('t').put('a');
        int dataSize = wavData.size(); // Corrected to use wavData for the actual audio data size
        file.write(reinterpret_cast<const char*>(&dataSize), sizeof(dataSize));
    } else {
        // File exists; append new audio data at the end and update sizes
        file.seekg(0, std::ios::end);
        std::streampos endOfFile = file.tellg();
        int currentDataSize = endOfFile - 44; // Current size of the data chunk

        // Corrected to use wavData for updated data size calculation
        int newDataSize = currentDataSize + wavData.size(); // Updated size of the data chunk with decoded PCM data
        int newFileSize = newDataSize + 36; // Updated file size for the header

        // Update the data chunk size in the header
        file.seekp(40, std::ios::beg); // Position of data size in header
        file.write(reinterpret_cast<const char*>(&newDataSize), sizeof(newDataSize));

        // Update RIFF chunk size
        file.seekp(4, std::ios::beg); // Position of file size in header
        file.write(reinterpret_cast<const char*>(&newFileSize), sizeof(newFileSize));

        // Append new audio data at the end
        file.seekp(0, std::ios::end);
    }

    // Corrected to write (or append) wavData instead of opusData
    file.write(reinterpret_cast<const char*>(wavData.data()), wavData.size());

    file.close();
    //std::cout << "Audio data written to " << filename << std::endl;
}

[pengzhendong]

Why not use this wav.h?