Independent control architecture for time expansion and pitch transition
In the digital reproduction of audio waveforms, the playback speed and pitch are physically closely related, and simply changing the sampling rate or increasing or decreasing the playback speed will cause a proportional change in the frequency characteristics.
This system implements a phase vocoder algorithm using phase inversion and window functions and the core processing architecture of the SoundTouch JS library on the browser in order to completely separate and manipulate these two acoustic elements.
In order to achieve time stretching while maintaining the pitch or pitch shifting while keeping the speed constant, which cannot be achieved using the playbackRate property of the AudioBufferSourceNode provided in the general Web Audio API alone, advanced digital signal processing is required to divide the input waveform into minute time frames and reconstruct it using the overlap add method.
By dynamically switching between waveform connection processing in the time domain and phase correction processing in the frequency domain, it is possible to achieve extremely natural sound modulation with minimal artifacts, from percussion sounds with sudden transient characteristics to sustained vowel components.
Dynamic parameter interpolation and mapping of chromatic scales and speed multipliers
In order to accurately meet musical and phonetic demands, this mechanism allows pitch changes to be specified in the range of minus 12 semitones to plus 12 semitones, that is, one octave above or below, and is equipped with a parameter interpolation engine that continuously and smoothly controls the speed multiplier from 0.5 times, which is half the original tempo, to 2.0 times, which is double the original tempo.
The specified chromatic scale value is internally precisely logarithmically converted into a frequency scaling factor and passed directly to the calculation algorithm.
At the same time, the speed multiplier variation is calculated as the stride width of frame skipping and duplicate reading, and these two factors are applied to the calculation grid independently without interfering in real time.
Even when dynamic parameters are changed during playback by user operations, the smoothing circuit is constantly active, detecting zero-crossing points in the audio waveform and asynchronously updating the coefficients to thoroughly eliminate click noise and glitches.
Real-time streaming preview and PCM binary rendering
We are building a real-time streaming listening circuit that utilizes AudioWorklet so that the user can immediately check the acoustic results of the modulation parameters specified.
On a background thread dedicated to audio processing, separate from the main thread, instances of SoundTouch JS sequentially perform time-stretching and pitch-shifting operations for each audio block, providing a delay-free audio preview for user interface operations.
On the other hand, for the final file output, an asynchronous PCM binary rendering process using OfflineAudioContext is started. This export-only rendering pass frees you from the constraints of calculation load during real-time playback, enables oversampling processing using higher calculation precision and a longer filter length, and performs waveform reconstruction with top priority on sound quality.
Confidential audio data pipeline with in-browser local processing
To fully protect user privacy, such as sensitive meeting recordings and unreleased music tracks, all signal processing in this system is designed to be completed locally within the user's web browser, without transmitting data to a server via a network.
Audio files loaded through the File API are decoded directly within the browser's memory space and quickly expanded as uncompressed audio buffers in Float32Array format.
In order to prevent memory leaks and browser crashes even for long audio data lasting several tens of minutes, we have adopted an advanced memory management method by dividing stream chunks, and have established a robust pipeline that continues stable audio processing in cooperation with the browser's garbage collection mechanism even for huge files.
WAV and MP3 export conversion mechanism for processed data
The modulated PCM data generated by the audio rendering process is passed to the encoding process for external output as a versatile file format.
When outputting in WAV format, RIFF chunks and fmt chunks are constructed accurately at the binary level, and interleaved PCM samples are directly written into the data chunks to generate an uncompressed file stream with no audio quality degradation.
Additionally, for export requests in MP3 format, which has a smaller file size, it is equipped with a mechanism that directly executes a highly compiled LAME encoder binary as WebAssembly on the browser, generates a compressed stream at high speed while applying bit rate control and joint stereo optimization, and immediately saves it as a Blob object to the user's local storage.
Application-specific models and domain-specific parameter tuning
The system's independent control model is designed to be highly practical in a variety of specialized use cases. In shadowing training for language learning, by slowing down the playback speed to about 0.7 times while maintaining pitch perfectly, it strongly supports accurate imitation of native speaker pronunciation without damaging the fine phonetic structures such as consonant plosives and vowel transitions.
When used for copying by ear for musical instrument players, it not only analyzes the fingerings of complex fast-playing phrases by extremely slowing down the playback speed, but also uses a pitch shifter to correct the pitch in semitone increments for songs with different tunings to match the absolute pitch of the instrument on hand, all by simply using a browser.
Furthermore, when practicing dance choreography, we provide a precise audio control platform tailored to the needs of each domain, such as making it possible to gradually speed up training by varying only the tempo while maintaining the groove of the song and the key of the vocal.