Environmental sound recording & keyboard sampler performance pad | ZeroTools

We will explain in detail the basic technology for capturing audio signals input from physical microphone devices as digital data in a browser environment and processing them with high resolution and low latency.

This system obtains access to the user's hardware microphone and establishes an audio stream by calling the getUserMedia method of the MediaDevices interface.

At this time, the sampling rate can be specified over a wide range from the standard 44100Hz to a maximum of 192000Hz, and the quantization bit rate is also directly mapped to a floating-point arithmetic-based array as a Float32Array within the audio context.

The MediaRecorder API is responsible for encoding the obtained MediaStream in chunks, but the essence of this engine is to bypass the stream to the AudioWorkletNode using createMediaStreamSource of AudioContext, and asynchronously process the raw waveform data in frames.

This architecture minimizes jitter and buffer underrun when passing through the OS's audio stack, and makes it possible to continuously store PCM 16-bit equivalent or Float32 waveform arrays for internal processing in memory, fully synchronized with timestamps.

During recording, the audio buffer is dynamically expanded as a ring buffer, with garbage collection avoidance algorithms applied to prevent memory exhaustion.

We will explain the trimming function and its playback control mechanism for eliminating unnecessary silent sections and noise parts from the acquired primary recording data and extracting samples that can be played back at musical timing.

A slicing process is performed on the Float32Array waveform data expanded in memory, which strictly specifies the starting In point and ending Out point in milliseconds or sample frames.

The trimmed waveform snippets are allocated in the memory pool as separate AudioBufferSourceNode instances from Sampler Pad 1 to Sampler Pad 8.

Each pad has its own independent playback queue, managed by a dynamic node creation/destruction cycle that does not violate polyphony restrictions even when multiple pads are playing simultaneously.

Additionally, a triggering algorithm has been implemented that captures specific key-down events on the computer's physical keyboard and instantly converts them from the browser's Event Loop to scheduled playback based on the AudioContext's currentTime.

This algorithm uses a high-precision timer based on performance counters rather than requestAnimationFrame, and by keeping the latency from keyboard input to pronunciation below the limits of human perception, it provides a one-shot playback feel comparable to dedicated hardware samplers such as MPC.

We describe the structure of the signal processing pipeline that provides volume control and dynamic changes in acoustic characteristics during playback for the audio waveforms assigned to each sampler pad.

For volume optimization, connecting GainNodes in series and using the linearRampToValueAtTime method activates a zero-crossing detection-based envelope generator that prevents clicks caused by sudden waveform fluctuations during attack and release.

Reverb and echo effects, which are implemented as spatial effects, are realized by impulse response convolution using ConvolverNode, or by a recursive delay circuit that combines DelayNode and Feedback GainNode.

This allows you to control the decay time of the reverberation component and the wet/dry mix ratio in real time.

Furthermore, regarding the pitch shift function, in addition to the traditional method of changing the playback speed and pitch at the same time by simply manipulating the playbackRate property of the AudioBufferSourceNode, by running a phase vocoder or granular synthesis algorithm inside the AudioWorklet, advanced signal processing is immediately applied that changes only the pitch by semitones or cents while maintaining formant characteristics while maintaining time stretch, which is the length of the time axis.

When handling highly confidential biometric information such as user voice data, we will discuss the security advantages of a complete client-side processing model that does not involve any communication to an external server.

All audio buffers recorded, edited, and processed by this tool are stored only in an isolated sandbox environment such as IndexedDB or inside the browser's volatile RAM.

Background data transmission processes through XMLHttpRequest and Fetch API are not physically implemented, and it is technically guaranteed that audio data will not be leaked externally even on the network inspector.

Regarding allowing microphone access, the browser only allows calls to getUserMedia in a secure context where an HTTPS connection is required, making it impossible for a man-in-the-middle attack to intercept the audio stream.

At the end of a session or the moment an explicit reset operation is performed by the user, all references to the Float32Array held in memory are intentionally severed and irreversibly discarded by the browser's garbage collector.This design reduces the risk of restoring audio from residual data to zero even in environments where devices are shared.

This article explains the encoding and exporting method for exporting audio buffers edited and processed in the browser to the local file system as a general-purpose media format.

When exporting to WAV format for uncompressed, high-quality audio storage, left and right channel PCM data is extracted from AudioBuffer, interleaved, and a standard WAV file header containing RIFF chunks, fmt chunks, and data chunks is manually constructed at the binary level.

This binary data is encapsulated as a Blob object. For conversion to MP3 format, the LAME encoder library compiled into Wasm is run asynchronously on the Web Worker to compress the PCM stream into fixed bitrate or variable bitrate frames without interfering with the main thread's UI drawing.

On the other hand, the WebM format uses a method of packing the Opus audio codec natively output by the MediaRecorder API into a container, achieving both extremely high compression efficiency and sound quality.

A temporary memory URI is allocated to the generated Blob object via URL.createObjectURL, and by dynamically firing the download attribute of HTMLAnchorElement by clicking it from the program, it is possible to save it directly to the user's local disk.

We will discuss specific operating procedures and application methods for performing a variety of audio tasks by making integrated use of the recording function and real-time waveform control capabilities of this system.

When recording simple voice memos, insert effects are bypassed and unprocessed PCM data is recorded straight to ensure high clarity suitable for later transcription and audio analysis.

On the other hand, in the process of building a sound effect sampler for video production and game development, it is effective to collect environmental sounds and object sounds with a microphone, then use a waveform editor to sharply cut out only the attack component, and add a gain boost and a weak short delay to amplify the presence of impact sounds and explosion sounds.

Furthermore, for live performances, by recording and assigning elements such as kicks, snares, hi-hats, and vocal chops to each slot from Sampler Pad 1 to 8 in advance or in real time on stage, and using keyboard shortcuts like a MIDI controller, it is possible to improvise beats with finger drumming and mashup performances that make extensive use of pitch shifting, all in a single browser.

Low-latency audio routing and an intuitive pad interface combine to make it more than just a recording tool, it's a practical live instrument.