跳到正文
DEV Community· yoking chan·· 2 小時前AI 評分58

yoking chan 建立無需上傳音訊的瀏覽器 MIDI 轉換工具

I turned a hummed melody into MIDI without uploading anything

AI 導讀

yoking chan 建立了一款在瀏覽器本機運行的音訊轉 MIDI 工具，使用 TensorFlow.js 執行 Spotify 開源 basic-pitch 模型，音訊毋須上傳即可轉成可編輯的 MIDI 檔案。

正文

How I run Spotify's basic-pitch audio-to-MIDI model entirely in the browser

I built an audio-to-MIDI converter where your audio never leaves your machine. Drop in a guitar riff, a hummed melody, or a piano recording, and you get an editable MIDI file back — no upload, no server, no queue. It runs Spotify's open-source basic-pitch model locally with TensorFlow.js.

[DEMO GIF]

Try it: https://tunenoodle.com/audio-to-midi

Two things pushed me client-side. Musicians are (rightly) protective of unreleased audio, and "upload your track to our servers" is a trust cost I didn't want to pay. The second reason is less noble: client-side inference means no GPU bill. The model is ~0.9 MB, modern browsers have everything else needed, so the tradeoff was easy.

Here's what the pipeline actually looks like.

Step 1 — Let the browser's audio graph do the decode and resample

basic-pitch expects mono audio at 22.05 kHz. Writing a resampler for that is unnecessary — the Web Audio API already has one. decodeAudioData handles any input format the browser can play (mp3, wav, ogg, m4a…), and an OfflineAudioContext with a 22,050 Hz sample rate does the downmix and resample in a single pass:

const decoded = await decodeFile(file) // decodeAudioData underneath

const offline = new OfflineAudioContext(1, Math.ceil(decoded.duration * 22050), 22050)
const source = offline.createBufferSource()
source.buffer = decoded
source.connect(offline.destination)
source.start()

// Float32Array, mono, 22.05 kHz — exactly what the model wants
const samples = (await offline.startRendering()).getChannelData(0)

Three lines of setup replaced what would have been a page of DSP code, and it reuses the browser's highly optimized resampler.

Step 2 — Hand the samples to a Web Worker without copying them

Inference takes seconds. That cannot run on the main thread, and a couple of minutes of audio is megabytes of Float32Array that I don't want to structured-clone. postMessage with a transfer list gives zero-copy handoff:

const worker = new Worker(
  new URL('./audio-to-midi.worker.ts', import.meta.url),
  { type: 'module' },
)

worker.postMessage(
  { samples, threshold, minFrames, modelUrl: '/models/basic-pitch/model.json' },
  [samples.buffer], // transferred, not copied
)

One bundler note: wrapping the worker path in new URL(..., import.meta.url) is what makes Vite emit it as a proper module worker instead of silently breaking on build.

Step 3 — Running basic-pitch in the worker

I self-host the model (~0.9 MB: model.json plus the weight shard) from my own origin rather than a CDN. Same-origin means it's cacheable, there's no third-party CORS or uptime dependency, and it keeps working under a strict CSP.

Inside the worker, the interesting part is short:

import { BasicPitch, noteFramesToTime, outputToNotesPoly } from '@spotify/basic-pitch'
import * as tf from '@tensorflow/tfjs'

await tf.setBackend('cpu')
await tf.ready()

const model = new BasicPitch(modelUrl)
const frames: number[][] = []
const onsets: number[][] = []

try {
  await model.evaluateModel(
    samples,
    (f, o) => { frames.push(...f); onsets.push(...o) },
    progress => self.postMessage({ progress }), // model reports progress per window
  )
  const notes = outputToNotesPoly(frames, onsets, threshold, 0.3, minFrames)
  self.postMessage({ notes: noteFramesToTime(notes) })
} finally {
  (await model.model).dispose()
}

A few deliberate choices:

  • CPU backend, forced. This is a one-shot batch job, and in a worker the CPU backend is the predictable one — no GPU context setup, no graphics-memory pressure competing with the page. For a model this size, the speed difference is measured in seconds, not minutes.
  • Progress comes free. evaluateModel calls back per analysis window, so I forward those numbers to the UI and the progress bar is honest.
  • dispose() in a finally. TF.js tensors are not garbage-collected promptly. Skip this and the third or fourth conversion in a session starts eating hundreds of MB.

Step 4 — From frame activations to notes

The model outputs per-frame pitch activations and onset activations. outputToNotesPoly turns them into note events, and three parameters matter for perceived quality:

const notes = noteFramesToTime(outputToNotesPoly(frames, onsets, threshold, 0.3, minFrames))
  • threshold is the confidence a pitch must reach — this is the sensitivity slider in the UI.
  • 0.3 is the onset threshold, kept fixed.
  • minFrames is how many consecutive frames a pitch must persist before it counts as a note. This is the parameter that kills flickering false positives on noisy input.

Then I map the notes to a MIDI document: round and clamp the pitch to the MIDI range, derive velocity from the note amplitude, and reject anything above 10,000 notes. The output lands in an in-browser MIDI editor where you can fix wrong notes and export a .mid file.

The unglamorous parts that matter in production

  • Cancellation is terminate(). A running evaluateModel loop won't politely stop, so I wire an AbortSignal straight to worker.terminate(). When a user drops the wrong 10-minute file, they should not wait for it.
  • "No notes found" is an outcome, not a bug. Silence, speech, and heavily percussive material legitimately produce nothing. It gets a specific error message and a hint ("try a clearer, more melodic recording"), not a silent failure.
  • Cap everything. A 20 MB limit before decoding, and transcription results above 10,000 notes get rejected outright. Both exist because a user will absolutely feed it a two-hour podcast.

The same pattern runs MP3 export

The converter's sibling feature — exporting a selection as MP3 — uses the identical worker + transferable pattern with lamejs. Two details from that side that cost me an afternoon:

const encoder = new Mp3Encoder(channels, sampleRate, 192) // kbps

for (let at = from; at < to; at += 1152) { // 1152 samples = one MP3 frame
  const pcm = Int16Array.from(chunk, v => {
    const c = Math.max(-1, Math.min(1, v))
    return Math.round(c * (c < 0 ? 32768 : 32767)) // asymmetric on purpose
  })
  out.push(encoder.encodeBuffer(pcm))
}
out.push(encoder.flush())

Feed encodeBuffer exactly 1152 samples at a time — that's the MP3 frame size, and the encoder's buffering gets weird if you guess wrong. And convert float to Int16 asymmetrically: −1.0 should map to −32768, but +1.0 must clamp to 32767, or your loudest frames overflow.

Try it

The tool is live at https://tunenoodle.com/audio-to-midi — it also pairs with a MIDI editor, a sheet-music scanner, and a MusicXML player if you want to go from recording to notation.

If you build something similar, or if you've hit the WebGL-vs-CPU tradeoff on TF.js in workers, I'd like to hear how you handled it in the comments.

來源：DEV Community · dev.to

#產品更新#多模態