Add PANNs CNN14 AudioSet tagging (fp16 CNN body + host log-mel assets)

README.md

@@ -0,0 +1,60 @@
+---
+license: cc-by-4.0
+library_name: LiteRT
+pipeline_tag: audio-classification
+tags: [litert, tflite, on-device, android, gpu, audio-tagging, audioset, sound-event-detection, panns, cnn14, fully-gpu]
+---
+
+# PANNs CNN14 — LiteRT (on-device AudioSet tagging, GPU CNN + host log-mel)
+
+[PANNs](https://github.com/qiuqiangkong/audioset_tagging_cnn) **CNN14** (`Cnn14_mAP=0.431`) general sound-event tagging, converted to **LiteRT** with the CNN body running **fully on the `CompiledModel` GPU** (ML Drift) on Android. Given ~10 s of audio it predicts probabilities over the **527 [AudioSet](https://research.google.com/audioset/) classes** — speech, music, instruments, animals, vehicles, alarms, household sounds, and so on. AudioSet tagging is **multi-label**: several tags can be high at once.
+
+```
+waveform[320000] (32 kHz) →[host: log-mel]→ logmel[1,1,1001,64] →[GPU: CNN14]→ probs[1,527] (sigmoid)
+```
+
+## On-device (Pixel 8a, Tensor G3 — verified)
+
+| | |
+|---|---|
+| nodes on GPU | **45 / 45** LITERT_CL (full residency, single graph, 1 partition) |
+| inference | **~124 ms** GPU + ~99 ms host log-mel ≈ **0.22 s** per 10 s clip |
+| size | 162 MB (fp16) |
+| accuracy | fp16 tflite-vs-PyTorch corr **1.000000**; self-test top tag "Speech" |
+
+## How it converts (litert-torch) — and why the log-mel is host-side
+
+PANNs builds its spectrogram with **torchlibrosa**, whose STFT is a *DFT-as-Conv1d* — so there is **no FFT op** and the whole raw-audio→tags graph is almost GPU-clean; the only blocker is the STFT centering **reflect-pad** (one `GATHER_ND`, removable via `pad_mode='constant'`, corr 1.0). **But** the converted spectral front-end is unusable: litert-torch lowers the giant 1024-tap DFT-conv incorrectly (fp32 tflite corr ≈ 0.19), and the power spectrum `|STFT|²` (~1e6) **overflows fp16 on Mali → NaN**.
+
+So the spectral front-end is computed on the **CPU** (the Whisper/Kokoro pattern), matched to torchlibrosa exactly, and only the CNN body rides the GPU:
+
+- **log-mel (host)** — reflect-pad center, periodic Hann, 1024-pt FFT, power, mel matmul (`librosa.filters.mel`, slaney), `10·log10(max(mel,1e-10))`. Validated host-vs-torch corr **1.000000** (max|d| 0.0017). The mel basis is shipped here as `mel_basis.bin` [64, 513].
+- **CNN14 body (GPU)** — `bn0` + 6 conv blocks + mean/max time-pool + 2 FC + sigmoid. Pure CNN, converts at corr **1.000000** in fp32 **and** fp16, op-check banned NONE / >4D 0, one delegatable graph.
+
+## Files
+
+| File | What |
+| ---- | ---- |
+| `cnn14_audioset_fp16.tflite` | the CNN body, fp16, input logmel [1,1,1001,64] → probs [1,527] |
+| `mel_basis.bin` | mel filterbank [64, 513] float32 for the host log-mel |
+| `audioset_labels.txt` | the 527 AudioSet class display names (row index = class id) |
+| `build_panns.py` | conversion + host-mel validation script |
+
+## Preprocessing
+
+Mono **32 kHz**, padded/truncated to 10 s (320000 samples), values in [-1, 1]. Compute the log-mel as above → [1,1,1001,64]. The output 527 sigmoid probabilities are per-class (multi-label); take the top-K as tags.
+
+## License
+
+Code [Apache-2.0](https://github.com/qiuqiangkong/audioset_tagging_cnn/blob/master/LICENSE); weights `Cnn14_mAP=0.431.pth` [CC-BY-4.0](https://zenodo.org/record/3987831) (Zenodo). AudioSet ontology © Google, [CC-BY-4.0](https://research.google.com/audioset/). Upstream: [qiuqiangkong/audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn).
+
+## Citation
+
+```bibtex
+@article{kong2020panns,
+  title={PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition},
+  author={Kong, Qiuqiang and Cao, Yin and Iqbal, Turab and Wang, Yuxuan and Wang, Wenwu and Plumbley, Mark D},
+  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
+  year={2020}
+}
+```

audioset_labels.txt

@@ -0,0 +1,527 @@
+Speech
+Male speech, man speaking
+Female speech, woman speaking
+Child speech, kid speaking
+Conversation
+Narration, monologue
+Babbling
+Speech synthesizer
+Shout
+Bellow
+Whoop
+Yell
+Battle cry
+Children shouting
+Screaming
+Whispering
+Laughter
+Baby laughter
+Giggle
+Snicker
+Belly laugh
+Chuckle, chortle
+Crying, sobbing
+Baby cry, infant cry
+Whimper
+Wail, moan
+Sigh
+Singing
+Choir
+Yodeling
+Chant
+Mantra
+Male singing
+Female singing
+Child singing
+Synthetic singing
+Rapping
+Humming
+Groan
+Grunt
+Whistling
+Breathing
+Wheeze
+Snoring
+Gasp
+Pant
+Snort
+Cough
+Throat clearing
+Sneeze
+Sniff
+Run
+Shuffle
+Walk, footsteps
+Chewing, mastication
+Biting
+Gargling
+Stomach rumble
+Burping, eructation
+Hiccup
+Fart
+Hands
+Finger snapping
+Clapping
+Heart sounds, heartbeat
+Heart murmur
+Cheering
+Applause
+Chatter
+Crowd
+Hubbub, speech noise, speech babble
+Children playing
+Animal
+Domestic animals, pets
+Dog
+Bark
+Yip
+Howl
+Bow-wow
+Growling
+Whimper (dog)
+Cat
+Purr
+Meow
+Hiss
+Caterwaul
+Livestock, farm animals, working animals
+Horse
+Clip-clop
+Neigh, whinny
+Cattle, bovinae
+Moo
+Cowbell
+Pig
+Oink
+Goat
+Bleat
+Sheep
+Fowl
+Chicken, rooster
+Cluck
+Crowing, cock-a-doodle-doo
+Turkey
+Gobble
+Duck
+Quack
+Goose
+Honk
+Wild animals
+Roaring cats (lions, tigers)
+Roar
+Bird
+Bird vocalization, bird call, bird song
+Chirp, tweet
+Squawk
+Pigeon, dove
+Coo
+Crow
+Caw
+Owl
+Hoot
+Bird flight, flapping wings
+Canidae, dogs, wolves
+Rodents, rats, mice
+Mouse
+Patter
+Insect
+Cricket
+Mosquito
+Fly, housefly
+Buzz
+Bee, wasp, etc.
+Frog
+Croak
+Snake
+Rattle
+Whale vocalization
+Music
+Musical instrument
+Plucked string instrument
+Guitar
+Electric guitar
+Bass guitar
+Acoustic guitar
+Steel guitar, slide guitar
+Tapping (guitar technique)
+Strum
+Banjo
+Sitar
+Mandolin
+Zither
+Ukulele
+Keyboard (musical)
+Piano
+Electric piano
+Organ
+Electronic organ
+Hammond organ
+Synthesizer
+Sampler
+Harpsichord
+Percussion
+Drum kit
+Drum machine
+Drum
+Snare drum
+Rimshot
+Drum roll
+Bass drum
+Timpani
+Tabla
+Cymbal
+Hi-hat
+Wood block
+Tambourine
+Rattle (instrument)
+Maraca
+Gong
+Tubular bells
+Mallet percussion
+Marimba, xylophone
+Glockenspiel
+Vibraphone
+Steelpan
+Orchestra
+Brass instrument
+French horn
+Trumpet
+Trombone
+Bowed string instrument
+String section
+Violin, fiddle
+Pizzicato
+Cello
+Double bass
+Wind instrument, woodwind instrument
+Flute
+Saxophone
+Clarinet
+Harp
+Bell
+Church bell
+Jingle bell
+Bicycle bell
+Tuning fork
+Chime
+Wind chime
+Change ringing (campanology)
+Harmonica
+Accordion
+Bagpipes
+Didgeridoo
+Shofar
+Theremin
+Singing bowl
+Scratching (performance technique)
+Pop music
+Hip hop music
+Beatboxing
+Rock music
+Heavy metal
+Punk rock
+Grunge
+Progressive rock
+Rock and roll
+Psychedelic rock
+Rhythm and blues
+Soul music
+Reggae
+Country
+Swing music
+Bluegrass
+Funk
+Folk music
+Middle Eastern music
+Jazz
+Disco
+Classical music
+Opera
+Electronic music
+House music
+Techno
+Dubstep
+Drum and bass
+Electronica
+Electronic dance music
+Ambient music
+Trance music
+Music of Latin America
+Salsa music
+Flamenco
+Blues
+Music for children
+New-age music
+Vocal music
+A capella
+Music of Africa
+Afrobeat
+Christian music
+Gospel music
+Music of Asia
+Carnatic music
+Music of Bollywood
+Ska
+Traditional music
+Independent music
+Song
+Background music
+Theme music
+Jingle (music)
+Soundtrack music
+Lullaby
+Video game music
+Christmas music
+Dance music
+Wedding music
+Happy music
+Funny music
+Sad music
+Tender music
+Exciting music
+Angry music
+Scary music
+Wind
+Rustling leaves
+Wind noise (microphone)
+Thunderstorm
+Thunder
+Water
+Rain
+Raindrop
+Rain on surface
+Stream
+Waterfall
+Ocean
+Waves, surf
+Steam
+Gurgling
+Fire
+Crackle
+Vehicle
+Boat, Water vehicle
+Sailboat, sailing ship
+Rowboat, canoe, kayak
+Motorboat, speedboat
+Ship
+Motor vehicle (road)
+Car
+Vehicle horn, car horn, honking
+Toot
+Car alarm
+Power windows, electric windows
+Skidding
+Tire squeal
+Car passing by
+Race car, auto racing
+Truck
+Air brake
+Air horn, truck horn
+Reversing beeps
+Ice cream truck, ice cream van
+Bus
+Emergency vehicle
+Police car (siren)
+Ambulance (siren)
+Fire engine, fire truck (siren)
+Motorcycle
+Traffic noise, roadway noise
+Rail transport
+Train
+Train whistle
+Train horn
+Railroad car, train wagon
+Train wheels squealing
+Subway, metro, underground
+Aircraft
+Aircraft engine
+Jet engine
+Propeller, airscrew
+Helicopter
+Fixed-wing aircraft, airplane
+Bicycle
+Skateboard
+Engine
+Light engine (high frequency)
+Dental drill, dentist's drill
+Lawn mower
+Chainsaw
+Medium engine (mid frequency)
+Heavy engine (low frequency)
+Engine knocking
+Engine starting
+Idling
+Accelerating, revving, vroom
+Door
+Doorbell
+Ding-dong
+Sliding door
+Slam
+Knock
+Tap
+Squeak
+Cupboard open or close
+Drawer open or close
+Dishes, pots, and pans
+Cutlery, silverware
+Chopping (food)
+Frying (food)
+Microwave oven
+Blender
+Water tap, faucet
+Sink (filling or washing)
+Bathtub (filling or washing)
+Hair dryer
+Toilet flush
+Toothbrush
+Electric toothbrush
+Vacuum cleaner
+Zipper (clothing)
+Keys jangling
+Coin (dropping)
+Scissors
+Electric shaver, electric razor
+Shuffling cards
+Typing
+Typewriter
+Computer keyboard
+Writing
+Alarm
+Telephone
+Telephone bell ringing
+Ringtone
+Telephone dialing, DTMF
+Dial tone
+Busy signal
+Alarm clock
+Siren
+Civil defense siren
+Buzzer
+Smoke detector, smoke alarm
+Fire alarm
+Foghorn
+Whistle
+Steam whistle
+Mechanisms
+Ratchet, pawl
+Clock
+Tick
+Tick-tock
+Gears
+Pulleys
+Sewing machine
+Mechanical fan
+Air conditioning
+Cash register
+Printer
+Camera
+Single-lens reflex camera
+Tools
+Hammer
+Jackhammer
+Sawing
+Filing (rasp)
+Sanding
+Power tool
+Drill
+Explosion
+Gunshot, gunfire
+Machine gun
+Fusillade
+Artillery fire
+Cap gun
+Fireworks
+Firecracker
+Burst, pop
+Eruption
+Boom
+Wood
+Chop
+Splinter
+Crack
+Glass
+Chink, clink
+Shatter
+Liquid
+Splash, splatter
+Slosh
+Squish
+Drip
+Pour
+Trickle, dribble
+Gush
+Fill (with liquid)
+Spray
+Pump (liquid)
+Stir
+Boiling
+Sonar
+Arrow
+Whoosh, swoosh, swish
+Thump, thud
+Thunk
+Electronic tuner
+Effects unit
+Chorus effect
+Basketball bounce
+Bang
+Slap, smack
+Whack, thwack
+Smash, crash
+Breaking
+Bouncing
+Whip
+Flap
+Scratch
+Scrape
+Rub
+Roll
+Crushing
+Crumpling, crinkling
+Tearing
+Beep, bleep
+Ping
+Ding
+Clang
+Squeal
+Creak
+Rustle
+Whir
+Clatter
+Sizzle
+Clicking
+Clickety-clack
+Rumble
+Plop
+Jingle, tinkle
+Hum
+Zing
+Boing
+Crunch
+Silence
+Sine wave
+Harmonic
+Chirp tone
+Sound effect
+Pulse
+Inside, small room
+Inside, large room or hall
+Inside, public space
+Outside, urban or manmade
+Outside, rural or natural
+Reverberation
+Echo
+Noise
+Environmental noise
+Static
+Mains hum
+Distortion
+Sidetone
+Cacophony
+White noise
+Pink noise
+Throbbing
+Vibration
+Television
+Radio
+Field recording

build_panns.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""PANNs CNN14 (AudioSet 527-tag) -> LiteRT CompiledModel GPU.
+
+Deployment = host-side log-mel + a single GPU graph for the CNN body:
+
+    waveform[320000] --[Kotlin log-mel]--> logmel[1,1,1001,64] --[GPU CNN14]--> probs[527]
+
+Why the log-mel is host-side and not in the graph: PANNs builds the spectrogram with torchlibrosa's
+STFT (a DFT-as-Conv1d, so there is NO FFT op and the full raw-audio graph IS op-clean — only the
+center reflect-pad emits a single GATHER_ND, removable with pad_mode='constant'). BUT the converted
+spectral front-end is numerically wrong (fp32 corr 0.19) and the power spectrum |STFT|^2 (~1e6)
+overflows fp16 on Mali -> NaN. The CNN body alone (logmel -> tags) converts at corr 1.000000 in both
+fp32 and fp16, so we keep it on the GPU and compute the log-mel on the CPU (Whisper/Kokoro pattern),
+matched to torchlibrosa exactly (validated here in numpy at corr 1.0).
+
+Deps: the qiuqiangkong model defs (`models.py` as panns_models.py + `pytorch_utils.py`) from
+github.com/qiuqiangkong/audioset_tagging_cnn, and Cnn14_mAP=0.431.pth (Zenodo, CC-BY-4.0):
+    https://zenodo.org/record/3987831/files/Cnn14_mAP%3D0.431.pth
+Run: ~/clipconv/bin/python build_panns.py
+"""
+import _stub_propack  # noqa: F401  (narrow scipy _propack shim; keeps librosa real)
+import sys, os, csv, collections, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
+sys.path.insert(0, ".")
+from panns_models import Cnn14
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+SR, NFFT, HOP, NMEL = 32000, 1024, 320, 64
+PAD = NFFT // 2
+CLIP_SAMPLES = 320000          # 10 s @ 32 kHz (PANNs canonical eval window)
+CFG = dict(sample_rate=SR, window_size=NFFT, hop_size=HOP, mel_bins=NMEL, fmin=50, fmax=14000, classes_num=527)
+CKPT = os.path.join(HERE, "Cnn14_mAP=0.431.pth")
+SAMPLE_WAV = os.path.expanduser("~/Downloads/meeting/wav2vec2-work/sample_speech.wav")
+LABELS_CSV = os.path.join(HERE, "class_labels_indices.csv")
+BANNED = {"GATHER", "GATHER_ND", "TOPK_V2", "GELU", "ERF", "WHERE", "SELECT", "SELECT_V2",
+          "BROADCAST_TO", "POW", "TRANSPOSE_CONV", "CAST", "EMBEDDING_LOOKUP",
+          "RFFT2D", "FFT", "STFT", "COMPLEX", "RFFT", "IRFFT", "CUMSUM"}
+
+
+class LogmelCNN(nn.Module):
+    """The GPU graph: logmel[B,1,T,64] -> clipwise_output[B,527]. Mirrors Cnn14.forward after logmel."""
+    def __init__(s, m): super().__init__(); s.m = m
+    def forward(s, x):
+        m = s.m
+        x = x.transpose(1, 3); x = m.bn0(x); x = x.transpose(1, 3)
+        x = m.conv_block1(x, pool_size=(2, 2), pool_type='avg')
+        x = m.conv_block2(x, pool_size=(2, 2), pool_type='avg')
+        x = m.conv_block3(x, pool_size=(2, 2), pool_type='avg')
+        x = m.conv_block4(x, pool_size=(2, 2), pool_type='avg')
+        x = m.conv_block5(x, pool_size=(2, 2), pool_type='avg')
+        x = m.conv_block6(x, pool_size=(1, 1), pool_type='avg')
+        x = torch.mean(x, dim=3)
+        (x1, _) = torch.max(x, dim=2); x2 = torch.mean(x, dim=2)
+        x = x1 + x2
+        x = F.relu_(m.fc1(x))
+        return torch.sigmoid(m.fc_audioset(x))
+
+
+def numpy_logmel(wav, melW):
+    """What the Kotlin front-end computes: reflect-pad center, periodic Hann, FFT, power, melW, 10log10."""
+    padded = np.pad(wav, (PAD, PAD), mode="reflect")
+    n = 1 + (len(padded) - NFFT) // HOP
+    win = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(NFFT) / NFFT)
+    power = np.empty((n, NFFT // 2 + 1), np.float64)
+    for t in range(n):
+        s = np.fft.rfft(padded[t * HOP: t * HOP + NFFT] * win, n=NFFT)
+        power[t] = s.real ** 2 + s.imag ** 2
+    return (10.0 * np.log10(np.maximum(power @ melW, 1e-10))).astype(np.float32)
+
+
+def opcheck(path, label):
+    from ai_edge_litert.interpreter import Interpreter
+    it = Interpreter(model_path=path); it.allocate_tensors()
+    ops = collections.Counter(d.get("op_name", "?") for d in it._get_ops_details())
+    bad = {k: v for k, v in ops.items() if k.upper() in BANNED}
+    over = sum(1 for d in it.get_tensor_details() if len(d.get("shape", [])) > 4)
+    print(f"[{label}] banned:{bad or 'NONE'} >4D:{over} size {os.path.getsize(path)/1e6:.1f}MB "
+          f"VERDICT {'GPU-CLEAN' if not bad and not over else bad}")
+    return it
+
+
+def tfl(it, x):
+    d = it.get_input_details()[0]; it.set_tensor(d["index"], x.astype(d["dtype"])); it.invoke()
+    return it.get_tensor(it.get_output_details()[0]["index"])
+
+
+def to_fp16(fp32, fp16):
+    from ai_edge_quantizer import quantizer, recipe_manager
+    from ai_edge_quantizer.recipe import AlgorithmName, qtyping
+    rm = recipe_manager.RecipeManager()
+    rm.add_quantization_config(regex=".*", operation_name=qtyping.TFLOperationName.ALL_SUPPORTED,
+        op_config=qtyping.OpQuantizationConfig(
+            weight_tensor_config=qtyping.TensorQuantizationConfig(num_bits=16, dtype=qtyping.TensorDataType.FLOAT),
+            compute_precision=qtyping.ComputePrecision.FLOAT), algorithm_key=AlgorithmName.FLOAT_CASTING)
+    if os.path.exists(fp16): os.remove(fp16)
+    qt = quantizer.Quantizer(float_model=fp32); qt.load_quantization_recipe(rm.get_quantization_recipe())
+    qt.quantize().export_model(fp16); return fp16
+
+
+def main():
+    labels = [r[2] for r in list(csv.reader(open(LABELS_CSV)))[1:]]
+    m = Cnn14(**CFG).eval()
+    m.load_state_dict(torch.load(CKPT, map_location="cpu")["model"])
+    melW = m.logmel_extractor.melW.detach().numpy()      # [513,64]
+
+    import librosa
+    wav, _ = librosa.load(SAMPLE_WAV, sr=SR, mono=True)
+    wav = np.pad(wav, (0, max(0, CLIP_SAMPLES - len(wav))))[:CLIP_SAMPLES].astype(np.float32)
+
+    with torch.no_grad():
+        logmel = m.logmel_extractor(m.spectrogram_extractor(torch.from_numpy(wav)[None]))  # [1,1,T,64]
+        y = LogmelCNN(m).eval()(logmel).numpy().ravel()
+
+    # validate the host-side (Kotlin) log-mel against torch
+    lm_np = numpy_logmel(wav, melW)
+    lm_t = logmel.numpy().reshape(-1, NMEL)
+    print(f"host log-mel vs torch: corr {np.corrcoef(lm_t.ravel(), lm_np.ravel())[0,1]:.6f} "
+          f"max|d| {np.abs(lm_t-lm_np).max():.4f}")
+
+    # convert the CNN body
+    import litert_torch
+    fp32 = os.path.join(HERE, "cnn14_audioset.tflite")
+    litert_torch.convert(LogmelCNN(m).eval(), (logmel,)).export(fp32)
+    it32 = opcheck(fp32, "fp32"); o32 = tfl(it32, logmel.numpy()).ravel()
+    print(f"[fp32] tflite-vs-torch corr {np.corrcoef(o32, y)[0,1]:.6f}")
+    fp16 = to_fp16(fp32, os.path.join(HERE, "cnn14_audioset_fp16.tflite"))
+    it16 = opcheck(fp16, "fp16"); o16 = tfl(it16, logmel.numpy()).ravel()
+    print(f"[fp16] tflite-vs-torch corr {np.corrcoef(o16, y)[0,1]:.6f}")
+
+    # assets + fixtures
+    melW.T.astype(np.float32).tofile(os.path.join(HERE, "mel_basis.bin"))   # [64,513] mel-major
+    wav.tofile(os.path.join(HERE, "panns_input.bin"))
+    np.save(os.path.join(HERE, "panns_ref.npy"), o16)
+    top = np.argsort(o16)[::-1][:8]
+    print("top tags:", [(labels[i], round(float(o16[i]), 3)) for i in top])
+    print("wrote cnn14_audioset_fp16.tflite + mel_basis.bin + fixtures")
+
+
+if __name__ == "__main__":
+    main()

cnn14_audioset_fp16.tflite

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26e52a285ff324778c7b90642e9733b47a46071622ecba93d42ca14ec19a467f
+size 161533712

mel_basis.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50edbf75f0ffdb5bf295a54cb3b9c4ad27b841be1119255f0f785552b99f13bd
+size 131328