Repository navigation
Add support for Yue2 #499
Description
Activity
ah checked a few days ago and the last update was a year ago. I will take this.
Reacted by lin72hThe demos sounds amazing, esp. Suno V6 is restricting.
Add the GitHub repo link of YuE2 here in case anyone doesn't find it: https://github.com/multimodal-art-projection/YuE
Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.
Update: in dev now.
yue2_demo.mp4
Reacted by William Liao, Yao chi, kooks28, propapanda, lin72h, Shinomakoi and LinaReacted by toawotaq05, Elin and LinaI think Q4_0 is safe. Added Q4_0 demos to the HF repo. Current perf:
Combo Audio dur Wall RTF Peak VRAM BF16 main + F32 VAE 224.96s 60.46s 0.2688 12535 MiB Q8_0 main + F16 VAE 194.84s 38.81s 0.1992 8867 MiB Q4_0 main + F16 VAE 221.12s 44.15s 0.1997 7755 MiB Reacted by kooks28, Lina and napa3umReacted by kooks28, Lina and napa3um@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.
Check this out: heavy metal jingle bells
Jingle_bells.wav@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.
Check this out: heavy metal jingle bells Jingle_bells.wav
no problem ;)
Reacted by 0xShug0 and napa3umDo you also plan to add support for https://huggingface.co/m-a-p/SheetSage2 - their model for creating ABC sheets to support cover functionality?
Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.
Update: in dev now.
yue2_demo.mp4is this on windows? can you upload it?
@tianshuo @prookyon Yue2 on
devalready supports ABC input: you can pass an existing score.abc withabc_fileor inline abc when usingcot=melody / cot=full. ABC export/edit/re-import is more of a UI-side workflow and hasn’t been implemented yet. Happy to support this once the main model impl is stabilized.SheetSage 2 inference has been implemented https://github.com/0xShug0/audio.cpp/blob/dev/src/framework/midi/sheetsage2_runtime.cpp. It will be exposed as a standalone model later.
Reacted by napa3um and lin72h@overallbit Building now! Once finished, you can find them in the “Artifacts” section on this page: https://github.com/0xShug0/audio.cpp/actions/runs/34642627647 (default step is 32. 8 is good enough)
Love this one so far using Windows, but if I use ABC score feature to generate a song, my system drive's remaining space would shrunk to less than 200 MB from 8 gigs left in usual.
Also, can you adapt the Cover feature from YuE 2's original WebUI into this project with SheetSage2 and MERT2?
@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.
Check this out: heavy metal jingle bells Jingle_bells.wav
The same prompt: https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json
BF16 weight + F32 vae
RTF: ~0.5
It's so big I can't upload. Are there any other prompts I can test?
Reacted by lin72h24 remaining items
- added a commit that references this issue
on Sep 22, 2026 @napa3um You should now be able to skip generating the full song and just export the ABC or semantic tokens. Kudos to @engival!
The cmd: https://github.com/0xShug0/audio.cpp/blob/main/docs/models/yue2.md#stopping-earlyReacted by napa3um and lin72hYuE2 on Apple Silicon: M5 Max 128 GB,
--backend metal, main61e062d, YuE2-only build withAUDIOCPP_DEPLOYMENT_BUILD=ON. Works out of the box, no patches.Case main / VAE steps cot audio wall RTF peak mem tonight-awake q8_0 / f16 8 full 216.3 s 115.2 s 0.53 9.8 GiB tonight-awake bf16 / f32 8 full 204.1 s 116.4 s 0.57 11.5 GiB tonight-awake q8_0 / f32 32 full 216.3 s 233.0 s 1.08 9.8 GiB demo1 q8_0 / f16 8 off 40.6 s 13.7 s 0.34 10.5 GiB demo2 q8_0 / f16 8 full 39.8 s 13.5 s 0.34 6.0 GiB Wall is
session.wall_ms(model load excluded).- Flash attention on AR and NAR; the full song runs as one NAR chunk (355 MB graph buffer); CPU time is negligible.
- Reruns are byte-identical.
cot=off,melodyandfullplus externalabc_fileall work. Stem-ASR lyric match was 91–100%, the same range as your CUDA example renders.
Two questions:
-
ar_runtime.cppskipsggml_round_bf16for the prefix K/V on Metal (3abd774, Sep 14), and the vendored ggml gained a MetalROUND_BF16kernel on Sep 19 (breeze: enable bf16 activation rounding on Metal #554). I tried removing the skip (skip_bf16_round = false): it runs on the GPU with no change in CPU or wall time, the semantic stream is byte-identical, and output stays deterministic. The render moves slightly: null vs unpatched is −52 dB on 40 s clips (q8_0 and bf16) and −41 dB on tonight-awake. I don't have the Python reference dumps, so I can't tell whether that's closer to the reference;yue2_nar_parity_probe --backend metalwith and without the skip would settle it. Is the skip still needed, or is it left over from before that kernel? -
You mentioned above that the dev-release CLI/server default is 32 steps, but on main
61e062dit's 8 (types.h:58,session.cpp:187,docs/models/yue2.md), and the sidecar'sode_steps: 32isn't read. Is 8 intended for main?
Reacted by lin72h@nazeshinjite Thanks for testing! For your two questions: (1) That patch may no longer be needed given the recent ggml Metal changes. (2) Yes. 8 is intended for main for a quick smoke run. Once it’s merged into main, the dev release will become stale and won’t be updated unless there's something new to test. But I recent found a better default could be cfg = 1.0 and 32 NAR steps.
I got it running, compared to the original python from the model authors I almost feel like this implementation struggles more with instrumentals and getting the length correct (mostly unwanted vocals, but maybe I just lucky with random seeds and settings with the python). All of this is based off of a low sample size doing covers with sheetsage.
It's almost as if sheetsage2 with the original python captured layered instruments better. (edit: probably not this)
Also, some of my instrumental gens, like the attached were with totally empty lyrics (no tags whatsoever), but the web UI forbids that.
edit: I tired using the same seed, abc file, caption, parameters, etc. as the python. It's hard to be say anything objective about it since it's expected to be different. But to my ear the 2x audiocpp gens I tried are missing a lot of information that the python version successfully reproduced.
instrumental cover generated with original python
audio insturmental python.mp3@incf-ai Did you test Python and audio.cpp with the same weight dtype? Could you please share the cmd, seed, abc file etc you used?
I’d appreciate more details here. In a controlled setting, audio.cpp and the Python reference implementation are parity-safe for YuE2 and SheetSage2, which confirms the correctness of the math and logic. If the issue is caused by precision drift, the only real solution is to force higher precision at the cost of increased VRAM usage.Also, you mentioned a “missing score” issue with SheetSage2 but never followed up with details. I haven’t been able to reproduce it, so I’d appreciate more information on that as well.
If you are using https://huggingface.co/Mothersuperior/YuE2-instrumental-cot-full-loras, the prompt needs to be at least "[instrumental]".
Update: BTW you can check https://github.com/0xShug0/audio.cpp/tree/main/tests/yue2 these are the parity test scripts. One thing we can’t fully control is the randomness in generation, but you can pass noise or semantic tokens generated by the Python implementation into audio.cpp
Let me know if I missed anything or can provide anything else, I'd be happy to answer any questions.
Did you test Python and audio.cpp with the same weight dtype?
yes
Could you please share the cmd, seed, abc file etc you used?
see json in zip file below, and screenshot below
Also, you mentioned a “missing score” issue
The missing score appears to have been an issue on my end with not specifying the correct vulkan device number and it possibly erroring out when loading sheetsage but not the main music model
If you are using
No Lora was used with either the python or audio.cpp
--
Using the following to inspect the safetensor files
python3 -c " import json, struct, sys, collections with open(sys.argv[1], 'rb') as f: n = struct.unpack('<Q', f.read(8))[0] h = json.loads(f.read(n)) h.pop('__metadata__', None) for k, v in h.items(): print(k, v['dtype'], v['shape']) print(collections.Counter(v['dtype'] for v in h.values())) " model.safetensorsaudiocpp vae: FP32
audiocpp model: BF16
python vae: FP32
python model: BF16
python sheetsage2: FP32
python mert2: FP32Python input/output folders - include json describing the exact parameters
cyclemagic-transcription-song-cover.zipinput audio also available upon request
screenshot of audiocpp page. (I tried cfg=1 and cfg=2)
python input: irdd-faexoptim-cyclemagic.mp3
python output: audio.mp3Reacted by 0xShug0I'll see if I can find any issues and fix them in time for tomorrow’s release. Let’s get YuE2 as close to perfect as we can!
@incf-ai I removed the empty-string check and used the Python parameters to generate this WAV (converted to 128 kbps for sharing). The empty check is incorrect and shouldn't be enforced, otherwise vocals can leak through. I will fix this.
Your logs show that Python used PyTorch 2.8.0 with ROCm 6.4 on an AMD GPU, while audio.cpp used Vulkan. This can change both numerical results and the random sampling trajectory, so the same seed does not guarantee identical output across the two backends.
And if you replay the same semantic tokens and noise in C++, the C++ and Python outputs are highly similar (CUDA):
alignment drift: 0 samples throughout Waveform correlation: 0.999177 RMSE: 0.00544 Local cosine similarity (random segments of audio): 0.9911–0.9999Check out the outpus:
They do sound similar and the similarity scores your present are high. Seems good.
I hadn't even thought to control for ROCM vs Vulkan.
I had halfway expected random seeds to not be reproducible due to the complexity and differences between the python and cpp engines.
Reacted by 0xShug0Moved to Discussion. In case you’re interested, this branch has additional VRAM optimizations that let YuE2 fit in just 1.7 GB of memory on an iPhone.
Yue2_ios_demo_20x.MP4
- Repository owner locked and limited conversation to collaborators
on Sep 25, 2026
A new music generation model was released today:
https://map-yue2.github.io/