You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Okay, so here's an overview of what I tested, including the corresponding result:
CPU & Vulkan backends/images -> both functional ✅
Frontend modules: audio_decode, mp3_encode and websocket -> all functional ✅
Open WebUI STT (transcription) & TTS via http/ws inside an internal docker network -> all functional ✅
Docker healthcheck via the /health endpoint -> fails because curl is missing, the endpoint is accessible from another container though and returns the expected status
Not tested: HTTPS module (because isolated/internal only)
Some annotations and thoughts:
Open WebUI slim images do not include the necessary transcoding components, therefore transcoding to MP3 is not possible before the file is sent to audio.cpp, resulting in a 400 bad request (because Open WebUI is sending webm). Switching to the default (full) image solves this. This is not an issue with audio.cpp, it's just meant for information as I ran into that issue.
You need to mount //.cache to a volume, if you want the Vulkan (Mesa) shader cache to be persistent. Additionally, if you run the container as non root (unprivileged user) and you do not mount //.cache to a volume, you get a warning that the user does not have the permissions to access //.cache and that it is/gets deactivated. The user running the container (obviously) must have permissions to access the mounted volume. This could be included in the docs for the Vulkan image.
I don't know if this already exists, but it would be great to have an optional ttl (time-to-live) variable. Globally or even for each model. Currently, the models remain permanently loaded in RAM/VRAM, which is good for fast responses, but not so much for efficient resource management (on a server). They can occupy several GB that could be used by or are needed for other tasks. It would be beneficial to unload them after a certain idle period in order to free up resources.
A somewhat more sophisticated access management would also be nice 😅, that is a more specific cors_origins and preferably API key authentication.
These are just some thoughts and ideas for the future, I hope it's not overwhelming.
@Tech-no-1 Thanks for testing and for the feedback! For TTL, there is a server config idle_unload_ms. This unloads resident models after X secs without model load/run activity across the server. Busy models are skipped, and subsequent requests reload models automatically. The default is 0 (disabled).
Let me look into the other issues you mentioned. I may defer access management for now, as it falls outside the intended scope of a thin inference server and adds maintenance overhead. If you'd like, you're welcome to submit a PR adding it as an optional frontend module tailored to your needs.
Activity
0xShug0 commentedon Sep 24, 2026
@easyjoh Thanks for your kind words! And yes, a separate Docker image/release is the plan (#541 (comment)). I just need some time to figure this out.
easyjoh commentedon Sep 24, 2026
Sounds good. Thank you.
0xShug0 commentedon Sep 28, 2026
@easyjoh Let me know if this works https://github.com/0xShug0/audio.cpp-server-frontends/pkgs/container/audio.cpp-server-frontends
Tech-no-1 commentedon Sep 29, 2026
Hi, could we please get a Vulkan/CPU version as well? Not everyone has got an overpriced Nvidia GPU/system 😅. I can offer testing it again.
0xShug0 commentedon Sep 29, 2026
Sure!
0xShug0 commentedon Sep 30, 2026
Please check the latest release of the frontend repo.
Tech-no-1 commentedon Sep 30, 2026
Okay, so here's an overview of what I tested, including the corresponding result:
Some annotations and thoughts:
//.cacheto a volume, if you want the Vulkan (Mesa) shader cache to be persistent. Additionally, if you run the container as non root (unprivileged user) and you do not mount//.cacheto a volume, you get a warning that the user does not have the permissions to access//.cacheand that it is/gets deactivated. The user running the container (obviously) must have permissions to access the mounted volume. This could be included in the docs for the Vulkan image.These are just some thoughts and ideas for the future, I hope it's not overwhelming.
0xShug0 commentedon Oct 1, 2026
@Tech-no-1 Thanks for testing and for the feedback! For TTL, there is a server config
idle_unload_ms. This unloads resident models after X secs without model load/run activity across the server. Busy models are skipped, and subsequent requests reload models automatically. The default is 0 (disabled).Let me look into the other issues you mentioned. I may defer access management for now, as it falls outside the intended scope of a thin inference server and adds maintenance overhead. If you'd like, you're welcome to submit a PR adding it as an optional frontend module tailored to your needs.
0xShug0 commentedon Oct 1, 2026
Doc updated!