feat(asr): add optional SenseVoice/FunASR provider

.github/workflows/ci.yml

@@ -37,7 +37,7 @@ jobs:
         run: uv run pyright videocaptioner/cli/
 
       - name: Unit tests
-        run: uv run pytest tests/test_cli tests/test_dubbing -q
+        run: uv run pytest tests/test_cli tests/test_dubbing tests/test_asr/test_sensevoice_asr.py -q
 
       - name: Build package
         run: uv build

.github/workflows/publish-pypi.yml

@@ -38,7 +38,7 @@ jobs:
         run: uv run pyright videocaptioner/cli/
 
       - name: Unit tests (CLI)
-        run: uv run pytest tests/test_cli/ -q
+        run: uv run pytest tests/test_cli/ tests/test_asr/test_sensevoice_asr.py -q
 
   build:
     needs: quality

README.md

@@ -10,8 +10,11 @@
 
 ```bash
 pip install videocaptioner          # 安装 CLI + GUI 桌面版
+pip install 'videocaptioner[sensevoice]'  # 可选：本地 SenseVoice/FunASR 转录
 ```
 
+SenseVoice extra 会安装 FunASR、PyTorch 和 torchaudio。GPU 用户如需特定 CUDA 构建，请先按 [PyTorch 官方说明](https://pytorch.org/get-started/locally/)安装相互匹配的 PyTorch 与 torchaudio。
+
 免费功能（必剪语音识别、必应/谷歌翻译）**无需任何配置，安装即用**。
 
 ## CLI 命令行
@@ -20,6 +23,9 @@ pip install videocaptioner          # 安装 CLI + GUI 桌面版
 # 语音转录（免费，无需 API Key）
 videocaptioner transcribe video.mp4 --asr bijian
 
+# 本地 SenseVoice 转录（支持词级时间戳）
+videocaptioner transcribe video.mp4 --asr sensevoice --language auto
+
 # 字幕翻译（免费必应翻译）
 videocaptioner subtitle input.srt --translator bing --target-language en
 
@@ -49,7 +55,7 @@ videocaptioner config set llm.model gpt-4o-mini
 | 命令 | 说明 |
 |------|------|
 | `gui` | 打开桌面版。也可以直接运行 `videocaptioner-gui` |
-| `transcribe` | 语音转字幕。引擎：`faster-whisper`、`whisper-api`、`bijian`（免费）、`jianying`（免费）、`whisper-cpp` |
+| `transcribe` | 语音转字幕。引擎：`faster-whisper`、`sensevoice`、`whisper-api`、`bijian`（免费）、`jianying`（免费）、`whisper-cpp` |
 | `subtitle` | 字幕优化/翻译。翻译服务：`llm`、`bing`（免费）、`google`（免费） |
 | `dub` | 根据字幕生成配音音轨或配音视频 |
 | `synthesize` | 字幕烧录到视频（软字幕/硬字幕） |

docs/cli.md

@@ -4,8 +4,11 @@
 
 ```bash
 pip install videocaptioner          # CLI + GUI 桌面版
+pip install 'videocaptioner[sensevoice]'  # 可选：本地 SenseVoice/FunASR 转录
 ```
 
+SenseVoice extra 会安装 FunASR、PyTorch 和 torchaudio。GPU 用户如需特定 CUDA 构建，请先按 [PyTorch 官方说明](https://pytorch.org/get-started/locally/)安装相互匹配的 PyTorch 与 torchaudio。
+
 免费功能（转录、必应/谷歌翻译）无需任何配置，安装后直接使用。
 需要桌面版时运行 `videocaptioner-gui`、`videocaptioner gui`，或直接运行无参数的 `videocaptioner`。
 
@@ -17,6 +20,9 @@ pip install videocaptioner          # CLI + GUI 桌面版
 # 语音转字幕（免费）
 videocaptioner transcribe video.mp4 --asr bijian
 
+# 本地 SenseVoice 转录
+videocaptioner transcribe video.mp4 --asr sensevoice --language auto
+
 # 翻译字幕（免费必应翻译）
 videocaptioner subtitle input.srt --translator bing --target-language en
 
@@ -48,12 +54,14 @@ videocaptioner transcribe <文件> [选项]
 
 | 选项 | 说明 |
 |------|------|
-| `--asr` | ASR 引擎：`bijian`(默认,免费) `jianying`(免费) `whisper-api` `whisper-cpp`。bijian/jianying 仅支持中英文，其他语言用 whisper-api 或 whisper-cpp |
+| `--asr` | ASR 引擎：`bijian`(默认,免费) `jianying`(免费) `whisper-api` `whisper-cpp` `sensevoice`。SenseVoice 在本地运行，默认模型支持中文、粤语、英语、日语和韩语 |
 | `--language CODE` | 源语言 ISO 639-1 代码，如 `zh` `en` `ja`，或 `auto`（默认） |
 | `--word-timestamps` | 输出词级时间戳（配合字幕断句使用） |
 | `--whisper-api-key` | Whisper API 密钥（仅 `--asr whisper-api`） |
 | `--whisper-api-base` | Whisper API 地址 |
 | `--whisper-model` | Whisper 模型名（whisper-api 默认 whisper-1，whisper-cpp 默认 large-v2） |
+| `--sensevoice-model` | FunASR 模型名或本地路径（默认 `iic/SenseVoiceSmall`） |
+| `--sensevoice-device` | SenseVoice 推理设备：`auto`(默认) `cuda` `cpu` `mps` |
 | `-o PATH` | 输出文件或目录路径 |
 | `--format` | 输出格式：`srt`(默认) `ass` `txt` `json` |
 

pyproject.toml

@@ -43,6 +43,11 @@ dependencies = [
 
 [project.optional-dependencies]
 gui = []
+sensevoice = [
+    "funasr>=1.3.9,<2",
+    "torch>=1.13",
+    "torchaudio>=0.13",
+]
 
 [project.urls]
 Homepage = "https://github.com/WEIFENG2333/VideoCaptioner"

tests/test_asr/test_platform_utils.py

@@ -0,0 +1,24 @@
+import importlib.util
+
+from videocaptioner.core.entities import TranscribeModelEnum
+from videocaptioner.core.utils import platform_utils
+
+
+def test_sensevoice_is_hidden_when_runtime_is_incomplete(monkeypatch):
+    monkeypatch.setattr(platform_utils, "is_macos", lambda: False)
+    monkeypatch.setattr(
+        importlib.util,
+        "find_spec",
+        lambda name: object() if name in {"funasr", "torch"} else None,
+    )
+
+    assert TranscribeModelEnum.SENSEVOICE not in platform_utils.get_available_transcribe_models()
+    assert platform_utils.is_model_available(TranscribeModelEnum.SENSEVOICE) is False
+
+
+def test_sensevoice_is_available_when_runtime_is_complete(monkeypatch):
+    monkeypatch.setattr(platform_utils, "is_macos", lambda: False)
+    monkeypatch.setattr(importlib.util, "find_spec", lambda _name: object())
+
+    assert TranscribeModelEnum.SENSEVOICE in platform_utils.get_available_transcribe_models()
+    assert platform_utils.is_model_available(TranscribeModelEnum.SENSEVOICE) is True

tests/test_asr/test_sensevoice_asr.py

@@ -0,0 +1,177 @@
+import importlib
+from pathlib import Path
+
+import pytest
+
+from videocaptioner.core.asr.sensevoice import SenseVoiceASR
+from videocaptioner.core.asr.transcribe import _create_asr_instance
+from videocaptioner.core.entities import TranscribeConfig, TranscribeModelEnum
+
+
+class FakeSenseVoiceModel:
+    def __init__(self, result):
+        self.result = result
+        self.generate_kwargs = None
+
+    def generate(self, **kwargs):
+        self.generate_kwargs = kwargs
+        progress_callback = kwargs.get("progress_callback")
+        if progress_callback:
+            progress_callback(1, 2)
+            progress_callback(2, 2)
+        return self.result
+
+
+@pytest.fixture(autouse=True)
+def clear_model_cache():
+    SenseVoiceASR._model_cache.clear()
+    yield
+    SenseVoiceASR._model_cache.clear()
+
+
+def test_preserves_native_word_timestamps(test_audio_path_zh, monkeypatch):
+    model = FakeSenseVoiceModel(
+        [
+            {
+                "text": "<|zh|><|NEUTRAL|><|Speech|><|withitn|>今天天气？",
+                "words": ["今", "天", "天", "气", "？"],
+                "timestamp": [[100, 180], [200, 280], [500, 580], [600, 680], [700, 760]],
+            }
+        ]
+    )
+    monkeypatch.setattr(SenseVoiceASR, "_get_model", lambda self: model)
+    callbacks = []
+
+    result = SenseVoiceASR(
+        str(test_audio_path_zh),
+        model="iic/SenseVoiceSmall",
+        device="cpu",
+        language="zh",
+        need_word_time_stamp=True,
+    ).run(lambda progress, message: callbacks.append((progress, message)))
+
+    assert [(seg.text, seg.start_time, seg.end_time) for seg in result] == [
+        ("今", 100, 180),
+        ("天", 200, 280),
+        ("天", 500, 580),
+        ("气", 600, 680),
+        ("？", 700, 760),
+    ]
+    assert model.generate_kwargs["language"] == "zh"
+    assert model.generate_kwargs["output_timestamp"] is True
+    assert model.generate_kwargs["merge_vad"] is True
+    assert callbacks[-1][0] == 100
+
+
+def test_groups_words_into_readable_sentence_segments(test_audio_path_en, monkeypatch):
+    model = FakeSenseVoiceModel(
+        [
+            {
+                "text": "<|en|><|NEUTRAL|><|Speech|><|withitn|>What's new? Fine.",
+                "words": ["What", "'", "s", "new", "?", "Fine", "."],
+                "timestamp": [
+                    [100, 180],
+                    [180, 220],
+                    [220, 260],
+                    [300, 420],
+                    [420, 480],
+                    [1800, 2000],
+                    [2000, 2060],
+                ],
+            }
+        ]
+    )
+    monkeypatch.setattr(SenseVoiceASR, "_get_model", lambda self: model)
+
+    result = SenseVoiceASR(
+        str(test_audio_path_en),
+        model="iic/SenseVoiceSmall",
+        device="cpu",
+        language="auto",
+    ).run()
+
+    assert [(seg.text, seg.start_time, seg.end_time) for seg in result] == [
+        ("What's new?", 100, 480),
+        ("Fine.", 1800, 2060),
+    ]
+
+
+def test_falls_back_to_clean_text_and_timestamp_bounds(test_audio_path_zh, monkeypatch):
+    model = FakeSenseVoiceModel(
+        [
+            {
+                "text": "<|zh|><|NEUTRAL|><|Speech|><|withitn|>你好。",
+                "timestamp": [[250, 310], [500, 720], [720, 800]],
+            }
+        ]
+    )
+    monkeypatch.setattr(SenseVoiceASR, "_get_model", lambda self: model)
+
+    result = SenseVoiceASR(str(test_audio_path_zh), device="cpu").run()
+
+    assert len(result) == 1
+    assert result.segments[0].text == "你好。"
+    assert result.segments[0].start_time == 250
+    assert result.segments[0].end_time == 800
+
+
+def test_encoded_audio_bytes_are_converted_to_wav(test_audio_path_zh, monkeypatch):
+    class InspectingModel(FakeSenseVoiceModel):
+        def generate(self, **kwargs):
+            input_path = Path(kwargs["input"])
+            assert input_path.suffix == ".wav"
+            assert input_path.exists()
+            assert input_path.read_bytes().startswith(b"RIFF")
+            return super().generate(**kwargs)
+
+    model = InspectingModel(
+        [
+            {
+                "text": "<|zh|><|NEUTRAL|><|Speech|><|withitn|>你好。",
+                "words": ["你", "好", "。"],
+                "timestamp": [[100, 200], [250, 350], [350, 400]],
+            }
+        ]
+    )
+    monkeypatch.setattr(SenseVoiceASR, "_get_model", lambda self: model)
+
+    result = SenseVoiceASR(test_audio_path_zh.read_bytes(), device="cpu").run()
+
+    assert result.segments[0].text == "你好。"
+
+
+def test_missing_optional_dependency_has_install_hint(test_audio_path_zh, monkeypatch):
+    asr = SenseVoiceASR(str(test_audio_path_zh), device="cpu")
+    real_import_module = importlib.import_module
+
+    def import_without_funasr(name, *args, **kwargs):
+        if name == "funasr":
+            raise ModuleNotFoundError("No module named 'funasr'")
+        return real_import_module(name, *args, **kwargs)
+
+    monkeypatch.setattr(importlib, "import_module", import_without_funasr)
+
+    with pytest.raises(RuntimeError, match=r"videocaptioner\[sensevoice\]"):
+        asr._create_model("cpu")
+
+
+def test_transcribe_factory_builds_single_worker_sensevoice(test_audio_path_zh):
+    config = TranscribeConfig(
+        transcribe_model=TranscribeModelEnum.SENSEVOICE,
+        transcribe_language="yue",
+        need_word_time_stamp=True,
+        sensevoice_model="iic/SenseVoiceSmall",
+        sensevoice_device="cpu",
+    )
+
+    chunked = _create_asr_instance(str(test_audio_path_zh), config)
+
+    assert chunked.asr_class is SenseVoiceASR
+    assert chunked.chunk_concurrency == 1
+    assert chunked.asr_kwargs == {
+        "use_cache": True,
+        "need_word_time_stamp": True,
+        "language": "yue",
+        "model": "iic/SenseVoiceSmall",
+        "device": "cpu",
+    }

tests/test_cli/test_config.py

@@ -130,6 +130,26 @@ def test_save_and_load(self, tmp_path):
         assert loaded["subtitle"]["thread_num"] == 8
         assert loaded["subtitle"]["optimize"] is False
 
+    def test_sensevoice_settings_save_and_load(self, tmp_path):
+        config_file = tmp_path / "config.toml"
+
+        save_config_value(
+            "transcribe.sensevoice.model",
+            "iic/SenseVoiceSmall",
+            config_path=config_file,
+        )
+        save_config_value(
+            "transcribe.sensevoice.device",
+            "cpu",
+            config_path=config_file,
+        )
+
+        loaded = load_config_file(config_file)
+        assert loaded["transcribe"]["sensevoice"] == {
+            "model": "iic/SenseVoiceSmall",
+            "device": "cpu",
+        }
+
 
 class TestBuildConfig:
     def test_defaults_only(self):
@@ -162,3 +182,14 @@ def test_env_values_are_typed(self, monkeypatch):
         assert overrides["dubbing"]["tts_workers"] == 3
         assert overrides["dubbing"]["rewrite_too_long"] is True
         assert overrides["dubbing"]["mix_original_audio"] is False
+
+    def test_sensevoice_env_overrides(self, monkeypatch):
+        monkeypatch.setenv("VIDEOCAPTIONER_SENSEVOICE_MODEL", "iic/SenseVoiceSmall")
+        monkeypatch.setenv("VIDEOCAPTIONER_SENSEVOICE_DEVICE", "cpu")
+
+        overrides = load_env_overrides()
+
+        assert overrides["transcribe"]["sensevoice"] == {
+            "model": "iic/SenseVoiceSmall",
+            "device": "cpu",
+        }