tckm.yan/aha
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update doc

Browse Source
This commit is contained in:
jhqxxx
2026-05-30 13:19:58 +08:00
parent 791f3ea7e3
commit 69af1ceba5
4 changed files with 472 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api.md
+1 -2
View File
@@ -356,8 +356,7 @@ curl http://127.0.0.1:10100/audio/speech \
{
"role": "user",
"content": [
{"type": "text", "text": "Hello, this is AHA speaking."},
{"type": "audio", "audio_url": {"url": "https://package-release.coderbox.cn/aiway/test/other/%E5%93%AA%E5%90%92.wav"}}
{"type": "text", "text": "Hello, this is AHA speaking."}
]
}
]
docs/api.zh-CN.md
+1 -2
View File
@@ -359,8 +359,7 @@ curl http://127.0.0.1:10100/audio/speech \
{
"role": "user",
"content": [
{"type": "text", "text": "你好，这是 AHA 在说话。"},
{"type": "audio", "audio_url": {"url": "https://package-release.coderbox.cn/aiway/test/other/%E5%93%AA%E5%90%92.wav"}}
{"type": "text", "text": "你好，这是 AHA 在说话。"}
]
}
]
docs/model-card.md
+237
View File
@@ -75,3 +75,240 @@ Please output the information in the image in the following JSON format:
| Parsing | 1. Identify the formula in the image and represent it using LaTeX format. <br> 2.Parse the table in the image into HTML. <br> 3. Parse the chart in the image; use Mermaid format for flowcharts and Markdown for other charts. <br> 4.Extract all information from the main body of the document image and represent it in markdown format, ignoring headers and footers. Tables should be expressed in HTML format, formulas in the document should be represented using LaTeX format, and the parsing should be organized according to the reading order. | 1. 识别图片中的公式，用 LaTeX 格式表示。 <br> 2. 把图中的表格解析为 HTML。 <br> 3. 解析图中的图表，对于流程图使用 Mermaid 格式表示，其他图表使用 Markdown 格式表示。 <br> 4. 提取文档图片中正文的所有信息用 markdown 格式表示，其中页眉、页脚部分忽略，表格用 html 格式表达，文档中公式用 latex 格式表示，按照阅读顺序组织进行解析。 |
| Information Extraction | 1. Output the value of Key. <br> 2. Extract the content of the fields: ['key1','key2', ...] from the image and return it in JSON format. <br> 3. Extract the subtitles from the image. | 1. 输出 Key 的值。 <br> 2. 提取图片中的: ['key1','key2', ...] 的字段内容，并按照 JSON 格式返回。 <br> 3. 提取图片中的字幕。 |
| Translation | First extract the text, then translate the text content into English. If it is a document, ignore the header and footer. Formulas should be represented in LaTeX format, and tables should be represented in HTML format. | 先提取文字，再将文字内容翻译为英文。若是文档，则其中页眉、页脚忽略。公式用latex格式表示，表格用html格式表示。 |
Here is the continuation for the English [model-card.md](file:///home/jhq/rust_code/aha/docs/model-card.md) file, translating and adapting the content from the Chinese version regarding VoxCPM models.
# TTS
## VoxCPM (0.5B / 1.5)
#### Mode Description
* **Zero-shot TTS (Default)**: Generates speech directly without reference audio.
* **Voice Cloning**: Requires reference audio (`audio_url`) and its corresponding transcript (`prompt_text`).
* **Note**: For versions 0.5B/1.5, if `audio_url` is provided, `prompt_text` **must** be provided in `metadata`. Conversely, if there is no `audio_url`, `prompt_text` should not be provided.
#### Metadata Parameters
| Parameter | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `prompt_text` | String | Conditional | **Required for Cloning**. The transcript of the reference audio. Used to extract speaker characteristics. |
| `min_len` | Integer | No | Minimum generation length (tokens/steps), default `2`. |
| `max_len` | Integer | No | Maximum generation length (tokens/steps), default `4096`. |
| `inference_timesteps` | Integer | No | Number of inference steps. Affects quality and speed. Default `10`. Higher values yield better quality but slower speed. |
| `cfg_value` | Float | No | Classifier-Free Guidance value. Controls adherence to the prompt. Default `2.0`. |
| `retry_badcase_ratio_threshold` | Float | No | Threshold for retrying bad cases. Default `6.0`. |
#### Example 1: Zero-shot TTS (No Reference)
```json
{
"model": "OpenBMB/VoxCPM-0.5B",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Hello, this is a test speech."
}
]
}
]
}
```
#### Example 2: Voice Cloning (With Reference)
```json
{
"model": "OpenBMB/VoxCPM-0.5B",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/reference.wav"
}
},
{
"type": "text",
"text": "This is the target text I want to generate."
}
]
}
],
"metadata": {
"prompt_text": "This is the text content corresponding to the reference audio."
}
}
```
#### Response Example
The model returns a JSON object containing Base64-encoded audio data. The audio format is WAV.
```json
{
"id": "chatcmpl-uuid...",
"object": "chat.completion",
"created": 1710000000,
"model": "OpenBMB/VoxCPM-0.5B",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": [
{
"type": "audio",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAEAfAAABAAgAZGF0YQAAAAA..."
}
}
]
},
"finish_reason": "stop"
}
],
"usage": null
}
```
## VoxCPM2
VoxCPM2 supports three advanced voice generation modes. Switch between modes by combining content types in `messages` and parameters in `metadata`.
### 1. Voice Design
Generate a completely new voice from natural language descriptions alone (gender, age, tone, emotion, pace, etc.). **No reference audio required.**
* **Input**: Text message only.
* **Metadata**: Optional `control_instruction` for finer control (e.g., emotion, pace).
#### Example:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Welcome to the future world."
}
]
}
],
"metadata": {
"control_instruction": "Young female, energetic"
}
}
```
### 2. Controllable Cloning
Clone a voice from a short audio clip, with optional style guidance to steer emotion, pace, and expression while preserving timbre.
* **Input**: Reference audio (`audio_url`) + Target text.
* **Metadata**: Optional `control_instruction` to adjust style (emotion, pace, etc.). **`prompt_text` is NOT required.**
#### Example:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/short_clip.wav"
}
},
{
"type": "text",
"text": "This news broadcast needs to be more serious and calm."
}
]
}
],
"metadata": {
"control_instruction": "serious, slow pace"
}
}
```
### 3. Ultimate Cloning
Provide reference audio and its verbatim transcript for audio-continuation cloning or high-fidelity cloning. Every vocal nuance is faithfully reproduced.
* **Input**: Reference audio (`audio_url`) + Target text.
* **Metadata**: **Must** provide `prompt_text` (accurate transcript of the reference audio).
#### Example:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/reference.wav"
}
},
{
"type": "text",
"text": "This is the new content to be spoken next."
}
]
}
],
"metadata": {
"prompt_text": "This is the actual text spoken in the reference audio."
}
}
```
#### General Metadata Parameters (Applicable to all VoxCPM2 modes)
| Parameter | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `prompt_text` | String | Ultimate Cloning Only | Verbatim transcript of the reference audio. Required only for Ultimate Cloning mode. |
| `control_instruction` | String | No | Natural language instruction to control style for Voice Design or Controllable Cloning (e.g., "angry", "whispering", "fast"). |
| `min_len` | Integer | No | Minimum generation length, default `2`. |
| `max_len` | Integer | No | Maximum generation length, default `4096`. |
| `inference_timesteps` | Integer | No | Inference steps, default `10`. |
| `cfg_value` | Float | No | CFG value, default `2.0`. |
| `retry_badcase_ratio_threshold` | Float | No | Bad case retry threshold, default `6.0`. |
#### Response Example
Same as VoxCPM 0.5B/1.5, returns a JSON object containing Base64-encoded WAV audio.
```json
{
"id": "chatcmpl-uuid...",
"object": "chat.completion",
"created": 1710000000,
"model": "OpenBMB/VoxCPM2",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": [
{
"type": "audio",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAEAfAAABAAgAZGF0YQAAAAA..."
}
}
]
},
"finish_reason": "stop"
}
],
"usage": null
}
```
docs/model-card.zh-CN.md
+233 -1
View File
@@ -75,5 +75,237 @@ extract structured information from documents. Prompts must follow a strict JSON
| 信息提取 | 1. Output the value of Key. <br> 2. Extract the content of the fields: ['key1','key2', ...] from the image and return it in JSON format. <br> 3. Extract the subtitles from the image. | 1. 输出 Key 的值。 <br> 2. 提取图片中的: ['key1','key2', ...] 的字段内容，并按照 JSON 格式返回。 <br> 3. 提取图片中的字幕。 |
| 翻译 | First extract the text, then translate the text content into English. If it is a document, ignore the header and footer. Formulas should be represented in LaTeX format, and tables should be represented in HTML format. | 先提取文字，再将文字内容翻译为英文。若是文档，则其中页眉、页脚忽略。公式用latex格式表示，表格用html格式表示。 |
# TTS
## VoxCPM
## VoxCPM (0.5B / 1.5)
### 模式说明
* **Zero-shot TTS (默认模式)**: 无需参考音频，直接生成语音。
* **Voice Cloning (克隆模式)**: 需要提供参考音频 (`audio_url`) 和对应的参考文本 (`prompt_text`)。
* **注意**: 对于 0.5B/1.5 版本，如果提供了 `audio_url`，则**必须**在 `metadata` 中提供 `prompt_text`；反之，如果没有 `audio_url`，则不应提供 `prompt_text`。
### Metadata 参数
| 参数名 | 类型 | 必填 | 说明 |
| :--- | :--- | :--- | :--- |
| `prompt_text` | String | 条件必填 | **克隆模式必填**。参考音频对应的转录文本。用于提取说话人特征。 |
| `min_len` | Integer | 否 | 最小生成长度 (tokens/steps)，默认 `2`。 |
| `max_len` | Integer | 否 | 最大生成长度 (tokens/steps)，默认 `4096`。 |
| `inference_timesteps` | Integer | 否 | 推理步数，影响生成质量和速度，默认 `10`。值越大质量越高，速度越慢。 |
| `cfg_value` | Float | 否 | Classifier-Free Guidance 值，控制对提示词的遵循程度，默认 `2.0`。 |
| `retry_badcase_ratio_threshold` | Float | 否 | 坏案例重试阈值，默认 `6.0`。 |
#### 示例 1: Zero-shot TTS (无参考)
```json
{
"model": "OpenBMB/VoxCPM-0.5B",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "你好，这是一个测试语音。"
}
]
}
]
}
```
#### 示例 2: Voice Cloning (有参考)
```json
{
"model": "OpenBMB/VoxCPM-0.5B",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/reference.wav"
}
},
{
"type": "text",
"text": "这是我要生成的目标文本。"
}
]
}
],
"metadata": {
"prompt_text": "这是参考音频对应的文字内容。"
}
}
```
#### 返回结果示例
模型将返回一个包含 Base64 编码音频数据的 JSON 对象。音频格式为 WAV。
```json
{
"id": "chatcmpl-uuid...",
"object": "chat.completion",
"created": 1710000000,
"model": "OpenBMB/VoxCPM-0.5B",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": [
{
"type": "audio",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAEAfAAABAAgAZGF0YQAAAAA..."
}
}
]
},
"finish_reason": "stop"
}
],
"usage": null
}
```
## VoxCPM2
VoxCPM2 支持三种高级语音生成模式。通过组合 `messages` 中的内容类型和 `metadata` 参数来切换模式。
### 1. Voice Design (声音设计)
仅通过自然语言描述生成全新声音（性别、年龄、语气、情感、语速等），**不需要**参考音频。
* **输入**: 纯文本消息。
* **Metadata**: 可选 `control_instruction` 用于更精细的控制（如情绪、语速）。
#### 示例:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "欢迎来到未来世界。"
}
]
}
],
"metadata": {
"control_instruction": "年轻女性，活力"
}
}
```
### 2. Controllable Cloning (可控克隆)
从短音频片段中克隆声音，并可选地通过风格指导来控制情感、语速和表达，同时保留音色。
* **输入**: 参考音频 (`audio_url`) + 目标文本。
* **Metadata**: 可选 `control_instruction` 用于调整风格（情绪、语速等）。**不需要** `prompt_text`。
#### 示例:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/short_clip.wav"
}
},
{
"type": "text",
"text": "这段新闻播报需要更加严肃和沉稳。"
}
]
}
],
"metadata": {
"control_instruction": "serious, slow pace"
}
}
```
### 3. Ultimate Cloning (极致克隆)
提供参考音频及其逐字稿，进行音频续写或高保真克隆。每个声音细节都被忠实还原。
* **输入**: 参考音频 (`audio_url`) + 目标文本。
* **Metadata**: **必须**提供 `prompt_text` (参考音频的准确转录)。
#### 示例:
```json
{
"model": "OpenBMB/VoxCPM2",
"messages": [
{
"role": "user",
"content": [
{
"type": "audio",
"audio_url": {
"url": "https://example.com/reference.wav"
}
},
{
"type": "text",
"text": "这是接下来要说的新内容。"
}
]
}
],
"metadata": {
"prompt_text": "这是参考音频中实际说的文字。"
}
}
```
#### 通用 Metadata 参数 (适用于所有 VoxCPM2 模式)
| 参数名 | 类型 | 必填 | 说明 |
| :--- | :--- | :--- | :--- |
| `prompt_text` | String | 仅 Ultimate Cloning | 参考音频的逐字稿。仅在极致克隆模式下需要。 |
| `control_instruction` | String | 否 | 自然语言指令，用于控制 Voice Design 或 Controllable Cloning 的风格（如 "angry", "whispering", "fast"）。 |
| `min_len` | Integer | 否 | 最小生成长度，默认 `2`。 |
| `max_len` | Integer | 否 | 最大生成长度，默认 `4096`。 |
| `inference_timesteps` | Integer | 否 | 推理步数，默认 `10`。 |
| `cfg_value` | Float | 否 | CFG 值，默认 `2.0`。 |
| `retry_badcase_ratio_threshold` | Float | 否 | 坏案例重试阈值，默认 `6.0`。 |
#### 返回结果示例
与 VoxCPM 0.5B/1.5 相同，返回包含 Base64 编码 WAV 音频的 JSON 对象。
```json
{
"id": "chatcmpl-uuid...",
"object": "chat.completion",
"created": 1710000000,
"model": "OpenBMB/VoxCPM2",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": [
{
"type": "audio",
"audio_url": {
"url": "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAEAfAAABAAgAZGF0YQAAAAA..."
}
}
]
},
"finish_reason": "stop"
}
],
"usage": null
}
```