diff --git a/.gitignore b/.gitignore index 2a0038a..c628483 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,19 @@ +.gitignore + /target -.idea \ No newline at end of file +.idea +/models + +/ope* +AG* +CLA* + +.claude +README.md.bak +test.mp3 +docs/MODEL_DEVELOPMENT_GUIDE.md +.DS_Store +.gitignore + +bak +.gitignore diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index b859a3e..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,112 +0,0 @@ -# Changelog - -## [Unreleased] - 2025-02-04 - -### Added -- Support for Qwen3-ASR model - -## [Unreleased] - -### Added - -- **CLI `run` Subcommand**: Direct model inference from CLI without HTTP service overhead: - - `aha run` - Run model inference directly - - `-m, --model ` - Specify which model to use - - `-in, --input ` - Input text or file path (model-specific interpretation) - - `-out, --output ` - Output file path (optional, auto-generated if not specified) - - `--weight-path ` - Local model weight path (required) - - -### Changed - -- **CLI Structure**: Refactored CLI to use clap's Subcommand feature while maintaining backward compatibility -- **Backward Compatibility**: Commands without subcommand now default to `cli` subcommand: - - `aha -m qwen3vl-2b` is equivalent to `aha cli -m qwen3vl-2b` - - All existing parameter options and defaults remain unchanged - -### Technical Details - -**Subcommand Parameters:** - -`aha cli`: -- `-a, --address
` - Server address (default: 127.0.0.1) -- `-p, --port ` - Server port (default: 10100) -- `-m, --model ` - Model to use (required) -- `--weight-path ` - Local model weight path (optional) -- `--save-dir ` - Directory to save downloaded model (optional) -- `--download-retries ` - Download retry attempts (default: 3) - -`aha serv`: -- `-a, --address
` - Server address (default: 127.0.0.1) -- `-p, --port ` - Server port (default: 10100) -- `-m, --model ` - Model to use (required) -- `--weight-path ` - Local model weight path (required) - -`aha download`: -- `-m, --model ` - Model to download (required) -- `-s, --save-dir ` - Directory to save downloaded model (optional) -- `--download-retries ` - Download retry attempts (default: 3) - -**Code Changes:** -- Modified `src/main.rs` only -- Extracted common functions: `get_model_id()`, `start_http_server()` -- Reused existing `download_model()` and `init()` functions -- No changes to other modules or dependencies - -## [0.1.8] - 2025-01-20 - -### Added - -- Support for Fun-ASR-Nano-2512 model -- Support for Qwen3-0.6B model - -## [0.1.7] - 2024-XX-XX - -### Added - -- Support for GLM-ASR-Nano-2512 model - -## [0.1.6] - 2024-XX-XX - -### Added - -- Support for RMBG-2.0 model (background removal) - -## [0.1.5] - 2024-XX-XX - -### Added - -- Support for VoxCPM1.5 model - -## [0.1.4] - 2024-XX-XX - -### Added - -- Support for PaddleOCR-VL model - -## [0.1.3] - 2024-XX-XX - -### Added - -- Support for Hunyuan-OCR model - -## [0.1.2] - 2024-XX-XX - -### Added - -- Support for DeepSeek-OCR model - -## [0.1.1] - 2024-XX-XX - -### Added - -- Support for Qwen3VL model family (2B, 4B, 8B, 32B) - -## [0.1.0] - 2024-XX-XX - -### Added - -- Initial release -- Support for Qwen2.5VL models (3B, 7B) -- Support for MiniCPM4-0.5B model -- Support for VoxCPM-0.5B model diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..641ffba --- /dev/null +++ b/README.en.md @@ -0,0 +1,228 @@ +

+ aha logo +

+ +

+ + + GitHub License + + + GitHub Stars + + + GitHub Issues + +

+ +

+ 简体中文 | English +

+ +# aha + +**Lightweight AI Inference Engine — All-in-one Solution for Text, Vision, Speech, and OCR** + +aha is a high-performance, cross-platform AI inference engine built with Rust and the Candle framework. It brings state-of-the-art AI models to your local machine—no API keys, no cloud dependencies, just pure, fast AI running directly on your hardware. + +## Changelog + +### v0.2.0 (2026-02-05) +- Added Qwen3-ASR speech recognition model + +### v0.1.9 (2026-01-31) +- Added CLI `list` subcommand to show supported models +- Added CLI subcommand structure support (`cli`, `serv`, `download`, `run`) +- Fixed Qwen3VL thinking startswith bug +- Fixed `aha run` multiple inputs bug + +### v0.1.8 (2026-01-17) +- Added Qwen3 text model support +- Added Fun-ASR-Nano-2512 speech recognition model +- Fixed ModelScope Fun-ASR-Nano model load error +- Updated audio resampling with rubato + +### v0.1.7 (2026-01-07) +- Added GLM-ASR-Nano-2512 speech recognition model +- Merged Metal (GPU) support for Apple Silicon +- Added dynamic home directory and model download script + +**[View full changelog](docs/changelog.md)** → + +## Quick Start + +### Installation + +```bash +git clone https://github.com/jhqxxx/aha.git +cd aha +cargo build --release +``` + +**Optional Features:** + +```bash +# CUDA (NVIDIA GPU acceleration) +cargo build --release --features cuda + +# Metal (Apple GPU acceleration for macOS) +cargo build --release --features metal + +# Flash Attention (faster inference) +cargo build --release --features flash-attn + +# FFmpeg (multimedia processing) +cargo build --release --features ffmpeg + +# Combine multiple features +cargo build --release --features "cuda,flash-attn" +``` + +### CLI Quick Reference + +```bash + +# List all supported models +aha list + +# Download model only +aha download -m qwen3asr-0.6b + +# Download model and start service +aha -m qwen3asr-0.6b + +# Run inference directly (without starting service) +aha run -m qwen3asr-0.6b -i "audio.wav" + +# Start service only (model already downloaded) +aha serv -m qwen3asr-0.6b -p 10100 + +``` + +### Chat + +```bash +aha serv -m qwen3-0.6b -p 10100 +``` + +Then use the unified (OpenAI-compatible) API: + +```bash +curl http://localhost:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [{"role": "user", "content": "Hello!"}] + } +' +``` + +### Supported Models + +| Category | Models | +|----------|--------| +| **Text** | Qwen3, MiniCPM4 | +| **Vision** | Qwen2.5-VL, Qwen3-VL | +| **OCR** | DeepSeek-OCR, Hunyuan-OCR, PaddleOCR-VL | +| **ASR** | GLM-ASR-Nano, Fun-ASR-Nano, Qwen3-ASR | +| **Audio** | VoxCPM, VoxCPM1.5 | +| **Image** | RMBG-2.0 (background removal) | + +## Documentation + +| Document | Description | +|----------|-------------| +| [Getting Started](docs/getting-started.md) | First steps with aha | +| [Installation](docs/installation.md) | Detailed installation guide | +| [CLI Reference](docs/cli.md) | Command-line interface | +| [API Documentation](docs/api.md) | Library & REST API | +| [Supported Models](docs/supported-models.md) | Available AI models | +| [Concepts](docs/concepts.md) | Architecture & design | +| [Development](docs/development.md) | Contributing guide | +| [Changelog](docs/changelog.md) | Version history | + +## Why aha? +- **🚀 High-Performance Inference** - Powered by Candle framework for efficient tensor computation and model inference +- **🔧 Unified Interface** — One tool for text, vision, speech, and OCR +- **📦 Local-First** — All processing runs locally, no data leaves your machine +- **🎯 Cross-Platform** — Works on Linux, macOS, and Windows +- **⚡ GPU Accelerated** — Optional CUDA support for faster inference +- **🛡️ Memory Safe** — Built with Rust for reliability +- **🧠 Attention Optimization** - Optional Flash Attention support for optimized long sequence processing + +## Development + +### Using aha as a Library +> cargo add aha + +```bash +# VoxCPM example +use aha::models::voxcpm::generate::VoxCPMGenerate; +use aha::utils::audio_utils::save_wav; +use anyhow::Result; + +fn main() -> Result<()> { + let model_path = "xxx/openbmb/VoxCPM-0.5B/"; + + let mut voxcpm_generate = VoxCPMGenerate::init(model_path, None, None)?; + + let generate = voxcpm_generate.generate( + "The sun is shining bright, flowers smile at me, birds say early early early".to_string(), + None, + None, + 2, + 100, + 10, + 2.0, + false, + 6.0, + )?; + + let _ = save_wav(&generate, "voxcpm.wav")?; + Ok(()) +} +``` + +### Extending New Models + +- Create new model file in src/models/ +- Export in src/models/mod.rs +- Add support for CLI model inference in src/exec/ +- Add tests and examples in tests/ + +## Features + +- High-performance inference via Candle framework +- Multi-modal model support (vision, language, speech) +- Clean, easy-to-use API design +- Minimal dependencies, compact binaries +- Flash Attention support for long sequences +- FFmpeg support for multimedia processing + +## License + +Apache-2.0 — See [LICENSE](LICENSE) for details. + +## Acknowledgments + +- [Candle](https://github.com/huggingface/candle) - Excellent Rust ML framework +- All model authors and contributors + +--- + +

+ Built with ❤️ by the aha team +

+ +

+ We're continuously expanding our model support. Contributions are welcome! +

+ +

+ If this project helps you, please consider giving us a ⭐ Star! +

diff --git a/README.md b/README.md index 6a08900..9193e6a 100644 --- a/README.md +++ b/README.md @@ -1,241 +1,167 @@ -# aha -一个基于 Candle 框架的 Rust 模型推理库，提供高效、易用的多模态模型推理能力。 +

+ aha logo +

-## 特性 -* 🚀 高性能推理 - 基于 Candle 框架，提供高效的张量计算和模型推理 -* 🎯 多模型支持 - 集成视觉、语言和语音多模态模型 -* 🔧 易于使用 - 简洁的 API 设计，快速上手 -* 🛡️ 内存安全 - 得益于 Rust 的所有权系统，确保内存安全 -* 📦 轻量级 - 最小化依赖，编译产物小巧 -* ⚡ GPU 加速 - 可选 CUDA 支持 -* 🧠 注意力优化 - 可选 Flash Attention 支持，优化长序列处理 +

+ + + GitHub License + + + GitHub Stars + + + GitHub Issues + +

-## 支持的模型 -### 当前已实现 -* [Qwen2.5VL](https://huggingface.co/collections/Qwen/qwen25-vl) - 阿里通义千问 2.5 多模态大语言模型 - - 模型：[Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) 开源协议：[Qwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE) - - 模型：[Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) -* [MiniCPM4](https://huggingface.co/collections/openbmb/minicpm4) - 面壁智能 MiniCPM 系列语言模型 - - 模型： - - huggingface: [MiniCPM4-0.5B](https://huggingface.co/openbmb/MiniCPM4-0.5B) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - AtomGit: [MiniCPM4-0.5B](https://ai.atomgit.com/OpenBMB/MiniCPM4-0.5B) -* [VoxCPM](https://huggingface.co/collections/openbmb/voxcpm) - 面壁智能语音生成模型 - - 模型： - - huggingface: [VoxCPM-0.5B](https://huggingface.co/openbmb/VoxCPM-0.5B) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - AtomGit: [VoxCPM-0.5B](https://ai.atomgit.com/OpenBMB/VoxCPM-0.5B) -* [Qwen3VL](https://huggingface.co/collections/Qwen/qwen3-vl) - 阿里通义千问 3 多模态大语言模型 - - 模型：[Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - 模型：[Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - 模型：[Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - 模型：[Qwen3-VL-32B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) -* DeepSeek-OCR - 深度求索光学文字识别模型 - - 模型：[DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) 开源协议：[MIT](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md) -* Hunyuan-OCR - 腾讯混元光学文字识别模型 - - 模型： - - huggingface: [HunyuanOCR](https://huggingface.co/tencent/HunyuanOCR) 开源协议：[TENCENT HUNYUAN COMMUNITY LICENSE AGREEMENT](https://huggingface.co/tencent/HunyuanOCR/blob/main/LICENSE) - - AtomGit: [HunyuanOCR](https://ai.atomgit.com/tencent_hunyuan/HunyuanOCR) -* [PaddleOCR-VL](https://huggingface.co/collections/PaddlePaddle/paddleocr-vl) - 百度飞桨光学文字识别模型 - - 模型： - - huggingface: [PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - AtomGit: [PaddleOCR-VL](https://ai.atomgit.com/paddlepaddle/PaddleOCR-VL) -* [VoxCPM1.5](https://huggingface.co/collections/openbmb/voxcpm) - 面壁智能语音生成模型1.5版本 - - 模型： - - huggingface: [VoxCPM1.5](https://huggingface.co/openbmb/VoxCPM1.5) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - AtomGit: [VoxCPM1.5](https://ai.atomgit.com/OpenBMB/VoxCPM1.5) -* [RMBG2.0](https://huggingface.co/collections/briaai/rmbg) - RMBGv2.0由BRIA AI开发，供非商业用途使用。 - - 模型：[RMBG2.0](https://huggingface.co/briaai/RMBG-2.0) 开源协议：[Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/deed.en) -* GLM-ASR-Nano-2512 - 智谱AI语音识别模型 - - 模型： - - huggingface: [GLM-ASR-Nano-2512](https://huggingface.co/zai-org/GLM-ASR-Nano-2512) 开源协议：[MIT](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md) - - AtomGit: [GLM-ASR-Nano-2512](https://ai.atomgit.com/zai-org/GLM-ASR-Nano-2512) -* Fun-ASR-Nano-2512 - 通义百聆语音识别模型 - - 模型:[Fun-ASR-Nano-2512](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) 开源协议未标明 -* [Qwen3](https://huggingface.co/collections/Qwen/qwen3) - 通义千问 Qwen3系列语言模型 - - 模型:[Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) 开源协议: [Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) -* [Qwen3-ASR](https://huggingface.co/collections/Qwen/qwen3-asr) - 通义千问 Qwen3语音识别模型 - - 模型:[Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) - - 模型:[Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) 开源协议：[Apache license 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) +

+ English | 简体中文 +

-## 计划支持 -我们持续扩展支持的模型列表，欢迎贡献！ +# aha -⭐ 如果这个项目对你有帮助，请给我们一个 Star！ +**轻量 AI 推理引擎 —— 文本、视觉、语音与 OCR 一站式解决方案** -## 环境依赖 -* 启用ffmpeg的feature时： - * ubuntu/WSL - ```bash - sudo apt-get update - sudo apt-get install -y clang pkg-config ffmpeg libavutil-dev libavcodec-dev libavformat-dev libavfilter-dev libavdevice-dev libswresample-dev libswscale-dev - ``` - * windows参考： https://github.com/zmwangx/rust-ffmpeg/wiki/Notes-on-building +aha 是一款基于 Rust 和 Candle 框架构建的高性能跨平台 AI 推理引擎。将最先进的 AI 模型带到您的本地机器——无需 API 密钥，无需云依赖，纯粹、快速的 AI 直接在您的硬件上运行。 + +## 更新日志 + +### v0.2.0 (2026-02-05) +- 新增 Qwen3-ASR 语音识别模型 + +### v0.1.9 (2026-01-31) +- 新增 CLI `list` 子命令，显示支持的模型 +- 新增 CLI 子命令结构支持（`cli`、`serv`、`download`、`run`） +- 修复 Qwen3VL thinking startswith bug +- 修复 `aha run` 多输入 bug + +### v0.1.8 (2026-01-17) +- 新增 Qwen3 文本模型支持 +- 新增 Fun-ASR-Nano-2512 语音识别模型 +- 修复 ModelScope Fun-ASR-Nano 模型加载错误 +- 使用 rubato 更新音频重采样 + +### v0.1.7 (2026-01-07) +- 新增 GLM-ASR-Nano-2512 语音识别模型 +- 合并 Metal (GPU) 支持，适用于 Apple Silicon +- 新增动态主目录和模型下载脚本 + +**[查看完整更新日志](docs/changelog.zh-CN.md)** → + +## 快速开始 + +### 安装 -## 功能特性 -项目提供了几个可选的功能特性，您可以根据需要启用它们： -* flash-attn: 启用 Flash Attention 支持以提升模型推理性能： ```bash -cargo build -r --features flash-attn +git clone https://github.com/jhqxxx/aha.git +cd aha +cargo build --release ``` -* cuda: 为 candle 核心组件启用 CUDA 支持，实现 Nvidia GPU 加速计算： -```bash -cargo build -r --features cuda -``` - -* cuda: 为 candle 核心组件启用 Metal 支持，利用 Apple GPU 加速计算（适用于 macOS/iOS 平台）： -```bash -cargo build -r --features metal -``` - -* ffmpeg: 启用 FFmpeg 支持，提供多媒体处理功能： -```bash -cargo build -r --features ffmpeg -``` -* 组合使用功能特性 +**可选特性：** ```bash -# 同时启用 CUDA 和 Flash Attention 以获得最佳性能 -cargo build -r --features "cuda,flash-attn" -``` -## 命令行工具 (CLI) -项目提供命令行工具，支持模型下载、服务启动和直接推理等多种操作。 +# CUDA (NVIDIA GPU 加速) +cargo build --release --features cuda + +# Metal (Apple GPU 加速，适用于 macOS) +cargo build --release --features metal + +# Flash Attention (更快推理) +cargo build --release --features flash-attn + +# FFmpeg (多媒体处理) +cargo build --release --features ffmpeg + +# 组合多个特性 +cargo build --release --features "cuda,flash-attn" +``` + +### CLI 快速参考 -### 快速开始 ```bash + +# 列出所有支持的模型 +aha list + +# 仅下载模型 +aha download -m qwen3asr-0.6b + # 下载模型并启动服务 aha -m qwen3asr-0.6b # 直接运行推理（无需启动服务） -aha run -m qwen3asr-0.6b -i "audio.wav" --weight-path /path/to/model +aha run -m qwen3asr-0.6b -i "audio.wav" + +# 仅启动服务（模型已下载） +aha serv -m qwen3asr-0.6b -p 10100 -# 仅下载模型 -aha download -m qwen3asr-0.6b ``` -### 主要功能 -* 模型管理 - 自动下载、本地模型加载 -* 服务模式 - 启动 HTTP API 服务 -* 直接推理 - 命令行直接执行推理任务 -* 多模型支持 - 支持aha已实现的模型类型 -### 详细使用说明 -请参阅 [CLI_USAGE.md](./docs/CLI_USAGE.md) 获取完整的命令行工具使用说明、参数详解和支持的模型列表。 +### 对话 -## 安装及使用 - -### 从源码构建部署 ```bash -git clone https://github.com/jhqxxx/aha.git -cd aha +aha serv -m qwen3-0.6b -p 10100 ``` -#### cargo run 运行参数说明 -##### 基本用法 +然后使用统一(兼容 OpenAI)的 API： + ```bash -cargo run -F cuda -r -- [参数] +curl http://localhost:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [{"role": "user", "content": "你好！"}] + }' ``` -##### 参数详解 -1. 端口设置 ------ - -p, --port -* 设置HTTP服务监听的端口号 -* 默认值：10100 -* 示例：--port 8080 或 -p 8080 -2. 模型选择（必选） ------ - -m, --model -* 指定要加载的模型类型 -* 可选值： - * minicpm4-0.5b：OpenBMB/MiniCPM4-0.5B 模型 - * qwen2.5vl-3b：Qwen/Qwen2.5-VL-3B-Instruct 模型 - * qwen2.5vl-7b：Qwen/Qwen2.5-VL-7B-Instruct 模型 - * qwen3-0.6b: Qwen/Qwen3-0.6B 模型 - * qwen3asr-0.6b: Qwen/Qwen3-ASR-0.6B 模型 - * qwen3asr-1.7b: Qwen/Qwen3-ASR-1.7B 模型 - * qwen3vl-2b：Qwen/Qwen3-VL-2B-Instruct 模型 - * qwen3vl-4b：Qwen/Qwen3-VL-4B-Instruct 模型 - * qwen3vl-8b：Qwen/Qwen3-VL-8B-Instruct 模型 - * qwen3vl-32b：Qwen/Qwen3-VL-32B-Instruct 模型 - * deepseek-ocr: deepseek-ai/DeepSeek-OCR 模型 - * hunyuan-ocr: Tencent-Hunyuan/HunyuanOCR 模型 - * paddleocr-vl: PaddlePaddle/PaddleOCR-VL 模型 - * rmbg2.0: AI-ModelScope/RMBG-2.0 模型 - * voxcpm: OpenBMB/VoxCPM-0.5B 模型 - * voxcpm1.5: OpenBMB/VoxCPM1.5 模型 - * glm-asr-nano-2512: ZhipuAI/GLM-ASR-Nano-2512 模型 - * fun-asr-nano-2512: FunAudioLLM/Fun-ASR-Nano-2512 模型 -* 示例：--model deepseek-ocr 或 -m qwen3vl-2b -3. 权重路径 ------ - --weight-path -* 指定本地模型权重文件路径 -* 如果指定此参数，则跳过模型下载步骤 -* 示例：--weight-path /path/to/model/dir +### 支持的模型 -4. 保存路径 ------ - --save-dir -* 指定模型下载保存的目录 -* 默认保存在用户主目录下的 .aha 文件夹中 -* 示例：--save-dir /custom/model/path +| 类别 | 模型 | +|------|------| +| **文本** | Qwen3, MiniCPM4 | +| **视觉** | Qwen2.5-VL, Qwen3-VL | +| **OCR** | DeepSeek-OCR, Hunyuan-OCR, PaddleOCR-VL | +| **ASR** | GLM-ASR-Nano, Fun-ASR-Nano,Qwen3-ASR | +| **音频** | VoxCPM, VoxCPM1.5 | +| **图像** | RMBG-2.0 (背景移除) | -5. 下载重试次数 ------ - --download-retries -* 设置模型下载失败时的最大重试次数 -* 默认值：3次 -* 示例：--download-retries 5 +## 文档 -##### 注意事项 -* 参数前需要使用双横线 -- 分隔 cargo 命令和应用程序参数 -* 模型参数 (--model 或 -m) 是必需的 -* 如果未指定 --weight-path，程序会自动下载指定模型 -* 下载的模型默认保存在 ~/.aha/ 目录下（除非指定了 --save-dir） +| 文档 | 描述 | +|------|------| +| [快速入门](docs/getting-started.zh-CN.md) | aha 入门指南 | +| [安装指南](docs/installation.zh-CN.md) | 详细安装说明 | +| [CLI 参考](docs/cli.zh-CN.md) | 命令行界面 | +| [API 文档](docs/api.zh-CN.md) | 库与 REST API | +| [支持的模型](docs/supported-models.zh-CN.md) | 可用的 AI 模型 | +| [核心概念](docs/concepts.zh-CN.md) | 架构与设计 | +| [开发指南](docs/development.zh-CN.md) | 贡献指南 | +| [更新日志](docs/changelog.zh-CN.md) | 版本历史 | -### API接口介绍 -项目提供基于 OpenAI API 兼容的 RESTful 接口，支持多种模型推理任务。 +## 为什么选择 aha？ +- **🚀 高性能推理** - 基于 Candle 框架，提供高效的张量计算和模型推理 +- **🔧 统一接口** — 一个工具搞定文本、视觉、语音和 OCR +- **📦 本地优先** — 所有处理在本地运行，数据不离境 +- **🎯 跨平台** — 支持 Linux、macOS 和 Windows +- **⚡ GPU 加速** — 可选 CUDA 支持以获得更快推理 +- **🛡️ 内存安全** — Rust 构建，稳定可靠 +- **🧠 注意力优化** - 可选 Flash Attention 支持，优化长序列处理 -#### 接口列表 -1. 对话接口 -- **端点**: `POST /chat/completions` -- **功能**: 多模态对话和文本生成 -- **支持模型**: Qwen2.5VL, Qwen3, Qwen3VL, DeepSeekOCR, GLM-ASR-Nano-2512, Fun-ASR-Nano-2512 等 -- **请求格式**: OpenAI Chat Completion 格式 -- **响应格式**: OpenAI Chat Completion 格式 -- **流式支持**: 支持 +## 开发 -2. 图像处理接口 -- **端点**: `POST /images/remove_background` -- **功能**: 图像背景移除 -- **支持模型**: RMBG-2.0 -- **请求格式**: OpenAI Chat Completion 格式 -- **响应格式**: OpenAI Chat Completion 格式 -- **流式支持**: 不支持 +### aha 作为库使用 +> cargo add aha -3. 语音生成接口 -- **端点**: `POST /audio/speech` -- **功能**: 语音合成和生成 -- **支持模型**: VoxCPM,VoxCPM1.5 -- **请求格式**: OpenAI Chat Completion 格式 -- **响应格式**: OpenAI Chat Completion 格式 -- **流式支持**: 不支持 - -### 作为库使用 -* cargo add aha -* 或者在Cargo.toml中添加 -```toml -[dependencies] -aha = { git = "https://github.com/jhqxxx/aha.git" } - -# 启用 CUDA 支持（可选） -aha = { git = "https://github.com/jhqxxx/aha.git", features = ["cuda"] } - -# 启用Flash Attention 支持（可选） -aha = { git = "https://github.com/jhqxxx/aha.git", features = ["cuda", "flash-attn"] } -``` -#### VoxCPM示例 -```rust +```bash +# VoxCPM示例 use aha::models::voxcpm::generate::VoxCPMGenerate; use aha::utils::audio_utils::save_wav; use anyhow::Result; @@ -262,108 +188,43 @@ fn main() -> Result<()> { } ``` -### 从源码构建运行测试 -```bash -git clone https://github.com/jhqxxx/aha.git -cd aha -# 修改测试用例中模型路径 -# 运行 PaddleOCR-Vl 示例 -cargo test -F cuda paddleocr_vl_generate -r -- --nocapture -# 运行 Hunyuan-OCR 示例 -cargo test -F cuda hunyuan_ocr_generate -r -- --nocapture +### 扩展新的模型 -# 运行 DeepSeek-OCR 示例 -cargo test -F cuda deepseek_ocr_generate -r -- --nocapture +- 在src/models/创建新模型文件 +- 在src/models/mod.rs中导出 +- 在src/exec/中添加支持cli运行模型推理 +- 在tests/中添加测试和示例 -# 运行 Qwen3VL 示例 -cargo test -F cuda qwen3vl_generate -r -- --nocapture -# 运行 MiniCPM4 示例 -cargo test -F cuda minicpm_generate -r -- --nocapture +## 特性 -# 运行 VoxCPM 示例 -cargo test -F cuda voxcpm_generate -r -- --nocapture -``` - -## 开发 -### 项目结构 -```text -. -├── Cargo.toml -├── README.md -├── src -│ ├── chat_template -│ ├── models -│ │ ├── common -│ │ ├── deepseek_ocr -│ │ ├── hunyuan_ocr -│ │ ├── minicpm4 -│ │ ├── paddleocr_vl -│ │ ├── qwen2_5vl -│ │ ├── qwen3vl -│ │ ├── voxcpm -│ │ └── mod.rs -│ ├── position_embed -│ ├── tokenizer -│ ├── utils -│ ├── api.rs -│ └── lib.rs -└── tests - ├── test_hunyuan_ocr.rs - ├── test_deepseek_ocr.rs - ├── test_minicpm4.rs - ├── test_paddleocr_vl.rs - ├── test_qwen2_5vl.rs - └── test_voxcpm.rs -``` - -### 添加新模型 -* 在src/models/创建新模型文件 -* 在src/models/mod.rs中导出 -* 在tests/中添加测试和示例 +- 基于 Candle 框架的高性能推理 +- 多模态模型支持（视觉、语言、语音） +- 简洁易用的 API 设计 +- 最小化依赖，紧凑的二进制文件 +- Flash Attention 支持长序列处理 +- FFmpeg 支持多媒体处理 ## 许可证 -本项目采用 Apache License, Version 2.0 许可证 - 查看 [LICENSE](./LICENSE) 文件了解详情。 + +Apache-2.0 — 详见 [LICENSE](LICENSE) ## 致谢 -* [Candle](https://github.com/huggingface/candle) - 优秀的 Rust 机器学习框架 -* 所有模型的原作者和贡献者 -## 支持 -#### 如果你遇到问题： -1. 查看 Issues 是否已有解决方案 -2. 提交新的 Issue，包含详细描述和复现步骤 +- [Candle](https://github.com/huggingface/candle) - 优秀的 Rust 机器学习框架 +- 所有模型作者和贡献者 -## 更新日志 -### [Unreleased] - 2025-02-04 -* 支持Qwen3-ASR 模型 -### v0.1.8 -* 支持Fun-ASR-Nano-2512, Qwen3 模型 +--- -### v0.1.7 -* 支持GLM-ASR-Nano-2512 模型 - -### v0.1.6 -* 支持RMGB2.0 模型 - -### v0.1.5 -* 支持VoxCPM1.5 模型 - -### v0.1.4 -* 添加PaddleOCR-VL 模型 - -### v0.1.3 -* 添加 Hunyuan-OCR 模型 - -### v0.1.2 -* 添加 DeepSeek-OCR 模型 - -### v0.1.1 -* 添加 Qwen3VL 模型 - -### v0.1.0 -* 初始版本发布 -* 支持 Qwen2.5VL, MiniCPM4, VoxCPM 模型 +

+ 由 aha 团队用 ❤️ 构建 +

+

+ 我们持续扩展支持的模型列表，欢迎贡献！ +

+

+ 如果这个项目对你有帮助，请给我们一个 ⭐ Star！ +

diff --git a/assets/img/logo.png b/assets/img/logo.png new file mode 100644 index 0000000..e50916c Binary files /dev/null and b/assets/img/logo.png differ diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..2bf3260 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,461 @@ +# API Reference + +Complete reference for the AHA REST API. + +## Overview + +AHA provides an OpenAI-compatible REST API for running AI model inference. All endpoints follow standard HTTP conventions and return JSON responses. + +### Base URL + +By default, the API server runs on: +``` +http://127.0.0.1:10100 +``` + +You can customize this when starting the service: +```bash +aha cli -m qwen3-0.6b -a 0.0.0.0 -p 8080 +``` + +### Authentication + +Currently, AHA does not require authentication. All endpoints are publicly accessible on the configured address/port. + +**Security Note**: If you expose the API to external networks, consider implementing authentication through a reverse proxy (e.g., nginx, traefik). + +### Content Types + +All requests should use: +``` +Content-Type: application/json +``` + +### Response Format + +Success responses follow this structure: +```json +{ + "data": { ... }, + "model": "model-name", + "usage": { + "prompt_tokens": 10, + "completion_tokens": 20, + "total_tokens": 30 + } +} +``` + +Error responses: +```json +{ + "error": { + "message": "Error description", + "type": "error_type", + "code": "error_code" + } +} +``` + +## Endpoints + +### Chat Completions + +Generate chat completions or text responses. + +#### Endpoint +``` +POST /chat/completions +``` + +#### Request Body + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `model` | string | Yes | Model identifier (e.g., "qwen3-0.6b") | +| `messages` | array | Yes | Array of message objects | +| `temperature` | number | No | Sampling temperature (0-2, default: 1) | +| `top_p` | number | No | Nucleus sampling (0-1, default: 1) | +| `max_tokens` | integer | No | Maximum tokens to generate | +| `stream` | boolean | No | Enable streaming (default: false) | + +#### Message Object + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `role` | string | Yes | "system", "user", or "assistant" | +| `content` | string/array | Yes | Message content (string or multimodal array) | + +#### Multimodal Content + +For vision/audio models, content can be an array: + +```json +{ + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] +} +``` + +Supported content types: +- `text` - Text content +- `image_url` - Image file (file://, base64://, or http://) +- `audio_url` - Audio file (file:// or base64://) + +#### Examples + +**Simple Chat:** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "Hello!"} + ] + }' +``` + +**With System Message:** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Explain Rust in one sentence."} + ], + "max_tokens": 50, + "temperature": 0.7 + }' +``` + +**Vision Understanding:** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3vl-2b", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What is in this image?"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] + } + ] + }' +``` + +**OCR (Text Extraction):** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ocr", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Extract all text"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/document.png"}} + ] + } + ] + }' +``` + +**ASR (Speech Recognition):** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "glm-asr-nano-2512", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Transcribe this audio"}, + {"type": "audio_url", "audio_url": {"url": "file:///path/to/audio.wav"}} + ] + } + ] + }' +``` + +**Streaming Response:** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "Tell me a story"} + ], + "stream": true + }' +``` + +Streaming responses are sent as Server-Sent Events (SSE): +``` +data: {"id": "1", "choices": [{"delta": {"content": "Once"}}]} + +data: {"id": "1", "choices": [{"delta": {"content": " upon"}}]} + +data: [DONE] +``` + +#### Response + +**Non-streaming:** + +```json +{ + "id": "chatcmpl-123", + "object": "chat.completion", + "created": 1677652288, + "model": "qwen3-0.6b", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I help you today?" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 9, + "total_tokens": 19 + } +} +``` + +#### Supported Models + +- Text: `qwen3-0.6b`, `minicpm4-0.5b` +- Vision: `qwen2.5vl-3b`, `qwen2.5vl-7b`, `qwen3vl-2b`, `qwen3vl-4b`, `qwen3vl-8b`, `qwen3vl-32b` +- OCR: `deepseek-ocr`, `hunyuan-ocr`, `paddleocr-vl` +- ASR: `glm-asr-nano-2512`, `fun-asr-nano-2512`, `qwen3asr-0.6b`, `qwen3asr-1.7b` + +### Audio Speech + +Generate speech from text (Text-to-Speech). + +#### Endpoint +``` +POST /audio/speech +``` + +#### Request Body + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `model` | string | Yes | Model identifier (e.g., "voxcpm1.5") | +| `input` | string | Yes | Text to convert to speech | +| `voice` | string | No | Voice selection (default: "default") | + +#### Example + +```bash +curl http://127.0.0.1:10100/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "voxcpm1.5", + "input": "你好，世界！", + "voice": "default" + }' \ + --output speech.wav +``` + +#### Response + +Returns audio data in WAV format. + +#### Supported Models + +- `voxcpm`, `voxcpm1.5` + +### Images Remove Background + +Remove background from images. + +#### Endpoint +``` +POST /images/remove_background +``` + +#### Request Body + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `model` | string | Yes | Model identifier (e.g., "rmbg2.0") | +| `image` | string | Yes | Image file path (file://) or base64 data | + +#### Example + +**From File:** + +```bash +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "file:///path/to/photo.png" + }' \ + --output no-background.png +``` + +**From Base64:** + +```bash +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "base64://$(base64 -w 0 photo.png)" + }' \ + --output no-background.png +``` + +#### Response + +Returns the processed image in PNG format. + +#### Supported Models + +- `rmbg2.0` + +## Error Handling + +### Error Codes + +| Code | Description | +|------|-------------| +| 400 | Bad Request - Invalid parameters | +| 404 | Not Found - Model or endpoint not found | +| 500 | Internal Server Error - Model inference error | +| 503 | Service Unavailable - Model not loaded | + +### Error Response Format + +```json +{ + "error": { + "message": "Model 'unknown-model' not found", + "type": "invalid_request_error", + "code": "model_not_found" + } +} +``` + +## Rate Limiting + +Currently, AHA does not implement rate limiting. The server can handle concurrent requests limited only by system resources. + +## File Upload Limits + +- String data: 5 MB +- File uploads: 100 MB + +## OpenAI Compatibility + +AHA's API is designed to be compatible with OpenAI's API format. This means you can use existing OpenAI client libraries with minimal changes: + +### Python Example + +```python +from openai import OpenAI + +client = OpenAI( + base_url="http://127.0.0.1:10100", + api_key="dummy" # Not used but required by library +) + +response = client.chat.completions.create( + model="qwen3-0.6b", + messages=[ + {"role": "user", "content": "Hello!"} + ] +) + +print(response.choices[0].message.content) +``` + +### JavaScript Example + +```javascript +import OpenAI from 'openai'; + +const client = new OpenAI({ + baseURL: 'http://127.0.0.1:10100', + apiKey: 'dummy' // Not used but required +}); + +const response = await client.chat.completions.create({ + model: 'qwen3-0.6b', + messages: [{ role: 'user', content: 'Hello!' }] +}); + +console.log(response.choices[0].message.content); +``` + +## Best Practices + +### 1. Use Streaming for Long Responses + +For long text generation, use streaming to get responses incrementally: + +```bash +curl ... -d '{"stream": true, ...}' +``` + +### 2. Set Appropriate Token Limits + +Prevent excessively long responses: + +```json +{ + "max_tokens": 500 +} +``` + +### 3. Adjust Temperature + +Control response creativity: +- `0.0-0.3`: Deterministic, focused +- `0.4-0.7`: Balanced (default: 1.0) +- `0.8-2.0`: Creative, varied + +### 4. Use System Messages + +Set behavior with system messages: + +```json +{ + "messages": [ + {"role": "system", "content": "You are a technical writer."}, + {"role": "user", "content": "..."} + ] +} +``` + +## See Also + +- [Getting Started](./getting-started.md) - Quick start guide +- [CLI Reference](./cli.md) - Command-line usage +- [Installation](./installation.md) - Installation guide +- [Development](./development.md) - Contributing guide diff --git a/docs/api.zh-CN.md b/docs/api.zh-CN.md new file mode 100644 index 0000000..226d7a5 --- /dev/null +++ b/docs/api.zh-CN.md @@ -0,0 +1,461 @@ +# API 参考 + +AHA REST API 的完整参考。 + +## 概述 + +AHA 提供了 OpenAI 兼容的 REST API 用于运行 AI 模型推理。所有端点遵循标准 HTTP 约定并返回 JSON 响应。 + +### 基础 URL + +默认情况下，API 服务器运行在： +``` +http://127.0.0.1:10100 +``` + +您可以在启动服务时自定义： +```bash +aha cli -m qwen3-0.6b -a 0.0.0.0 -p 8080 +``` + +### 身份验证 + +目前，AHA 不需要身份验证。所有端点在配置的地址/端口上公开访问。 + +**安全提示**：如果您将 API 暴露到外部网络，请考虑通过反向代理（如 nginx、traefik）实现身份验证。 + +### 内容类型 + +所有请求应使用： +``` +Content-Type: application/json +``` + +### 响应格式 + +成功响应遵循此结构： +```json +{ + "data": { ... }, + "model": "model-name", + "usage": { + "prompt_tokens": 10, + "completion_tokens": 20, + "total_tokens": 30 + } +} +``` + +错误响应： +```json +{ + "error": { + "message": "错误描述", + "type": "error_type", + "code": "error_code" + } +} +``` + +## 端点 + +### 对话补全 + +生成对话补全或文本响应。 + +#### 端点 +``` +POST /chat/completions +``` + +#### 请求体 + +| 参数 | 类型 | 必需 | 描述 | +|------|------|------|------| +| `model` | string | 是 | 模型标识符（如 "qwen3-0.6b"） | +| `messages` | array | 是 | 消息对象数组 | +| `temperature` | number | 否 | 采样温度（0-2，默认：1） | +| `top_p` | number | 否 | 核采样（0-1，默认：1） | +| `max_tokens` | integer | 否 | 要生成的最大令牌数 | +| `stream` | boolean | 否 | 启用流式传输（默认：false） | + +#### 消息对象 + +| 字段 | 类型 | 必需 | 描述 | +|------|------|------|------| +| `role` | string | 是 | "system"、"user" 或 "assistant" | +| `content` | string/array | 是 | 消息内容（字符串或多模态数组） | + +#### 多模态内容 + +对于视觉/音频模型，内容可以是数组： + +```json +{ + "role": "user", + "content": [ + {"type": "text", "text": "描述这张图片"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] +} +``` + +支持的内容类型： +- `text` - 文本内容 +- `image_url` - 图像文件（file://、base64:// 或 http://） +- `audio_url` - 音频文件（file:// 或 base64://） + +#### 示例 + +**简单对话：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "你好！"} + ] + }' +``` + +**带系统消息：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "system", "content": "你是一个有用的助手。"}, + {"role": "user", "content": "用一句话解释 Rust。"} + ], + "max_tokens": 50, + "temperature": 0.7 + }' +``` + +**视觉理解：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3vl-2b", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "这张图片里有什么？"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] + } + ] + }' +``` + +**OCR（文本提取）：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ocr", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "提取所有文本"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/document.png"}} + ] + } + ] + }' +``` + +**ASR（语音识别）：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "glm-asr-nano-2512", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "转写这段音频"}, + {"type": "audio_url", "audio_url": {"url": "file:///path/to/audio.wav"}} + ] + } + ] + }' +``` + +**流式响应：** + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "给我讲个故事"} + ], + "stream": true + }' +``` + +流式响应作为服务器发送事件（SSE）发送： +``` +data: {"id": "1", "choices": [{"delta": {"content": "从前"}}]} + +data: {"id": "1", "choices": [{"delta": {"content": "有"}}]} + +data: [DONE] +``` + +#### 响应 + +**非流式：** + +```json +{ + "id": "chatcmpl-123", + "object": "chat.completion", + "created": 1677652288, + "model": "qwen3-0.6b", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "你好！今天我能帮你什么？" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 9, + "total_tokens": 19 + } +} +``` + +#### 支持的模型 + +- 文本：`qwen3-0.6b`、`minicpm4-0.5b` +- 视觉：`qwen2.5vl-3b`、`qwen2.5vl-7b`、`qwen3vl-2b`、`qwen3vl-4b`、`qwen3vl-8b`、`qwen3vl-32b` +- OCR：`deepseek-ocr`、`hunyuan-ocr`、`paddleocr-vl` +- ASR：`glm-asr-nano-2512`、`fun-asr-nano-2512`、`qwen3asr-0.6b`、`qwen3asr-1.7b` + +### 语音生成 + +从文本生成语音（文本转语音）。 + +#### 端点 +``` +POST /audio/speech +``` + +#### 请求体 + +| 参数 | 类型 | 必需 | 描述 | +|------|------|------|------| +| `model` | string | 是 | 模型标识符（如 "voxcpm1.5"） | +| `input` | string | 是 | 要转换为语音的文本 | +| `voice` | string | 否 | 语音选择（默认："default"） | + +#### 示例 + +```bash +curl http://127.0.0.1:10100/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "voxcpm1.5", + "input": "你好，世界！", + "voice": "default" + }' \ + --output speech.wav +``` + +#### 响应 + +以 WAV 格式返回音频数据。 + +#### 支持的模型 + +- `voxcpm`、`voxcpm1.5` + +### 图像背景移除 + +从图像中移除背景。 + +#### 端点 +``` +POST /images/remove_background +``` + +#### 请求体 + +| 参数 | 类型 | 必需 | 描述 | +|------|------|------|------| +| `model` | string | 是 | 模型标识符（如 "rmbg2.0"） | +| `image` | string | 是 | 图像文件路径（file://）或 base64 数据 | + +#### 示例 + +**从文件：** + +```bash +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "file:///path/to/photo.png" + }' \ + --output no-background.png +``` + +**从 Base64：** + +```bash +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "base64://$(base64 -w 0 photo.png)" + }' \ + --output no-background.png +``` + +#### 响应 + +以 PNG 格式返回处理后的图像。 + +#### 支持的模型 + +- `rmbg2.0` + +## 错误处理 + +### 错误代码 + +| 代码 | 描述 | +|------|------| +| 400 | 错误请求 - 无效参数 | +| 404 | 未找到 - 模型或端点未找到 | +| 500 | 内部服务器错误 - 模型推理错误 | +| 503 | 服务不可用 - 模型未加载 | + +### 错误响应格式 + +```json +{ + "error": { + "message": "未找到模型 'unknown-model'", + "type": "invalid_request_error", + "code": "model_not_found" + } +} +``` + +## 速率限制 + +目前，AHA 不实现速率限制。服务器可以处理并发请求，仅受系统资源限制。 + +## 文件上传限制 + +- 字符串数据：5 MB +- 文件上传：100 MB + +## OpenAI 兼容性 + +AHA 的 API 设计为与 OpenAI 的 API 格式兼容。这意味着您可以使用现有的 OpenAI 客户端库，只需最少的更改： + +### Python 示例 + +```python +from openai import OpenAI + +client = OpenAI( + base_url="http://127.0.0.1:10100", + api_key="dummy" # 不使用但库需要 +) + +response = client.chat.completions.create( + model="qwen3-0.6b", + messages=[ + {"role": "user", "content": "你好！"} + ] +) + +print(response.choices[0].message.content) +``` + +### JavaScript 示例 + +```javascript +import OpenAI from 'openai'; + +const client = new OpenAI({ + baseURL: 'http://127.0.0.1:10100', + apiKey: 'dummy' // 不使用但需要 +}); + +const response = await client.chat.completions.create({ + model: 'qwen3-0.6b', + messages: [{ role: 'user', content: '你好！' }] +}); + +console.log(response.choices[0].message.content); +``` + +## 最佳实践 + +### 1. 对长响应使用流式传输 + +对于长文本生成，使用流式传输增量获取响应： + +```bash +curl ... -d '{"stream": true, ...}' +``` + +### 2. 设置适当的令牌限制 + +防止过长的响应： + +```json +{ + "max_tokens": 500 +} +``` + +### 3. 调整温度 + +控制响应创造性： +- `0.0-0.3`：确定性、专注 +- `0.4-0.7`：平衡（默认：1.0） +- `0.8-2.0`：创造性、多样 + +### 4. 使用系统消息 + +使用系统消息设置行为： + +```json +{ + "messages": [ + {"role": "system", "content": "你是一个技术写作助手。"}, + {"role": "user", "content": "..."} + ] +} +``` + +## 另见 + +- [快速入门](./getting-started.zh-CN.md) - 快速入门指南 +- [CLI 参考](./cli.zh-CN.md) - 命令行使用 +- [安装](./installation.zh-CN.md) - 安装指南 +- [开发](./development.zh-CN.md) - 贡献指南 diff --git a/docs/changelog.md b/docs/changelog.md new file mode 100644 index 0000000..3b0267b --- /dev/null +++ b/docs/changelog.md @@ -0,0 +1,90 @@ +# Changelog + +All notable changes to aha will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.2.0] - 2026-02-05 + +### Added +- Qwen3-ASR speech recognition model + +## [0.1.9] - 2026-01-31 + +### Added +- CLI `list` subcommand to show supported models +- CLI subcommand structure support (`cli`, `serv`, `download`, `run`) +- Direct model inference via new `run` subcommand + +### Fixed +- Qwen3VL thinking startswith bug +- `aha run` multiple inputs bug + +## [0.1.8] - 2026-01-17 + +### Added +- Qwen3 text model support +- Fun-ASR-Nano-2512 speech recognition model + +### Fixed +- ModelScope Fun-ASR-Nano model load error + +### Changed +- Updated audio resampling with rubato + +## [0.1.7] - 2026-01-07 + +### Added +- GLM-ASR-Nano-2512 speech recognition model +- Metal (GPU) support for Apple Silicon +- Dynamic home directory and model download script + +## [0.1.6] - 2025-12-23 + +### Added +- RMBG-2.0 background removal model +- Image and audio API endpoints + +### Changed +- Performance optimizations for RMBG2.0 image processing + +## [0.1.5] - 2025-12-11 + +### Added +- VoxCPM1.5 voice generation model +- PaddleOCR-VL text recognition model + +## [0.1.4] - 2025-12-09 + +### Added +- PaddleOCR-VL model support +- FFmpeg feature for multimedia processing + +## [0.1.3] - 2025-12-03 + +### Added +- Hunyuan-OCR model support + +## [0.1.2] - 2025-11-23 + +### Added +- DeepSeek-OCR model support + +## [0.1.1] - 2025-11-12 + +### Added +- Qwen3-VL models (2B, 4B, 8B, 32B) + +### Fixed +- Added serde default for tie_word_embeddings in Qwen3VL + +## [0.1.0] - 2025-10-10 + +### Added +- Initial release +- Qwen2.5-VL model support +- MiniCPM4 model support +- VoxCPM voice generation model +- OpenAI-compatible REST API +- CLI interface for all model types diff --git a/docs/changelog.zh-CN.md b/docs/changelog.zh-CN.md new file mode 100644 index 0000000..24ba10d --- /dev/null +++ b/docs/changelog.zh-CN.md @@ -0,0 +1,90 @@ +# 更新日志 + +所有 aha 的重大更改都将记录在此文件中。 + +格式基于 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.0.0/)， +本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/spec/v2.0.0.html)。 + +## [0.2.0] - 2026-02-05 + +### 新增 +- Qwen3-ASR 语音识别模型 + +## [0.1.9] - 2026-01-31 + +### 新增 +- CLI `list` 子命令，显示支持的模型 +- CLI 子命令结构支持（`cli`、`serv`、`download`、`run`） +- 通过新的 `run` 子命令直接进行模型推理 + +### 修复 +- Qwen3VL thinking startswith bug +- `aha run` 多输入 bug + +## [0.1.8] - 2026-01-17 + +### 新增 +- Qwen3 文本模型支持 +- Fun-ASR-Nano-2512 语音识别模型 + +### 修复 +- ModelScope Fun-ASR-Nano 模型加载错误 + +### 变更 +- 使用 rubato 更新音频重采样 + +## [0.1.7] - 2026-01-07 + +### 新增 +- GLM-ASR-Nano-2512 语音识别模型 +- Metal (GPU) 支持，适用于 Apple Silicon +- 动态主目录和模型下载脚本 + +## [0.1.6] - 2025-12-23 + +### 新增 +- RMBG-2.0 背景移除模型 +- 图像和音频 API 端点 + +### 变更 +- RMBG2.0 图像处理性能优化 + +## [0.1.5] - 2025-12-11 + +### 新增 +- VoxCPM1.5 语音生成模型 +- PaddleOCR-VL 文字识别模型 + +## [0.1.4] - 2025-12-09 + +### 新增 +- PaddleOCR-VL 模型支持 +- FFmpeg 多媒体处理功能 + +## [0.1.3] - 2025-12-03 + +### 新增 +- Hunyuan-OCR 模型支持 + +## [0.1.2] - 2025-11-23 + +### 新增 +- DeepSeek-OCR 模型支持 + +## [0.1.1] - 2025-11-12 + +### 新增 +- Qwen3-VL 系列模型 (2B, 4B, 8B, 32B) + +### 修复 +- 为 Qwen3VL 的 tie_word_embeddings 添加 serde 默认值 + +## [0.1.0] - 2025-10-10 + +### 新增 +- 初始版本发布 +- Qwen2.5-VL 模型支持 +- MiniCPM4 模型支持 +- VoxCPM 语音生成模型 +- 兼容 OpenAI 的 REST API +- 所有模型类型的 CLI 界面 diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..adb6af6 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,300 @@ +# CLI Reference + +Complete command-line interface reference for aha. + +AHA is a high-performance model inference library based on the Candle framework, supporting various multimodal models including vision, language, and audio models. + +```bash +aha [COMMAND] [OPTIONS] +``` + +## Global Options + +| Option | Description | Default | +|--------|-------------|---------| +| `-a, --address
` | Service listen address | 127.0.0.1 | +| `-p, --port ` | Service listen port | 10100 | +| `-m, --model ` | Model type (required) | - | +| `--weight-path ` | Local model weight path | - | +| `--save-dir ` | Model download save directory | ~/.aha/ | +| `--download-retries ` | Download retry count | 3 | +| `-h, --help` | Display help information | - | +| `-V, --version` | Display version number | - | + +## Commands + +### cli - Download model and start service (default) + +Download the specified model and start an HTTP service. This command is used by default when no subcommand is specified. + +**Syntax:** +```bash +aha cli [OPTIONS] --model +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `-a, --address
` | Service listen address | 127.0.0.1 | +| `-p, --port ` | Service listen port | 10100 | +| `-m, --model ` | Model type (required) | - | +| `--weight-path ` | Local model weight path (skip download if specified) | - | +| `--save-dir ` | Model download save directory | ~/.aha/ | +| `--download-retries ` | Download retry count | 3 | + +**Examples:** + +```bash +# Download model and start service (default port 10100) +aha cli -m qwen3vl-2b + +# Specify port and save directory +aha cli -m qwen3vl-2b -p 8080 --save-dir /data/models + +# Use local model (skip download) +aha cli -m qwen3vl-2b --weight-path /path/to/model + +# Backward compatible way (equivalent to cli subcommand) +aha -m qwen3vl-2b +``` + +### run - Direct model inference + +Run model inference directly without starting an HTTP service. Suitable for one-time inference tasks or batch processing. + +**Syntax:** +```bash +aha run [OPTIONS] --model --input [--input ] --weight-path +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `-m, --model ` | Model type (required) | - | +| `-i, --input ` | Input text or file path (model-specific interpretation, supports 1-2 parameters: input1: prompt text, input2: file path) | - | +| `-o, --output ` | Output file path (optional, auto-generated if not specified) | - | +| `--weight-path ` | Local model weight path (required) | - | + +**Examples:** + +```bash +# VoxCPM1.5 text-to-speech (single input) +aha run -m voxcpm1.5 -i "太阳当空照" -o output.wav --weight-path /path/to/model + +# VoxCPM1.5 read input from file (single input) +aha run -m voxcpm1.5 -i "file://./input.txt" --weight-path /path/to/model + +# MiniCPM4 text generation (single input) +aha run -m minicpm4-0.5b -i "你好" --weight-path /path/to/model + +# DeepSeek OCR image recognition (single input) +aha run -m deepseek-ocr -i "image.jpg" --weight-path /path/to/model + +# RMBG2.0 background removal (single input) +aha run -m RMBG2.0 -i "photo.png" -o "no_bg.png" --weight-path /path/to/model + +# GLM-ASR speech recognition (two inputs: prompt text + audio file) +aha run -m glm-asr-nano-2512 -i "请转写这段音频" -i "audio.wav" --weight-path /path/to/model + +# Fun-ASR speech recognition (two inputs: prompt text + audio file) +aha run -m fun-asr-nano-2512 -i "语音转写：" -i "audio.wav" --weight-path /path/to/model + +# qwen3 text generation (single input) +aha run -m qwen3-0.6b -i "你好" --weight-path /path/to/model + +# qwen2.5vl image understanding (two inputs: prompt text + image file) +aha run -m qwen2.5vl-3b -i "请分析图片并提取所有可见文本内容，按从左到右、从上到下的布局，返回纯文本" -i "image.jpg" --weight-path /path/to/model + +# Qwen3-ASR speech recognition (single input: audio file) +aha run -m qwen3asr-0.6b -i "audio.wav" --weight-path /path/to/model +``` + +### serv - Start service + +Start HTTP service only, without downloading models. Must specify local model path via `--weight-path`. + +**Syntax:** +```bash +aha serv [OPTIONS] --model --weight-path +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `-a, --address
` | Service listen address | 127.0.0.1 | +| `-p, --port ` | Service listen port | 10100 | +| `-m, --model ` | Model type (required) | - | +| `--weight-path ` | Local model weight path (required) | - | + +**Examples:** + +```bash +# Start service with local model +aha serv -m qwen3vl-2b --weight-path /path/to/model + +# Start with specified port +aha serv -m qwen3vl-2b --weight-path /path/to/model -p 8080 + +# Specify listen address +aha serv -m qwen3vl-2b --weight-path /path/to/model -a 0.0.0.0 +``` + +### download - Download model + +Download the specified model only, without starting the service. + +**Syntax:** +```bash +aha download [OPTIONS] --model +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `-m, --model ` | Model type (required) | - | +| `-s, --save-dir ` | Model download save directory | ~/.aha/ | +| `--download-retries ` | Download retry count | 3 | + +**Examples:** + +```bash +# Download model to default directory +aha download -m qwen3vl-2b + +# Specify save directory +aha download -m qwen3vl-2b -s /data/models + +# Specify download retry count +aha download -m qwen3vl-2b --download-retries 5 + +# Download MiniCPM4-0.5B model +aha download -m minicpm4-0.5b -s models +``` + +## Supported Models + +| Model ID | Model Name | Description | +|----------|------------|-------------| +| `minicpm4-0.5b` | OpenBMB/MiniCPM4-0.5B | OpenBMB MiniCPM4 0.5B model | +| `qwen2.5vl-3b` | Qwen/Qwen2.5-VL-3B-Instruct | Qwen 2.5 VL 3B model | +| `qwen2.5vl-7b` | Qwen/Qwen2.5-VL-7B-Instruct | Qwen 2.5 VL 7B model | +| `qwen3-0.6b` | Qwen/Qwen3-0.6B | Qwen 3 0.6B model | +| `qwen3vl-2b` | Qwen/Qwen3-VL-2B-Instruct | Qwen 3 VL 2B model | +| `qwen3vl-4b` | Qwen/Qwen3-VL-4B-Instruct | Qwen 3 VL 4B model | +| `qwen3vl-8b` | Qwen/Qwen3-VL-8B-Instruct | Qwen 3 VL 8B model | +| `qwen3vl-32b` | Qwen/Qwen3-VL-32B-Instruct | Qwen 3 VL 32B model | +| `deepseek-ocr` | deepseek-ai/DeepSeek-OCR | DeepSeek OCR model | +| `hunyuan-ocr` | Tencent-Hunyuan/HunyuanOCR | Tencent Hunyuan OCR model | +| `paddleocr-vl` | PaddlePaddle/PaddleOCR-VL | Baidu PaddleOCR VL model | +| `RMBG2.0` | AI-ModelScope/RMBG-2.0 | RMBG 2.0 background removal model | +| `voxcpm` | OpenBMB/VoxCPM-0.5B | OpenBMB VoxCPM 0.5B speech synthesis model | +| `voxcpm1.5` | OpenBMB/VoxCPM1.5 | OpenBMB VoxCPM 1.5 speech synthesis model | +| `glm-asr-nano-2512` | ZhipuAI/GLM-ASR-Nano-2512 | Zhipu AI ASR Nano 2512 speech recognition model | +| `fun-asr-nano-2512` | FunAudioLLM/Fun-ASR-Nano-2512 | FunAudioLLM ASR Nano 2512 speech recognition model | + +## Common Use Cases + +### Scenario 1: Quick start inference service + +```bash +# One command to download and start service +aha -m qwen3vl-2b +``` + +### Scenario 2: Start service with existing model + +```bash +# Assuming model is downloaded to /data/models/Qwen/Qwen3-VL-2B-Instruct +aha serv -m qwen3vl-2b --weight-path /data/models/Qwen/Qwen3-VL-2B-Instruct +``` + +### Scenario 3: Pre-download model + +```bash +# Download model to specified directory for later use +aha download -m qwen3vl-2b -s /data/models + +# Later start with local model +aha serv -m qwen3vl-2b --weight-path /data/models/Qwen/Qwen3-VL-2B-Instruct +``` + +### Scenario 4: Custom service port and address + +```bash +# Start service on 0.0.0.0:8080, allow external access +aha -m qwen3vl-2b -a 0.0.0.0 -p 8080 +``` + +## API Endpoints + +After the service starts, the following API endpoints are available: + +### Chat Completion Endpoint +- **Endpoint**: `POST /chat/completions` +- **Function**: Multimodal chat and text generation +- **Supported Models**: Qwen2.5VL, Qwen3, Qwen3VL, DeepSeekOCR, GLM-ASR-Nano-2512, Fun-ASR-Nano-2512, etc. +- **Format**: OpenAI Chat Completion format +- **Streaming Support**: Yes + +### Image Processing Endpoint +- **Endpoint**: `POST /images/remove_background` +- **Function**: Image background removal +- **Supported Models**: RMBG-2.0 +- **Format**: OpenAI Chat Completion format +- **Streaming Support**: No + +### Audio Generation Endpoint +- **Endpoint**: `POST /audio/speech` +- **Function**: Speech synthesis and generation +- **Supported Models**: VoxCPM, VoxCPM1.5 +- **Format**: OpenAI Chat Completion format +- **Streaming Support**: No + +## Backward Compatibility + +To maintain compatibility with older versions, the following two usage methods are equivalent: + +```bash +# New way (recommended) +aha cli -m qwen3vl-2b + +# Old way (backward compatible) +aha -m qwen3vl-2b +``` + +## Notes + +1. **serv subcommand requires `--weight-path`**: Since the `serv` subcommand does not download models, you must specify the path to an already downloaded model via `--weight-path`. + +2. **Download retry mechanism**: By default, retries 3 times, waiting 2 seconds after each failure before retrying. You can adjust the retry count with `--download-retries`. + +3. **Default save directory**: Models are saved to `~/.aha/` directory by default, which can be customized via `--save-dir` or `-s` parameter. + +4. **Port occupation**: Ensure the specified port is not occupied before starting the service. The default port is 10100. + +5. **Permission issues**: If saving to a system directory (such as `/data/models`), ensure you have the corresponding write permissions. + +## Getting Help + +```bash +# View main help +aha --help + +# View subcommand help +aha cli --help +aha serv --help +aha download --help + +# View version information +aha --version +``` + +## See Also + +- [Getting Started](./getting-started.md) - Quick start guide +- [API Documentation](./api.md) - REST API reference +- [Supported Models](./supported-tools.md) - Available models diff --git a/docs/CLI_USAGE.md b/docs/cli.zh-CN.md similarity index 96% rename from docs/CLI_USAGE.md rename to docs/cli.zh-CN.md index f23a025..84b2c5e 100644 --- a/docs/CLI_USAGE.md +++ b/docs/cli.zh-CN.md @@ -1,11 +1,11 @@ -# AHA 命令行使用说明 +# CLI 参考 -## 概述 +aha 的完整命令行界面参考。 AHA 是一个基于 Candle 框架的高性能模型推理库，支持多种多模态模型，包括视觉、语言和语音模型。 ```bash -aha [COMMAND] [OPTIONS] +aha [COMMAND] [OPTIONS] ``` ## 全局选项 @@ -105,7 +105,7 @@ aha run -m fun-asr-nano-2512 -i "语音转写：" -i "audio.wav" --weight-path / aha run -m qwen3-0.6b -i "你好" --weight-path /path/to/model # qwen2.5vl 图像理解（两个输入：提示文本 + 图片文件） -aha run -m qwen2.5vl-3b -i "请分析图片并提取所有可见文本内容，按从左到右、从上到下的布局，返回纯文本" -i "image.jpg" --weight-path /path/to/model +aha run -m qwen2.5vl-3b -i "请分析图片并提取所有可见文本内容，按从左到右、从上到下的布局，返回纯文本" -i "image.jpg" --weight-path /path/to/model # Qwen3-ASR 语音识别（单个输入：音频文件） aha run -m qwen3asr-0.6b -i "audio.wav" --weight-path /path/to/model @@ -272,7 +272,7 @@ aha -m qwen3vl-2b 2. **下载重试机制**：默认重试 3 次，每次失败后等待 2 秒再重试。可通过 `--download-retries` 调整重试次数。 -3. **默认保存目录**：模型默认保存到 `~/.aha/` 目录下，可通过 `--save-dir` 或 `-d` 参数自定义。 +3. **默认保存目录**：模型默认保存到 `~/.aha/` 目录下，可通过 `--save-dir` 或 `-s` 参数自定义。 4. **端口占用**：启动服务前确保指定的端口未被占用，默认端口为 10100。 @@ -292,3 +292,9 @@ aha download --help # 查看版本信息 aha --version ``` + +## 另见 + +- [快速入门](./getting-started.zh-CN.md) - 快速入门指南 +- [API 文档](./api.zh-CN.md) - REST API 参考 +- [支持的模型](./supported-tools.zh-CN.md) - 可用模型 diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 0000000..db4c784 --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,411 @@ +# Architecture & Design + +This document provides an in-depth look at the architecture and design principles behind AHA. + +## Overview + +AHA (High-performance AI inference engine) is a Rust-based library built on the [Candle](https://github.com/huggingface/candle) framework. It provides a unified interface for running multiple state-of-the-art AI models locally, without requiring API keys or cloud services. + +### Key Characteristics + +- **Local-First**: All inference runs on your machine +- **Multi-Modal**: Support for text, vision, audio, OCR, and ASR models +- **Cross-Platform**: Linux, macOS, and Windows support +- **GPU-Accelerated**: Optional CUDA and Metal support +- **Memory-Safe**: Built with Rust for safety and performance +- **OpenAI-Compatible**: Easy integration with existing tools + +## Architecture Principles + +### 1. Local-First Design + +AHA is designed to run entirely on your local machine: + +- **No cloud dependencies**: All models are downloaded and run locally +- **Privacy-preserving**: Your data never leaves your machine +- **No API keys required**: Once downloaded, models work indefinitely +- **Offline capable**: Models work without internet connection after download + +### 2. Unified Model Interface + +All models implement a common `GenerateModel` trait, providing: + +- Consistent API across different model types +- Easy model switching without code changes +- Streaming response support for real-time outputs +- Standardized error handling + +### 3. Cross-Platform Support + +AHA abstracts platform differences: + +- **Device abstraction**: Automatic CPU/GPU detection and selection +- **Precision handling**: Dynamic F32/F16/BF16 selection based on hardware +- **Path management**: Consistent model storage across platforms + +## Core Components + +``` +┌─────────────────────────────────────────────────────────────┐ +│ CLI Layer │ +│ (main.rs - Command parsing, model download, service mgmt) │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ HTTP API Layer │ +│ (api.rs - OpenAI-compatible endpoints, streaming, auth) │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ Model Abstraction Layer │ +│ (GenerateModel trait - unified interface) │ +└─────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ +┌───────▼────────┐ ┌────────▼─────────┐ ┌───────▼────────┐ +│ Text Models │ │ Vision Models │ │ Audio Models │ +│ - Qwen3 │ │ - Qwen2.5VL │ │ - VoxCPM │ +│ - MiniCPM4 │ │ - Qwen3VL │ │ - VoxCPM1.5 │ +└────────────────┘ └──────────────────┘ └────────────────┘ + │ │ │ +┌───────▼────────┐ ┌────────▼─────────┐ ┌───────▼────────┐ +│ OCR Models │ │ ASR Models │ │ Image Models │ +│ - DeepSeek │ │ - GLM-ASR │ │ - RMBG2.0 │ +│ - Hunyuan │ │ - Fun-ASR │ │ │ +│ - PaddleOCR │ │ - Qwen3-ASR │ │ │ +└────────────────┘ └──────────────────┘ └────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ Utility Modules │ +│ - tokenizer: Tokenization utilities │ +│ - chat_template: Chat format handling │ +│ - position_embed: Positional embeddings │ +│ - utils: Common utilities (audio, image, download) │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ Candle ML Framework │ +│ (Tensor operations, model loading, device management) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### CLI Layer (`src/main.rs`) + +The CLI layer provides command-line interface functionality: + +- **Command parsing**: Uses `clap` for argument parsing +- **Model management**: Automatic download and caching +- **Service control**: Start/stop HTTP server +- **Direct inference**: Run models without server + +**Available Commands**: +- `cli` - Download model and start service (default) +- `serv` - Start service with existing model +- `download` - Download model only +- `run` - Direct model inference +- `list` - List supported models + +### HTTP API Layer (`src/api.rs`) + +The HTTP API layer provides REST endpoints: + +- **OpenAI-compatible**: Matches OpenAI API format +- **Streaming support**: Real-time response generation +- **Multi-modal**: Handles text, images, and audio +- **Thread-safe**: Uses RwLock for concurrent requests + +**Endpoints**: +- `POST /chat/completions` - Chat and text generation +- `POST /images/remove_background` - Image background removal +- `POST /audio/speech` - Text-to-speech synthesis + +### Model Abstraction Layer + +All models implement the `GenerateModel` trait: + +```rust +pub trait GenerateModel { + // Initialize model with weights + fn init(model_path: &str, config: Option) -> Result; + + // Generate response + fn generate(&mut self, prompt: &str, params: GenerationParams) -> Result; + + // Generate with streaming + fn generate_stream(&mut self, prompt: &str, params: GenerationParams) + -> Result>>>; +} +``` + +This provides: +- **Polymorphism**: Treat different models uniformly +- **Extensibility**: Easy to add new models +- **Type safety**: Compile-time guarantees + +### Utility Modules + +#### Tokenizer (`src/tokenizer/`) + +- Loads tokenizers from model configurations +- Handles special tokens +- Manages vocabulary + +#### Chat Template (`src/chat_template/`) + +- Formats chat messages into model prompts +- Supports multiple chat formats (ChatML, etc.) +- Handles system messages and role tags + +#### Position Embeddings (`src/position_embed/`) + +- Implements positional encoding for transformers +- Supports RoPE (Rotary Position Embedding) +- Handles M-RoPE for multimodal models + +#### Utils (`src/utils/`) + +- `audio_utils.rs` - Audio processing (WAV, MP3) +- `image_utils.rs` - Image processing (resize, encode/decode) +- `download.rs` - Model downloading with retry logic +- `common.rs` - Common utilities and constants + +## Design Patterns + +### 1. Trait-Based Abstraction + +The `GenerateModel` trait provides a unified interface: + +```rust +// All models implement this trait +impl GenerateModel for Qwen3VL { /* ... */ } +impl GenerateModel for VoxCPM { /* ... */ } +impl GenerateModel for DeepSeekOCR { /* ... */ } + +// Usage is model-agnostic +let mut model: Box = load_model(model_type)?; +let result = model.generate(prompt, params)?; +``` + +### 2. Factory Pattern + +Model loading uses a factory function: + +```rust +pub fn load_model( + model_type: &str, + model_path: &str, + device: &Device, +) -> Result> { + match model_type { + "qwen3vl-2b" => Ok(Box::new(qwen3vl::generate::Qwen3VLGenerate::init(...)?)), + "voxcpm1.5" => Ok(Box::new(voxcpm::generate::VoxCPMGenerate::init(...)?)), + // ... other models + _ => Err(anyhow!("Unsupported model: {}", model_type)), + } +} +``` + +### 3. Command Pattern + +CLI subcommands encapsulate different operations: + +```rust +match command { + Commands::Cli { model, port, address } => { /* download and serve */ } + Commands::Serv { model, weight_path, port } => { /* serve only */ } + Commands::Download { model, save_dir } => { /* download only */ } + Commands::Run { model, input, weight_path } => { /* direct inference */ } + Commands::List => { /* list models */ } +} +``` + +## Model Organization + +Each model follows a consistent structure: + +``` +src/models/{model_name}/ +├── config.rs # Model configuration and generation parameters +├── model.rs # Core model architecture (layers, attention) +├── generate.rs # Inference logic (implements GenerateModel trait) +├── processor.rs # Model-specific processing (for complex models) +└── mod.rs # Module declaration and exports +``` + +### Example: Qwen3VL + +``` +src/models/qwen3vl/ +├── config.rs # Qwen3VLConfig, GenerationConfig +├── model.rs # Qwen3VL transformer layers, attention mechanisms +├── generate.rs # Qwen3VLGenerate implementation +├── processor.rs # Image and text processing for multimodal input +└── mod.rs # Exports public API +``` + +## Performance Optimizations + +### GPU Acceleration + +AHA supports GPU acceleration through: + +- **CUDA**: For NVIDIA GPUs (Linux, Windows) +- **Metal**: For Apple Silicon (macOS) + +Enable with: +```bash +cargo build --features cuda # NVIDIA GPUs +cargo build --features metal # Apple Silicon +``` + +### Flash Attention + +Flash Attention optimizes long-sequence processing: + +- Reduces memory usage +- Improves inference speed +- Especially beneficial for vision models + +Enable with: +```bash +cargo build --features "cuda,flash-attn" +``` + +### Memory-Mapped Tensors + +Models use memory-mapped files for: + +- Faster loading times +- Reduced memory footprint +- Concurrent model loading + +### Precision Optimization + +Dynamic precision selection based on hardware: + +- **F32**: Maximum accuracy (CPU-only) +- **F16**: Balanced performance (GPU) +- **BF16**: Best for modern GPUs + +## Security Considerations + +### Local-Only Processing + +- No external API calls after model download +- No telemetry or data collection +- Full control over your data + +### Memory Safety + +- Rust's ownership system prevents memory leaks +- No buffer overflows or use-after-free bugs +- Thread-safe concurrent operations + +### Input Validation + +- File size limits (5MB strings, 100MB files) +- Path validation to prevent directory traversal +- Type-safe request handling + +## Data Flow + +### Request Flow + +``` +┌─────────┐ +│ Client │ +└────┬────┘ + │ HTTP Request + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Rocket HTTP Server │ +│ - Route request to endpoint │ +│ - Parse request body │ +│ - Extract parameters │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ API Handler (api.rs) │ +│ - Acquire model lock │ +│ - Prepare input (tokenize, process images/audio) │ +│ - Call model.generate() or generate_stream() │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Model Implementation (models/{model}/generate.rs) │ +│ - Load weights from memory-mapped files │ +│ - Run forward pass through Candle tensors │ +│ - Decode output tokens │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Candle Framework │ +│ - Execute on CPU or GPU device │ +│ - Manage tensor operations │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Response Generation │ +│ - Format response (JSON / streaming) │ +│ - Return to client │ +└──────────────────────────────────────────────────────────┘ +``` + +### Model Loading Flow + +``` +User specifies model + │ + ▼ +Check if --weight-path provided + │ + ┌───┴───┐ + │ │ + Yes No + │ │ + ▼ ▼ +Use local Download from ModelScope +path │ + │ ▼ + │ Save to ~/.aha/{model}/ + │ │ + └───┬────┘ + ▼ +Load model weights into memory + │ + ▼ +Initialize model (init()) + │ + ▼ +Ready for inference +``` + +## Extension Points + +### Adding a New Model + +1. Create model directory under `src/models/` +2. Implement `GenerateModel` trait +3. Add model to factory function in `mod.rs` +4. Add CLI mapping in `main.rs` +5. Add test case in `tests/` + +### Custom Processing + +Models can override default processing: + +- Custom tokenization +- Special input/output formats +- Model-specific optimizations + +## See Also + +- [Installation Guide](./installation.md) - Setup and installation +- [Getting Started](./getting-started.md) - Quick start guide +- [API Reference](./api.md) - REST API documentation +- [Development](./development.md) - Contributing guide diff --git a/docs/concepts.zh-CN.md b/docs/concepts.zh-CN.md new file mode 100644 index 0000000..2e1020e --- /dev/null +++ b/docs/concepts.zh-CN.md @@ -0,0 +1,411 @@ +# 架构与设计 + +本文档深入介绍 AHA 的架构和设计原则。 + +## 概述 + +AHA（高性能 AI 推理引擎）是一个基于 Rust 的库，构建在 [Candle](https://github.com/huggingface/candle) 框架之上。它提供了统一的接口，可以在本地运行多个最先进的 AI 模型，无需 API 密钥或云服务。 + +### 核心特性 + +- **本地优先**：所有推理都在您的机器上运行 +- **多模态支持**：支持文本、视觉、音频、OCR 和 ASR 模型 +- **跨平台**：支持 Linux、macOS 和 Windows +- **GPU 加速**：可选的 CUDA 和 Metal 支持 +- **内存安全**：使用 Rust 构建，确保安全性和性能 +- **OpenAI 兼容**：易于与现有工具集成 + +## 架构原则 + +### 1. 本地优先设计 + +AHA 设计为完全在本地运行： + +- **无云依赖**：所有模型都在本地下载和运行 +- **隐私保护**：您的数据永远不会离开您的机器 +- **无需 API 密钥**：下载后模型可永久使用 +- **离线可用**：下载后模型无需互联网连接 + +### 2. 统一模型接口 + +所有模型都实现通用的 `GenerateModel` trait，提供： + +- 不同模型类型之间的一致 API +- 无需更改代码即可轻松切换模型 +- 支持实时输出的流式响应 +- 标准化的错误处理 + +### 3. 跨平台支持 + +AHA 抽象了平台差异： + +- **设备抽象**：自动 CPU/GPU 检测和选择 +- **精度处理**：基于硬件动态选择 F32/F16/BF16 +- **路径管理**：跨平台一致的模型存储 + +## 核心组件 + +``` +┌─────────────────────────────────────────────────────────────┐ +│ CLI 层 │ +│ (main.rs - 命令解析、模型下载、服务管理) │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ HTTP API 层 │ +│ (api.rs - OpenAI 兼容端点、流式传输、认证) │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ 模型抽象层 │ +│ (GenerateModel trait - 统一接口) │ +└─────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ +┌───────▼────────┐ ┌────────▼─────────┐ ┌───────▼────────┐ +│ 文本模型 │ │ 视觉模型 │ │ 音频模型 │ +│ - Qwen3 │ │ - Qwen2.5VL │ │ - VoxCPM │ +│ - MiniCPM4 │ │ - Qwen3VL │ │ - VoxCPM1.5 │ +└────────────────┘ └──────────────────┘ └────────────────┘ + │ │ │ +┌───────▼────────┐ ┌────────▼─────────┐ ┌───────▼────────┐ +│ OCR 模型 │ │ ASR 模型 │ │ 图像模型 │ +│ - DeepSeek │ │ - GLM-ASR │ │ - RMBG2.0 │ +│ - Hunyuan │ │ - Fun-ASR │ │ │ +│ - PaddleOCR │ │ - Qwen3-ASR │ │ │ +└────────────────┘ └──────────────────┘ └────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ 工具模块 │ +│ - tokenizer: 分词工具 │ +│ - chat_template: 聊天格式处理 │ +│ - position_embed: 位置编码 │ +│ - utils: 通用工具（音频、图像、下载） │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────────────────────────────────────────────┐ +│ Candle ML 框架 │ +│ (张量操作、模型加载、设备管理) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### CLI 层 (`src/main.rs`) + +CLI 层提供命令行界面功能： + +- **命令解析**：使用 `clap` 进行参数解析 +- **模型管理**：自动下载和缓存 +- **服务控制**：启动/停止 HTTP 服务器 +- **直接推理**：无需服务器即可运行模型 + +**可用命令**： +- `cli` - 下载模型并启动服务（默认） +- `serv` - 使用现有模型启动服务 +- `download` - 仅下载模型 +- `run` - 直接模型推理 +- `list` - 列出支持的模型 + +### HTTP API 层 (`src/api.rs`) + +HTTP API 层提供 REST 端点： + +- **OpenAI 兼容**：匹配 OpenAI API 格式 +- **流式支持**：实时响应生成 +- **多模态**：处理文本、图像和音频 +- **线程安全**：使用 RwLock 处理并发请求 + +**端点**： +- `POST /chat/completions` - 聊天和文本生成 +- `POST /images/remove_background` - 图像背景移除 +- `POST /audio/speech` - 文本转语音合成 + +### 模型抽象层 + +所有模型都实现 `GenerateModel` trait： + +```rust +pub trait GenerateModel { + // 使用权重初始化模型 + fn init(model_path: &str, config: Option) -> Result; + + // 生成响应 + fn generate(&mut self, prompt: &str, params: GenerationParams) -> Result; + + // 流式生成 + fn generate_stream(&mut self, prompt: &str, params: GenerationParams) + -> Result>>>; +} +``` + +这提供了： +- **多态性**：统一处理不同模型 +- **可扩展性**：易于添加新模型 +- **类型安全**：编译时保证 + +### 工具模块 + +#### 分词器 (`src/tokenizer/`) + +- 从模型配置加载分词器 +- 处理特殊标记 +- 管理词汇表 + +#### 聊天模板 (`src/chat_template/`) + +- 将聊天消息格式化为模型提示 +- 支持多种聊天格式（ChatML 等） +- 处理系统消息和角色标签 + +#### 位置编码 (`src/position_embed/`) + +- 为 transformer 实现位置编码 +- 支持 RoPE（旋转位置编码） +- 处理多模态模型的 M-RoPE + +#### 工具 (`src/utils/`) + +- `audio_utils.rs` - 音频处理（WAV、MP3） +- `image_utils.rs` - 图像处理（调整大小、编码/解码） +- `download.rs` - 带重试逻辑的模型下载 +- `common.rs` - 通用工具和常量 + +## 设计模式 + +### 1. 基于 Trait 的抽象 + +`GenerateModel` trait 提供统一接口： + +```rust +// 所有模型都实现此 trait +impl GenerateModel for Qwen3VL { /* ... */ } +impl GenerateModel for VoxCPM { /* ... */ } +impl GenerateModel for DeepSeekOCR { /* ... */ } + +// 使用方式与模型无关 +let mut model: Box = load_model(model_type)?; +let result = model.generate(prompt, params)?; +``` + +### 2. 工厂模式 + +模型加载使用工厂函数： + +```rust +pub fn load_model( + model_type: &str, + model_path: &str, + device: &Device, +) -> Result> { + match model_type { + "qwen3vl-2b" => Ok(Box::new(qwen3vl::generate::Qwen3VLGenerate::init(...)?)), + "voxcpm1.5" => Ok(Box::new(voxcpm::generate::VoxCPMGenerate::init(...)?)), + // ... 其他模型 + _ => Err(anyhow!("不支持的模型: {}", model_type)), + } +} +``` + +### 3. 命令模式 + +CLI 子命令封装不同的操作： + +```rust +match command { + Commands::Cli { model, port, address } => { /* 下载并启动服务 */ } + Commands::Serv { model, weight_path, port } => { /* 仅启动服务 */ } + Commands::Download { model, save_dir } => { /* 仅下载 */ } + Commands::Run { model, input, weight_path } => { /* 直接推理 */ } + Commands::List => { /* 列出模型 */ } +} +``` + +## 模型组织结构 + +每个模型都遵循一致的结构： + +``` +src/models/{model_name}/ +├── config.rs # 模型配置和生成参数 +├── model.rs # 核心模型架构（层、注意力） +├── generate.rs # 推理逻辑（实现 GenerateModel trait） +├── processor.rs # 模型特定处理（用于复杂模型） +└── mod.rs # 模块声明和导出 +``` + +### 示例：Qwen3VL + +``` +src/models/qwen3vl/ +├── config.rs # Qwen3VLConfig、GenerationConfig +├── model.rs # Qwen3VL transformer 层、注意力机制 +├── generate.rs # Qwen3VLGenerate 实现 +├── processor.rs # 多模态输入的图像和文本处理 +└── mod.rs # 导出公共 API +``` + +## 性能优化 + +### GPU 加速 + +AHA 通过以下方式支持 GPU 加速： + +- **CUDA**：用于 NVIDIA GPU（Linux、Windows） +- **Metal**：用于 Apple Silicon（macOS） + +启用方式： +```bash +cargo build --features cuda # NVIDIA GPU +cargo build --features metal # Apple Silicon +``` + +### Flash Attention + +Flash Attention 优化长序列处理： + +- 减少内存使用 +- 提高推理速度 +- 对视觉模型特别有益 + +启用方式： +```bash +cargo build --features "cuda,flash-attn" +``` + +### 内存映射张量 + +模型使用内存映射文件： + +- 更快的加载时间 +- 减少内存占用 +- 支持并发模型加载 + +### 精度优化 + +基于硬件的动态精度选择： + +- **F32**：最高精度（仅 CPU） +- **F16**：平衡性能（GPU） +- **BF16**：最适合现代 GPU + +## 安全考虑 + +### 本地处理 + +- 模型下载后无外部 API 调用 +- 无遥测或数据收集 +- 完全控制您的数据 + +### 内存安全 + +- Rust 所有权系统防止内存泄漏 +- 无缓冲区溢出或使用后释放错误 +- 线程安全的并发操作 + +### 输入验证 + +- 文件大小限制（字符串 5MB，文件 100MB） +- 路径验证防止目录遍历 +- 类型安全的请求处理 + +## 数据流 + +### 请求流程 + +``` +┌─────────┐ +│ 客户端 │ +└────┬────┘ + │ HTTP 请求 + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Rocket HTTP 服务器 │ +│ - 将请求路由到端点 │ +│ - 解析请求体 │ +│ - 提取参数 │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ API 处理器 (api.rs) │ +│ - 获取模型锁 │ +│ - 准备输入（分词、处理图像/音频） │ +│ - 调用 model.generate() 或 generate_stream() │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ 模型实现 (models/{model}/generate.rs) │ +│ - 从内存映射文件加载权重 │ +│ - 通过 Candle 张量运行前向传播 │ +│ - 解码输出标记 │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Candle 框架 │ +│ - 在 CPU 或 GPU 设备上执行 │ +│ - 管理张量操作 │ +└────────────┬─────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ 响应生成 │ +│ - 格式化响应（JSON / 流式） │ +│ - 返回给客户端 │ +└──────────────────────────────────────────────────────────┘ +``` + +### 模型加载流程 + +``` +用户指定模型 + │ + ▼ +检查是否提供了 --weight-path + │ + ┌───┴───┐ + │ │ + 是 否 + │ │ + ▼ ▼ +使用本地 从 ModelScope 下载 +路径 │ + │ ▼ + │ 保存到 ~/.aha/{model}/ + │ │ + └───┬────┘ + ▼ +将模型权重加载到内存 + │ + ▼ +初始化模型 (init()) + │ + ▼ +准备就绪，可以进行推理 +``` + +## 扩展点 + +### 添加新模型 + +1. 在 `src/models/` 下创建模型目录 +2. 实现 `GenerateModel` trait +3. 在 `mod.rs` 的工厂函数中添加模型 +4. 在 `main.rs` 中添加 CLI 映射 +5. 在 `tests/` 中添加测试用例 + +### 自定义处理 + +模型可以覆盖默认处理： + +- 自定义分词 +- 特殊的输入/输出格式 +- 模型特定的优化 + +## 另见 + +- [安装指南](./installation.zh-CN.md) - 设置和安装 +- [快速入门](./getting-started.zh-CN.md) - 快速入门指南 +- [API 参考](./api.zh-CN.md) - REST API 文档 +- [开发指南](./development.zh-CN.md) - 贡献指南 diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..7a4a15d --- /dev/null +++ b/docs/development.md @@ -0,0 +1,601 @@ +# Development Guide + +This guide covers contributing to AHA, including development setup, adding new models, and submitting contributions. + +## Table of Contents + +- [Development Setup](#development-setup) +- [Project Structure](#project-structure) +- [Adding a New Model](#adding-a-new-model) +- [Testing](#testing) +- [Code Style](#code-style) +- [Submitting Contributions](#submitting-contributions) +- [Release Process](#release-process) + +## Development Setup + +### Prerequisites + +- Rust 1.85+ (edition 2024) +- Git +- Optional: CUDA Toolkit, Metal, FFmpeg (for feature development) + +### Clone and Build + +```bash +# Clone the repository +git clone https://github.com/jhqxxx/aha.git +cd aha + +# Build in debug mode (faster compilation) +cargo build + +# Build in release mode (better performance) +cargo build --release + +# Build with features +cargo build --features "cuda,flash-attn" +cargo build --features "metal" +cargo build --features "ffmpeg" +``` + +### Development Workflow + +```bash +# Run the CLI +cargo run -- -m qwen3-0.6b + +# Run tests +cargo test + +# Run specific test +cargo test test_qwen3vl_generate + +# Run with logging +RUST_LOG=debug cargo run -- -m qwen3-0.6b + +# Check code without building +cargo check + +# Format code +cargo fmt + +# Run linter +cargo clippy +``` + +## Project Structure + +``` +aha/ +├── Cargo.toml # Build configuration +├── src/ +│ ├── main.rs # CLI entry point +│ ├── lib.rs # Library exports +│ ├── api.rs # HTTP API handlers +│ ├── exec/ # CLI command implementations +│ │ ├── mod.rs +│ │ ├── cli.rs +│ │ ├── serv.rs +│ │ ├── download.rs +│ │ └── run.rs +│ ├── models/ # Model implementations +│ │ ├── mod.rs # Model factory and exports +│ │ ├── common/ # Shared model utilities +│ │ │ ├── config.rs +│ │ │ └── mod.rs +│ │ ├── qwen3vl/ # Qwen3-VL model +│ │ │ ├── config.rs +│ │ │ ├── model.rs +│ │ │ ├── generate.rs +│ │ │ ├── processor.rs +│ │ │ └── mod.rs +│ │ ├── voxcpm/ # VoxCPM model +│ │ └── ... # Other models +│ ├── tokenizer/ # Tokenization utilities +│ ├── chat_template/ # Chat template handling +│ ├── position_embed/ # Positional embeddings +│ └── utils/ # Utility functions +│ ├── audio_utils.rs +│ ├── image_utils.rs +│ ├── download.rs +│ └── common.rs +├── tests/ # Integration tests +│ ├── test_qwen2_5vl.rs +│ ├── test_qwen3vl.rs +│ └── ... +├── examples/ # Example code +└── docs/ # Documentation +``` + +## Adding a New Model + +This section provides a step-by-step guide for adding a new model to AHA. + +### Step 1: Create Model Directory + +Create a new directory under `src/models/`: + +```bash +mkdir -p src/models/newmodel +``` + +### Step 2: Implement Model Files + +Create the following files in `src/models/newmodel/`: + +#### config.rs + +Define model configuration: + +```rust +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct NewModelConfig { + pub vocab_size: usize, + pub hidden_size: usize, + pub num_hidden_layers: usize, + pub num_attention_heads: usize, + // ... other config fields +} + +impl Default for NewModelConfig { + fn default() -> Self { + Self { + vocab_size: 32000, + hidden_size: 2048, + num_hidden_layers: 24, + num_attention_heads: 32, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerationConfig { + pub temperature: f64, + pub top_p: f64, + pub max_tokens: usize, +} + +impl Default for GenerationConfig { + fn default() -> Self { + Self { + temperature: 1.0, + top_p: 1.0, + max_tokens: 100, + } + } +} +``` + +#### model.rs + +Implement the model architecture: + +```rust +use candle::{Result, Tensor}; +use candle_nn::{Module, VarBuilder}; + +use super::config::NewModelConfig; + +pub struct NewModel { + // Define model components + config: NewModelConfig, +} + +impl NewModel { + pub fn load(vb: VarBuilder, config: &NewModelConfig) -> Result { + // Load model weights + Ok(Self { + config: config.clone(), + }) + } + + pub fn forward(&self, input_ids: &Tensor) -> Result { + // Implement forward pass + Ok(input_ids.clone()) + } +} +``` + +#### generate.rs + +Implement the GenerateModel trait: + +```rust +use std::path::Path; +use anyhow::Result; +use candle::Device; + +use super::config::{GenerationConfig, NewModelConfig}; +use aha::models::common::GenerateModel; + +pub struct NewModelGenerate { + // Define generate state + model: super::model::NewModel, + config: GenerationConfig, + device: Device, +} + +impl GenerateModel for NewModelGenerate { + type Config = NewModelConfig; + type GenerateConfig = GenerationConfig; + + fn init( + model_path: &str, + config: Option, + generate_config: Option, + ) -> Result { + // Load model from disk + let device = Device::Cpu; + let config = config.unwrap_or_default(); + let generate_config = generate_config.unwrap_or_default(); + + // Load weights + let model_path = Path::new(model_path); + // ... load model implementation + + Ok(Self { + model: todo!(), + config: generate_config, + device, + }) + } + + fn generate(&mut self, prompt: &str) -> Result { + // Tokenize prompt + // Run inference + // Decode output + Ok(prompt.to_string()) + } +} +``` + +#### processor.rs (optional) + +For complex models with multimodal input: + +```rust +use anyhow::Result; + +pub struct NewModelProcessor { + // Processing state +} + +impl NewModelProcessor { + pub fn new() -> Self { + Self {} + } + + pub fn process_text(&self, text: &str) -> Result> { + // Process text input + Ok(vec![text.to_string()]) + } + + pub fn process_image(&self, image_path: &str) -> Result { + // Process image input + todo!() + } +} +``` + +#### mod.rs + +Export the model: + +```rust +mod config; +mod model; +mod generate; +pub mod processor; + +pub use generate::NewModelGenerate; +pub use config::{GenerationConfig, NewModelConfig}; +``` + +### Step 3: Register Model + +Update `src/models/mod.rs`: + +```rust +// Add to imports +pub mod newmodel; + +// Add to WhichModel enum +#[derive(ValueEnum, Clone, Debug, PartialEq, Eq)] +pub enum WhichModel { + // ... existing models + NewModel, +} + +// Add to model loading +pub fn load_model( + model_type: &WhichModel, + model_path: &str, + device: &Device, +) -> Result> { + match model_type { + // ... existing models + WhichModel::NewModel => { + Ok(Box::new(newmodel::NewModelGenerate::init(model_path, None, None)?)) + } + } +} +``` + +### Step 4: Update CLI + +Update `src/main.rs` to include the new model in help text and CLI options if needed. + +### Step 5: Add Tests + +Create `tests/test_newmodel.rs`: + +```rust +use anyhow::Result; + +#[test] +fn test_newmodel_generate() -> Result<()> { + let model_path = "path/to/test/model"; + let mut model = aha::models::newmodel::NewModelGenerate::init(model_path, None, None)?; + + let result = model.generate("Test prompt")?; + assert!(!result.is_empty()); + + Ok(()) +} +``` + +### Step 6: Update Documentation + +Update the following files: +- `README.md` - Add model to supported models list +- `docs/cli.md` - Add model ID to model list +- `docs/api.md` - Add model to supported models section +- `CHANGELOG.md` - Add entry for new model + +## Testing + +### Unit Tests + +```bash +# Run all tests +cargo test + +# Run specific test +cargo test test_newmodel_generate + +# Run with output +cargo test -- --nocapture + +# Run tests in parallel +cargo test -- --test-threads=4 +``` + +### Integration Tests + +Integration tests are located in `tests/`: + +```bash +# Run specific integration test +cargo test --test test_qwen3vl +``` + +### Manual Testing + +Test the CLI and API manually: + +```bash +# Test CLI +cargo run -- -m newmodel + +# Test API +cargo run -- -m newmodel -p 8080 +curl http://127.0.0.1:8080/chat/completions -d '{...}' +``` + +### Model-Specific Tests + +Each model should have tests covering: +- Model loading +- Basic inference +- Edge cases +- Error handling + +## Code Style + +### Formatting + +```bash +# Format all code +cargo fmt + +# Check formatting without making changes +cargo fmt --check +``` + +### Linting + +```bash +# Run clippy +cargo clippy + +# Fix clippy warnings +cargo clippy --fix +``` + +### Conventions + +- Use 4 spaces for indentation +- Prefer `Result` over `Option` for errors +- Use `anyhow::Result` for application errors +- Document public APIs with rustdoc comments +- Keep functions focused and small +- Use meaningful variable names + +### Documentation + +```rust +/// Generates text using the NewModel. +/// +/// # Arguments +/// +/// * `prompt` - The input prompt text +/// * `max_tokens` - Maximum tokens to generate +/// +/// # Returns +/// +/// Generated text as a String +/// +/// # Examples +/// +/// ```no_run +/// let result = model.generate("Hello")?; +/// ``` +pub fn generate(&mut self, prompt: &str) -> Result { + // Implementation +} +``` + +## Submitting Contributions + +### Pull Request Process + +1. Fork the repository +2. Create a feature branch: + ```bash + git checkout -b feature/new-model + ``` +3. Make your changes +4. Add tests +5. Update documentation +6. Ensure all tests pass: + ```bash + cargo test + cargo clippy + cargo fmt --check + ``` +7. Commit and push: + ```bash + git commit -m "Add NewModel support" + git push origin feature/new-model + ``` +8. Create a pull request on GitHub + +### Pull Request Checklist + +- [ ] Tests added/updated +- [ ] Documentation updated +- [ ] CHANGELOG.md updated +- [ ] Code formatted (`cargo fmt`) +- [ ] Clippy warnings fixed (`cargo clippy`) +- [ ] All tests pass (`cargo test`) +- [ ] Commit messages follow conventions + +### Commit Message Conventions + +Follow conventional commits: + +``` +feat: add NewModel support +fix: correct tensor dimensions in Qwen3VL +docs: update installation guide +test: add integration test for VoxCPM +refactor: simplify model loading logic +perf: improve inference speed by 20% +``` + +## Release Process + +### Version Bump + +Update `Cargo.toml`: + +```toml +[package] +version = "0.1.9" # Bump version +``` + +### Update Changelog + +Add entry to `CHANGELOG.md`: + +```markdown +## [0.1.9] - 2025-02-06 + +### Added +- NewModel support + +### Fixed +- Memory leak in image processing + +### Changed +- Improved error messages +``` + +### Create Release + +```bash +# Tag the release +git tag -a v0.1.9 -m "Release v0.1.9" +git push origin v0.1.9 + +# Build release artifacts +cargo build --release + +# Publish to crates.io (if applicable) +cargo publish +``` + +## Debugging + +### Logging + +Enable debug logging: + +```bash +RUST_LOG=debug cargo run -- -m qwen3-0.6b +``` + +Set specific module logging: + +```bash +RUST_LOG=aha::models::qwen3vl=debug cargo run -- -m qwen3-0.6b +``` + +### Debugging Tests + +```bash +# Print test output +cargo test -- --nocapture + +# Show backtrace +RUST_BACKTRACE=1 cargo test +``` + +### Common Issues + +#### Build Errors + +- **Linking errors**: Install required system dependencies +- **CUDA errors**: Ensure CUDA toolkit is installed +- **Metal errors**: Check you're on Apple Silicon + +#### Runtime Errors + +- **Model not found**: Check model path and download +- **Out of memory**: Use smaller model or enable GPU +- **Slow inference**: Enable GPU acceleration + +## Resources + +- [Candle Documentation](https://github.com/huggingface/candle) +- [Rust Guidelines](https://rust-lang.github.io/api-guidelines/) +- [Conventional Commits](https://www.conventionalcommits.org/) + +## See Also + +- [Architecture & Design](./concepts.md) - How AHA works +- [Installation](./installation.md) - Setup guide +- [API Reference](./api.md) - API documentation diff --git a/docs/development.zh-CN.md b/docs/development.zh-CN.md new file mode 100644 index 0000000..6170b47 --- /dev/null +++ b/docs/development.zh-CN.md @@ -0,0 +1,601 @@ +# 开发指南 + +本指南涵盖为 AHA 做贡献，包括开发设置、添加新模型和提交贡献。 + +## 目录 + +- [开发设置](#开发设置) +- [项目结构](#项目结构) +- [添加新模型](#添加新模型) +- [测试](#测试) +- [代码风格](#代码风格) +- [提交贡献](#提交贡献) +- [发布流程](#发布流程) + +## 开发设置 + +### 前置要求 + +- Rust 1.85+ (edition 2024) +- Git +- 可选：CUDA Toolkit、Metal、FFmpeg（用于功能开发） + +### 克隆和构建 + +```bash +# 克隆仓库 +git clone https://github.com/jhqxxx/aha.git +cd aha + +# 调试模式构建（编译更快） +cargo build + +# 发布模式构建（性能更好） +cargo build --release + +# 使用功能构建 +cargo build --features "cuda,flash-attn" +cargo build --features "metal" +cargo build --features "ffmpeg" +``` + +### 开发工作流程 + +```bash +# 运行 CLI +cargo run -- -m qwen3-0.6b + +# 运行测试 +cargo test + +# 运行特定测试 +cargo test test_qwen3vl_generate + +# 启用日志运行 +RUST_LOG=debug cargo run -- -m qwen3-0.6b + +# 检查代码而不构建 +cargo check + +# 格式化代码 +cargo fmt + +# 运行 linter +cargo clippy +``` + +## 项目结构 + +``` +aha/ +├── Cargo.toml # 构建配置 +├── src/ +│ ├── main.rs # CLI 入口点 +│ ├── lib.rs # 库导出 +│ ├── api.rs # HTTP API 处理程序 +│ ├── exec/ # CLI 命令实现 +│ │ ├── mod.rs +│ │ ├── cli.rs +│ │ ├── serv.rs +│ │ ├── download.rs +│ │ └── run.rs +│ ├── models/ # 模型实现 +│ │ ├── mod.rs # 模型工厂和导出 +│ │ ├── common/ # 共享模型工具 +│ │ │ ├── config.rs +│ │ │ └── mod.rs +│ │ ├── qwen3vl/ # Qwen3-VL 模型 +│ │ │ ├── config.rs +│ │ │ ├── model.rs +│ │ │ ├── generate.rs +│ │ │ ├── processor.rs +│ │ │ └── mod.rs +│ │ ├── voxcpm/ # VoxCPM 模型 +│ │ └── ... # 其他模型 +│ ├── tokenizer/ # 分词工具 +│ ├── chat_template/ # 聊天模板处理 +│ ├── position_embed/ # 位置编码 +│ └── utils/ # 工具函数 +│ ├── audio_utils.rs +│ ├── image_utils.rs +│ ├── download.rs +│ └── common.rs +├── tests/ # 集成测试 +│ ├── test_qwen2_5vl.rs +│ ├── test_qwen3vl.rs +│ └── ... +├── examples/ # 示例代码 +└── docs/ # 文档 +``` + +## 添加新模型 + +本节提供了向 AHA 添加新模型的分步指南。 + +### 步骤 1：创建模型目录 + +在 `src/models/` 下创建新目录： + +```bash +mkdir -p src/models/newmodel +``` + +### 步骤 2：实现模型文件 + +在 `src/models/newmodel/` 中创建以下文件： + +#### config.rs + +定义模型配置： + +```rust +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct NewModelConfig { + pub vocab_size: usize, + pub hidden_size: usize, + pub num_hidden_layers: usize, + pub num_attention_heads: usize, + // ... 其他配置字段 +} + +impl Default for NewModelConfig { + fn default() -> Self { + Self { + vocab_size: 32000, + hidden_size: 2048, + num_hidden_layers: 24, + num_attention_heads: 32, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerationConfig { + pub temperature: f64, + pub top_p: f64, + pub max_tokens: usize, +} + +impl Default for GenerationConfig { + fn default() -> Self { + Self { + temperature: 1.0, + top_p: 1.0, + max_tokens: 100, + } + } +} +``` + +#### model.rs + +实现模型架构： + +```rust +use candle::{Result, Tensor}; +use candle_nn::{Module, VarBuilder}; + +use super::config::NewModelConfig; + +pub struct NewModel { + // 定义模型组件 + config: NewModelConfig, +} + +impl NewModel { + pub fn load(vb: VarBuilder, config: &NewModelConfig) -> Result { + // 加载模型权重 + Ok(Self { + config: config.clone(), + }) + } + + pub fn forward(&self, input_ids: &Tensor) -> Result { + // 实现前向传播 + Ok(input_ids.clone()) + } +} +``` + +#### generate.rs + +实现 GenerateModel trait： + +```rust +use std::path::Path; +use anyhow::Result; +use candle::Device; + +use super::config::{GenerationConfig, NewModelConfig}; +use aha::models::common::GenerateModel; + +pub struct NewModelGenerate { + // 定义生成状态 + model: super::model::NewModel, + config: GenerationConfig, + device: Device, +} + +impl GenerateModel for NewModelGenerate { + type Config = NewModelConfig; + type GenerateConfig = GenerationConfig; + + fn init( + model_path: &str, + config: Option, + generate_config: Option, + ) -> Result { + // 从磁盘加载模型 + let device = Device::Cpu; + let config = config.unwrap_or_default(); + let generate_config = generate_config.unwrap_or_default(); + + // 加载权重 + let model_path = Path::new(model_path); + // ... 加载模型实现 + + Ok(Self { + model: todo!(), + config: generate_config, + device, + }) + } + + fn generate(&mut self, prompt: &str) -> Result { + // 分词提示 + // 运行推理 + // 解码输出 + Ok(prompt.to_string()) + } +} +``` + +#### processor.rs（可选） + +对于具有多模态输入的复杂模型： + +```rust +use anyhow::Result; + +pub struct NewModelProcessor { + // 处理状态 +} + +impl NewModelProcessor { + pub fn new() -> Self { + Self {} + } + + pub fn process_text(&self, text: &str) -> Result> { + // 处理文本输入 + Ok(vec![text.to_string()]) + } + + pub fn process_image(&self, image_path: &str) -> Result { + // 处理图像输入 + todo!() + } +} +``` + +#### mod.rs + +导出模型： + +```rust +mod config; +mod model; +mod generate; +pub mod processor; + +pub use generate::NewModelGenerate; +pub use config::{GenerationConfig, NewModelConfig}; +``` + +### 步骤 3：注册模型 + +更新 `src/models/mod.rs`： + +```rust +// 添加到导入 +pub mod newmodel; + +// 添加到 WhichModel 枚举 +#[derive(ValueEnum, Clone, Debug, PartialEq, Eq)] +pub enum WhichModel { + // ... 现有模型 + NewModel, +} + +// 添加到模型加载 +pub fn load_model( + model_type: &WhichModel, + model_path: &str, + device: &Device, +) -> Result> { + match model_type { + // ... 现有模型 + WhichModel::NewModel => { + Ok(Box::new(newmodel::NewModelGenerate::init(model_path, None, None)?)) + } + } +} +``` + +### 步骤 4：更新 CLI + +更新 `src/main.rs` 以在帮助文本和 CLI 选项中包含新模型（如果需要）。 + +### 步骤 5：添加测试 + +创建 `tests/test_newmodel.rs`： + +```rust +use anyhow::Result; + +#[test] +fn test_newmodel_generate() -> Result<()> { + let model_path = "path/to/test/model"; + let mut model = aha::models::newmodel::NewModelGenerate::init(model_path, None, None)?; + + let result = model.generate("测试提示")?; + assert!(!result.is_empty()); + + Ok(()) +} +``` + +### 步骤 6：更新文档 + +更新以下文件： +- `README.md` - 将模型添加到支持的模型列表 +- `docs/cli.md` - 将模型 ID 添加到模型列表 +- `docs/api.md` - 将模型添加到支持的模型部分 +- `CHANGELOG.md` - 为新模型添加条目 + +## 测试 + +### 单元测试 + +```bash +# 运行所有测试 +cargo test + +# 运行特定测试 +cargo test test_newmodel_generate + +# 启用输出运行 +cargo test -- --nocapture + +# 并行运行测试 +cargo test -- --test-threads=4 +``` + +### 集成测试 + +集成测试位于 `tests/` 中： + +```bash +# 运行特定集成测试 +cargo test --test test_qwen3vl +``` + +### 手动测试 + +手动测试 CLI 和 API： + +```bash +# 测试 CLI +cargo run -- -m newmodel + +# 测试 API +cargo run -- -m newmodel -p 8080 +curl http://127.0.0.1:8080/chat/completions -d '{...}' +``` + +### 模型特定测试 + +每个模型都应该有涵盖以下内容的测试： +- 模型加载 +- 基本推理 +- 边缘情况 +- 错误处理 + +## 代码风格 + +### 格式化 + +```bash +# 格式化所有代码 +cargo fmt + +# 检查格式化而不进行更改 +cargo fmt --check +``` + +### Linting + +```bash +# 运行 clippy +cargo clippy + +# 修复 clippy 警告 +cargo clippy --fix +``` + +### 约定 + +- 使用 4 个空格缩进 +- 对于错误，优先使用 `Result` 而不是 `Option` +- 对于应用程序错误使用 `anyhow::Result` +- 使用 rustdoc 注释记录公共 API +- 保持函数专注和小型 +- 使用有意义的变量名 + +### 文档 + +```rust +/// 使用 NewModel 生成文本。 +/// +/// # 参数 +/// +/// * `prompt` - 输入提示文本 +/// * `max_tokens` - 要生成的最大令牌数 +/// +/// # 返回 +/// +/// 生成的文本作为字符串 +/// +/// # 示例 +/// +/// ```no_run +/// let result = model.generate("你好")?; +/// ``` +pub fn generate(&mut self, prompt: &str) -> Result { + // 实现 +} +``` + +## 提交贡献 + +### Pull Request 流程 + +1. Fork 仓库 +2. 创建功能分支： + ```bash + git checkout -b feature/new-model + ``` +3. 进行更改 +4. 添加测试 +5. 更新文档 +6. 确保所有测试通过： + ```bash + cargo test + cargo clippy + cargo fmt --check + ``` +7. 提交并推送： + ```bash + git commit -m "添加 NewModel 支持" + git push origin feature/new-model + ``` +8. 在 GitHub 上创建 pull request + +### Pull Request 清单 + +- [ ] 已添加/更新测试 +- [ ] 已更新文档 +- [ ] 已更新 CHANGELOG.md +- [ ] 代码已格式化（`cargo fmt`） +- [ ] 已修复 Clippy 警告（`cargo clippy`） +- [ ] 所有测试通过（`cargo test`） +- [ ] 提交消息遵循约定 + +### 提交消息约定 + +遵循约定式提交： + +``` +feat: 添加 NewModel 支持 +fix: 修正 Qwen3VL 中的张量维度 +docs: 更新安装指南 +test: 添加 VoxCPM 集成测试 +refactor: 简化模型加载逻辑 +perf: 将推理速度提高 20% +``` + +## 发布流程 + +### 版本升级 + +更新 `Cargo.toml`： + +```toml +[package] +version = "0.1.9" # 升级版本 +``` + +### 更新变更日志 + +将条目添加到 `CHANGELOG.md`： + +```markdown +## [0.1.9] - 2025-02-06 + +### 新增 +- NewModel 支持 + +### 修复 +- 图像处理中的内存泄漏 + +### 更改 +- 改进错误消息 +``` + +### 创建发布 + +```bash +# 标记发布 +git tag -a v0.1.9 -m "Release v0.1.9" +git push origin v0.1.9 + +# 构建发布产物 +cargo build --release + +# 发布到 crates.io（如果适用） +cargo publish +``` + +## 调试 + +### 日志 + +启用调试日志： + +```bash +RUST_LOG=debug cargo run -- -m qwen3-0.6b +``` + +设置特定模块日志： + +```bash +RUST_LOG=aha::models::qwen3vl=debug cargo run -- -m qwen3-0.6b +``` + +### 调试测试 + +```bash +# 打印测试输出 +cargo test -- --nocapture + +# 显示回溯 +RUST_BACKTRACE=1 cargo test +``` + +### 常见问题 + +#### 构建错误 + +- **链接错误**：安装所需的系统依赖 +- **CUDA 错误**：确保已安装 CUDA toolkit +- **Metal 错误**：检查您是否在 Apple Silicon 上 + +#### 运行时错误 + +- **未找到模型**：检查模型路径和下载 +- **内存不足**：使用更小的模型或启用 GPU +- **推理速度慢**：启用 GPU 加速 + +## 资源 + +- [Candle 文档](https://github.com/huggingface/candle) +- [Rust 指南](https://rust-lang.github.io/api-guidelines/) +- [约定式提交](https://www.conventionalcommits.org/) + +## 另见 + +- [架构与设计](./concepts.zh-CN.md) - AHA 的工作原理 +- [安装](./installation.zh-CN.md) - 设置指南 +- [API 参考](./api.zh-CN.md) - API 文档 diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..921752d --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,376 @@ +# Getting Started + +Welcome to AHA! This guide will help you get up and running quickly. + +## Quick Start (5 Minutes) + +### 1. Check Available Models + +```bash +aha list +``` + +### 2. Download Your First Model + +```bash +# Download a small text model to start +aha download -m qwen3-0.6b +``` + +### 3. Start the Service + +```bash +# Start the HTTP API server +aha cli -m qwen3-0.6b +``` + +The service will start on `http://127.0.0.1:10100` + +### 4. Make Your First API Call + +In a new terminal: + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "Hello, AHA!"} + ] + }' +``` + +## Basic Concepts + +### What is AHA? + +AHA is a local AI inference engine that: +- Runs models on your machine (no cloud API) +- Supports multiple model types (text, vision, audio, OCR, ASR) +- Provides an OpenAI-compatible API +- Works offline once models are downloaded + +### Model Categories + +| Category | Description | Example Models | +|----------|-------------|----------------| +| **Text** | Text generation and chat | Qwen3, MiniCPM4 | +| **Vision** | Image understanding | Qwen2.5VL, Qwen3VL | +| **OCR** | Text extraction from images | DeepSeek-OCR, Hunyuan-OCR | +| **ASR** | Speech-to-text | GLM-ASR, Fun-ASR, Qwen3-ASR | +| **Audio** | Text-to-speech | VoxCPM, VoxCPM1.5 | +| **Image** | Image processing | RMBG2.0 (background removal) | + +### CLI Commands + +| Command | Purpose | +|---------|---------| +| `aha cli` | Download model and start service | +| `aha serv` | Start service with existing model | +| `aha download` | Download model only | +| `aha run` | Direct inference without server | +| `aha list` | List available models | + +## Common Workflows + +### Text Generation + +```bash +# Start the service +aha cli -m qwen3-0.6b + +# In another terminal, make a request +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Explain quantum computing in simple terms."} + ], + "max_tokens": 200, + "temperature": 0.7 + }' +``` + +### Vision Understanding + +```bash +# Start a vision model +aha cli -m qwen3vl-2b + +# Analyze an image +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3vl-2b", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image in detail."}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] + } + ] + }' +``` + +### OCR (Text Extraction) + +```bash +# Start an OCR model +aha cli -m deepseek-ocr + +# Extract text from an image +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ocr", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Extract all text from this image."}, + {"type": "image_url", "image_url": {"url": "file:///path/to/document.jpg"}} + ] + } + ] + }' +``` + +### Speech Recognition (ASR) + +```bash +# Start an ASR model +aha cli -m glm-asr-nano-2512 + +# Transcribe audio +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "glm-asr-nano-2512", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Transcribe this audio."}, + {"type": "audio_url", "audio_url": {"url": "file:///path/to/audio.wav"}} + ] + } + ] + }' +``` + +### Text-to-Speech + +```bash +# Start a TTS model +aha cli -m voxcpm1.5 + +# Generate speech +curl http://127.0.0.1:10100/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "voxcpm1.5", + "input": "Hello, this is AHA speaking.", + "voice": "default" + }' \ + --output speech.wav +``` + +### Background Removal + +```bash +# Start RMBG2.0 model +aha cli -m rmbg2.0 + +# Remove background from image +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "file:///path/to/photo.png" + }' \ + --output no-background.png +``` + +### Direct Inference (Without Server) + +```bash +# Run inference directly without starting HTTP server +aha run -m qwen3-0.6b \ + -i "Write a haiku about AI" \ + --weight-path ~/.aha/qwen3-0.6b +``` + +## Configuration Options + +### Change Port + +```bash +# Use port 8080 instead of default 10100 +aha cli -m qwen3-0.6b -p 8080 +``` + +### Bind to All Interfaces + +```bash +# Allow external access (use with caution) +aha cli -m qwen3-0.6b -a 0.0.0.0 -p 8080 +``` + +### Use Local Model + +```bash +# Skip download, use existing model +aha serv -m qwen3-0.6b \ + --weight-path /path/to/model \ + -p 8080 +``` + +### Custom Save Directory + +```bash +# Download model to specific directory +aha download -m qwen3vl-2b -s /data/models +``` + +## Streaming Responses + +For real-time responses, use streaming: + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "Tell me a story"} + ], + "stream": true + }' +``` + +## Model Selection Guide + +### For Text Generation +- **qwen3-0.6b**: Fast, lightweight (~1.2 GB) +- **minicpm4-0.5b**: Small, efficient (~1 GB) + +### For Vision Tasks +- **qwen3vl-2b**: Balanced performance (~4 GB) +- **qwen3vl-8b**: Better quality (~16 GB) + +### For OCR +- **deepseek-ocr**: General purpose +- **hunyuan-ocr**: Good for Chinese text +- **paddleocr-vl**: Lightweight option + +### For Speech Recognition +- **glm-asr-nano-2512**: Fast, accurate +- **fun-asr-nano-2512**: Good for Chinese +- **qwen3asr-0.6b**: Lightweight + +### For Text-to-Speech +- **voxcpm1.5**: High quality Chinese + +### For Background Removal +- **rmbg2.0**: State-of-the-art results + +## Tips & Best Practices + +### 1. Start Small + +Begin with smaller models to understand the workflow: +```bash +aha download -m qwen3-0.6b +``` + +### 2. Use GPU Acceleration + +Build with GPU support for better performance: +```bash +# NVIDIA GPUs +cargo build --release --features cuda + +# Apple Silicon +cargo build --release --features metal +``` + +### 3. Pre-download Models + +Download models when you have good internet: +```bash +aha download -m qwen3vl-2b +``` + +Then use them later without internet: +```bash +aha serv -m qwen3vl-2b --weight-path ~/.aha/qwen3vl-2b +``` + +### 4. Manage Disk Space + +Models are stored in `~/.aha/` by default. Clean up if needed: +```bash +# Check disk usage +du -sh ~/.aha/* + +# Remove old models +rm -rf ~/.aha/old-model-name +``` + +### 5. Monitor Resources + +For large models, monitor your resources: +```bash +# Linux +htop +nvidia-smi # For NVIDIA GPUs + +# macOS +Activity Monitor +``` + +## Troubleshooting + +### Port Already in Use + +```bash +# Use a different port +aha cli -m qwen3-0.6b -p 8080 +``` + +### Model Download Failed + +```bash +# Retry with more attempts +aha download -m qwen3vl-2b --download-retries 5 +``` + +### Out of Memory + +```bash +# Use a smaller model +aha cli -m qwen3-0.6b +``` + +## Next Steps + +1. Explore the [API Reference](./api.md) for detailed endpoint documentation +2. Read the [CLI Reference](./cli.md) for all command options +3. Check [Architecture & Design](./concepts.md) to understand how AHA works +4. See [Development](./development.md) if you want to contribute + +## Examples Repository + +For more examples, check out the [examples](../examples/) directory in the repository. + +## See Also + +- [API Reference](./api.md) - Complete API documentation +- [CLI Reference](./cli.md) - Command-line reference +- [Installation Guide](./installation.md) - Installation instructions +- [Development Guide](./development.md) - Contributing guide diff --git a/docs/getting-started.zh-CN.md b/docs/getting-started.zh-CN.md new file mode 100644 index 0000000..ce9856e --- /dev/null +++ b/docs/getting-started.zh-CN.md @@ -0,0 +1,376 @@ +# 快速入门 + +欢迎使用 AHA！本指南将帮助您快速上手。 + +## 快速开始（5 分钟） + +### 1. 查看可用模型 + +```bash +aha list +``` + +### 2. 下载第一个模型 + +```bash +# 下载一个小型文本模型开始 +aha download -m qwen3-0.6b +``` + +### 3. 启动服务 + +```bash +# 启动 HTTP API 服务器 +aha cli -m qwen3-0.6b +``` + +服务将在 `http://127.0.0.1:10100` 上启动 + +### 4. 发起第一个 API 调用 + +在新终端中： + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "你好，AHA！"} + ] + }' +``` + +## 基本概念 + +### 什么是 AHA？ + +AHA 是一个本地 AI 推理引擎，具有以下特点： +- 在您的机器上运行模型（无需云 API） +- 支持多种模型类型（文本、视觉、音频、OCR、ASR） +- 提供 OpenAI 兼容的 API +- 模型下载后可离线工作 + +### 模型类别 + +| 类别 | 描述 | 示例模型 | +|------|------|----------| +| **文本** | 文本生成和对话 | Qwen3、MiniCPM4 | +| **视觉** | 图像理解 | Qwen2.5VL、Qwen3VL | +| **OCR** | 从图像中提取文本 | DeepSeek-OCR、Hunyuan-OCR | +| **ASR** | 语音转文本 | GLM-ASR、Fun-ASR、Qwen3-ASR | +| **音频** | 文本转语音 | VoxCPM、VoxCPM1.5 | +| **图像** | 图像处理 | RMBG2.0（背景移除） | + +### CLI 命令 + +| 命令 | 用途 | +|------|------| +| `aha cli` | 下载模型并启动服务 | +| `aha serv` | 使用现有模型启动服务 | +| `aha download` | 仅下载模型 | +| `aha run` | 直接推理，无需服务器 | +| `aha list` | 列出可用模型 | + +## 常见工作流程 + +### 文本生成 + +```bash +# 启动服务 +aha cli -m qwen3-0.6b + +# 在另一个终端中，发起请求 +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "system", "content": "你是一个有用的助手。"}, + {"role": "user", "content": "用简单的术语解释量子计算。"} + ], + "max_tokens": 200, + "temperature": 0.7 + }' +``` + +### 视觉理解 + +```bash +# 启动视觉模型 +aha cli -m qwen3vl-2b + +# 分析图像 +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3vl-2b", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "详细描述这张图片。"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/image.jpg"}} + ] + } + ] + }' +``` + +### OCR（文本提取） + +```bash +# 启动 OCR 模型 +aha cli -m deepseek-ocr + +# 从图像中提取文本 +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ocr", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "提取这张图片中的所有文本。"}, + {"type": "image_url", "image_url": {"url": "file:///path/to/document.jpg"}} + ] + } + ] + }' +``` + +### 语音识别（ASR） + +```bash +# 启动 ASR 模型 +aha cli -m glm-asr-nano-2512 + +# 转写音频 +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "glm-asr-nano-2512", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "转写这段音频。"}, + {"type": "audio_url", "audio_url": {"url": "file:///path/to/audio.wav"}} + ] + } + ] + }' +``` + +### 文本转语音 + +```bash +# 启动 TTS 模型 +aha cli -m voxcpm1.5 + +# 生成语音 +curl http://127.0.0.1:10100/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "voxcpm1.5", + "input": "你好，这是 AHA 在说话。", + "voice": "default" + }' \ + --output speech.wav +``` + +### 背景移除 + +```bash +# 启动 RMBG2.0 模型 +aha cli -m rmbg2.0 + +# 移除图像背景 +curl http://127.0.0.1:10100/images/remove_background \ + -H "Content-Type: application/json" \ + -d '{ + "model": "rmbg2.0", + "image": "file:///path/to/photo.png" + }' \ + --output no-background.png +``` + +### 直接推理（无需服务器） + +```bash +# 直接运行推理，无需启动 HTTP 服务器 +aha run -m qwen3-0.6b \ + -i "写一首关于 AI 的俳句" \ + --weight-path ~/.aha/qwen3-0.6b +``` + +## 配置选项 + +### 更改端口 + +```bash +# 使用端口 8080 而不是默认的 10100 +aha cli -m qwen3-0.6b -p 8080 +``` + +### 绑定到所有接口 + +```bash +# 允许外部访问（请谨慎使用） +aha cli -m qwen3-0.6b -a 0.0.0.0 -p 8080 +``` + +### 使用本地模型 + +```bash +# 跳过下载，使用现有模型 +aha serv -m qwen3-0.6b \ + --weight-path /path/to/model \ + -p 8080 +``` + +### 自定义保存目录 + +```bash +# 将模型下载到特定目录 +aha download -m qwen3vl-2b -s /data/models +``` + +## 流式响应 + +对于实时响应，使用流式传输： + +```bash +curl http://127.0.0.1:10100/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-0.6b", + "messages": [ + {"role": "user", "content": "给我讲个故事"} + ], + "stream": true + }' +``` + +## 模型选择指南 + +### 文本生成 +- **qwen3-0.6b**：快速、轻量级（~1.2 GB） +- **minicpm4-0.5b**：小型、高效（~1 GB） + +### 视觉任务 +- **qwen3vl-2b**：平衡性能（~4 GB） +- **qwen3vl-8b**：更高质量（~16 GB） + +### OCR +- **deepseek-ocr**：通用 +- **hunyuan-ocr**：适合中文文本 +- **paddleocr-vl**：轻量级选项 + +### 语音识别 +- **glm-asr-nano-2512**：快速、准确 +- **fun-asr-nano-2512**：适合中文 +- **qwen3asr-0.6b**：轻量级 + +### 文本转语音 +- **voxcpm1.5**：高质量中文 + +### 背景移除 +- **rmbg2.0**：最先进的结果 + +## 提示与最佳实践 + +### 1. 从小开始 + +从小型模型开始了解工作流程： +```bash +aha download -m qwen3-0.6b +``` + +### 2. 使用 GPU 加速 + +使用 GPU 支持构建以获得更好的性能： +```bash +# NVIDIA GPU +cargo build --release --features cuda + +# Apple Silicon +cargo build --release --features metal +``` + +### 3. 预先下载模型 + +在网络良好时下载模型： +```bash +aha download -m qwen3vl-2b +``` + +稍后在没有网络的情况下使用： +```bash +aha serv -m qwen3vl-2b --weight-path ~/.aha/qwen3vl-2b +``` + +### 4. 管理磁盘空间 + +模型默认存储在 `~/.aha/` 中。如需要，清理： +```bash +# 检查磁盘使用情况 +du -sh ~/.aha/* + +# 删除旧模型 +rm -rf ~/.aha/old-model-name +``` + +### 5. 监控资源 + +对于大型模型，监控您的资源： +```bash +# Linux +htop +nvidia-smi # 对于 NVIDIA GPU + +# macOS +活动监视器 +``` + +## 故障排除 + +### 端口已被占用 + +```bash +# 使用不同的端口 +aha cli -m qwen3-0.6b -p 8080 +``` + +### 模型下载失败 + +```bash +# 重试更多次数 +aha download -m qwen3vl-2b --download-retries 5 +``` + +### 内存不足 + +```bash +# 使用更小的模型 +aha cli -m qwen3-0.6b +``` + +## 后续步骤 + +1. 探索 [API 参考](./api.zh-CN.md) 了解详细的端点文档 +2. 阅读 [CLI 参考](./cli.zh-CN.md) 了解所有命令选项 +3. 查看 [架构与设计](./concepts.zh-CN.md) 了解 AHA 的工作原理 +4. 如果您想贡献，请参阅 [开发指南](./development.zh-CN.md) + +## 示例仓库 + +更多示例，请查看仓库中的 [examples](../examples/) 目录。 + +## 另见 + +- [API 参考](./api.zh-CN.md) - 完整的 API 文档 +- [CLI 参考](./cli.zh-CN.md) - 命令行参考 +- [安装指南](./installation.zh-CN.md) - 安装说明 +- [开发指南](./development.zh-CN.md) - 贡献指南 diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..3aff0cf --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,420 @@ +# Installation Guide + +This guide covers installing and setting up AHA on your system. + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Installation Methods](#installation-methods) +- [Platform-Specific Instructions](#platform-specific-instructions) +- [Feature Flags](#feature-flags) +- [Verification](#verification) +- [Troubleshooting](#troubleshooting) + +## Prerequisites + +### Required + +- **Rust toolchain**: Rust 1.85 or later (edition 2024) + ```bash + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + ``` + +- **Git**: For cloning the repository + ```bash + # Ubuntu/Debian + sudo apt-get install git + + # macOS + brew install git + + # Windows + # Download from https://git-scm.com/download/win + ``` + +### Optional (for FFmpeg feature) + +- **FFmpeg development libraries**: Required for audio/video processing + +## Installation Methods + +### Method 1: Build from Source + +Clone the repository and build: + +```bash +git clone https://github.com/jhqxxx/aha.git +cd aha + +# Build release version +cargo build --release + +# The binary will be at target/release/aha +``` + +### Method 2: Install from Crates.io (when available) + +```bash +cargo install aha +``` + +### Method 3: Install with Features + +Build with specific features enabled: + +```bash +# With CUDA support (NVIDIA GPUs) +cargo build --release --features cuda + +# With Metal support (Apple Silicon) +cargo build --release --features metal + +# With Flash Attention +cargo build --release --features "cuda,flash-attn" + +# With FFmpeg support +cargo build --release --features ffmpeg +``` + +## Platform-Specific Instructions + +### Linux + +#### Ubuntu/Debian + +```bash +# Install build dependencies +sudo apt-get update +sudo apt-get install -y build-essential pkg-config git clang + +# For FFmpeg feature +sudo apt-get install -y ffmpeg libavutil-dev libavcodec-dev \ + libavformat-dev libavfilter-dev libavdevice-dev \ + libswresample-dev libswscale-dev + +# For CUDA support, install CUDA toolkit +# See https://developer.nvidia.com/cuda-downloads +``` + +#### Fedora/RHEL + +```bash +# Install build dependencies +sudo dnf install gcc gcc-c++ make git clang pkg-config + +# For FFmpeg feature +sudo dnf install ffmpeg-devel + +# For CUDA support +sudo dnf install cuda-devel +``` + +### macOS + +#### Apple Silicon (M1/M2/M3/M4) + +```bash +# Install Rust (if not already installed) +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# Install command line tools +xcode-select --install + +# For FFmpeg feature +brew install ffmpeg + +# Build with Metal support for GPU acceleration +cargo build --release --features metal +``` + +#### Intel Mac + +```bash +# Install Rust (if not already installed) +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# Install command line tools +xcode-select --install + +# For FFmpeg feature +brew install ffmpeg + +# For CUDA support (if you have NVIDIA GPU) +# Install CUDA from https://developer.nvidia.com/cuda-downloads +cargo build --release --features cuda +``` + +### Windows + +#### Using MSVC + +```bash +# Install Rust from https://rustup.rs/ +# Install Visual Studio Build Tools from https://visualstudio.microsoft.com/downloads/ + +# For FFmpeg feature +# Download FFmpeg from https://ffmpeg.org/download.html +# Set FFMPEG_DIR environment variable to your FFmpeg installation + +# Build +cargo build --release +``` + +#### Using WSL2 (Recommended) + +```bash +# Follow Linux instructions inside WSL2 +wsl +sudo apt-get update +sudo apt-get install -y build-essential pkg-config git clang +``` + +## Feature Flags + +aha supports several optional features: + +### cuda + +Enables CUDA support for NVIDIA GPU acceleration. + +```bash +cargo build --release --features cuda +``` + +**Requirements**: +- NVIDIA GPU +- CUDA Toolkit 11.0 or later +- cuDNN library + +**Benefits**: +- 10-50x faster inference +- Support for larger models +- Lower CPU usage + +### metal + +Enables Metal support for Apple Silicon GPU acceleration. + +```bash +cargo build --release --features metal +``` + +**Requirements**: +- Apple Silicon (M1/M2/M3/M4) +- macOS 11.0 or later + +**Benefits**: +- 5-20x faster inference +- Lower power consumption +- Support for larger models + +### flash-attn + +Enables Flash Attention for optimized long-sequence processing. + +```bash +cargo build --release --features "cuda,flash-attn" +``` + +**Requirements**: +- CUDA feature enabled +- Supported GPU architecture (compute capability 7.0+) + +**Benefits**: +- Reduced memory usage +- Faster inference for long sequences +- Especially beneficial for vision models + +**Note**: Must be used with `cuda` feature. + +### ffmpeg + +Enables FFmpeg support for audio/video processing. + +```bash +cargo build --release --features ffmpeg +``` + +**Requirements**: +- FFmpeg development libraries +- Platform-specific (see above) + +**Benefits**: +- Extended audio format support (MP3, AAC, etc.) +- Video processing capabilities +- Better audio resampling + +### Combining Features + +You can combine multiple features: + +```bash +# Maximum performance on NVIDIA GPU +cargo build --release --features "cuda,flash-attn" + +# Apple Silicon with audio support +cargo build --release --features "metal,ffmpeg" + +# Everything enabled +cargo build --release --features "cuda,flash-attn,ffmpeg" +``` + +## Verification + +After installation, verify that AHA is working: + +```bash +# Check version +./target/release/aha --version + +# List supported models +./target/release/aha list + +# (Or if installed to PATH) +aha --version +aha list +``` + +Expected output for `aha list`: + +``` +#Supported models: + + Available models: + +Model Name ModelScope ID +----------------------------------------------------------- +minicpm4-0.5b OpenBMB/MiniCPM4-0.5B +qwen2.5vl-3b Qwen/Qwen2.5-VL-3B-Instruct +qwen2.5vl-7b Qwen/Qwen2.5-VL-7B-Instruct +qwen3-0.6b Qwen/Qwen3-0.6B +qwen3asr-0.6b Qwen/Qwen3-ASR-0.6B +qwen3asr-1.7b Qwen/Qwen3-ASR-1.7B +qwen3vl-4b Qwen/Qwen3-VL-2B-Instruct +qwen3vl-4b Qwen/Qwen3-VL-4B-Instruct +qwen3vl-8b Qwen/Qwen3-VL-8B-Instruct +qwen3vl-32b Qwen/Qwen3-VL-32B-Instruct +deepseek-ocr deepseek-ai/DeepSeek-OCR +hunyuan-ocr Tencent-Hunyuan/HunyuanOCR +paddleocr-vl PaddlePaddle/PaddleOCR-VL +rmbg2.0 AI-ModelScope/RMBG-2.0 +voxcpm OpenBMB/VoxCPM-0.5B +voxcpm1.5 OpenBMB/VoxCPM1.5 +glm-asr-nano-2512 ZhipuAI/GLM-ASR-Nano-2512 +fun-asr-nano-2512 FunAudioLLM/Fun-ASR-Nano-2512 + +``` + +## Troubleshooting + +### Build Errors + +#### "error: linking with cc failed" + +This usually indicates missing system dependencies. + +**Solution**: Install required build tools for your platform (see Platform-Specific Instructions). + +#### "error: CUDA not found" + +CUDA feature is enabled but CUDA toolkit is not installed. + +**Solution**: +- Install CUDA toolkit from https://developer.nvidia.com/cuda-downloads +- Or build without CUDA: `cargo build --release` + +#### "error: Metal not available" + +Metal feature is enabled but not on supported hardware. + +**Solution**: +- Ensure you're on Apple Silicon +- Or build without Metal: `cargo build --release` + +### Runtime Errors + +#### "error while loading shared libraries" + +Missing runtime libraries. + +**Solution**: Install required libraries (see Platform-Specific Instructions). + +#### "Out of memory" + +Model is too large for available RAM/VRAM. + +**Solution**: +- Use a smaller model +- Close other applications +- Enable GPU acceleration for better memory efficiency + +#### "Model download failed" + +Network issue or insufficient disk space. + +**Solution**: +- Check internet connection +- Ensure sufficient disk space in `~/.aha/` +- Try again: download will resume if interrupted + +### Performance Issues + +#### Slow inference + +**Solutions**: +1. Enable GPU acceleration: `--features cuda` or `--features metal` +2. Enable Flash Attention: `--features "cuda,flash-attn"` +3. Use a smaller model +4. Check if GPU is being used (should see GPU usage in monitoring tools) + +#### High CPU usage + +**Solutions**: +1. Enable GPU acceleration +2. Reduce batch size +3. Use model with lower precision + +## System Requirements +*Different models require different hardware and software, for reference.* + +### Minimum Requirements + +- **CPU**: x86_64 or ARM64 +- **RAM**: 8 GB (16 GB recommended) +- **Disk**: 10 GB for models (varies by model) +- **OS**: Linux, macOS, or Windows + +### Recommended Requirements + +- **CPU**: Modern multi-core processor +- **RAM**: 32 GB or more +- **GPU**: NVIDIA GPU (with CUDA) or Apple Silicon +- **Disk**: SSD with 50+ GB free space +- **OS**: Linux (Ubuntu 22.04+) or macOS (Monterey+) + +## Model Sizes + +Approximate download sizes for popular models: + +| Model | Size | RAM Usage | +|-------|------|-----------| +| qwen3-0.6b | ~1.2 GB | ~2 GB | +| qwen3vl-2b | ~4 GB | ~6 GB | +| qwen3vl-8b | ~16 GB | ~20 GB | +| qwen3vl-32b | ~64 GB | ~70 GB | + +## Next Steps + +After successful installation: + +1. Read the [Getting Started Guide](./getting-started.md) +2. Download your first model: `aha download -m qwen3-0.6b` +3. Start the service: `aha cli -m qwen3-0.6b` +4. Explore the [API Reference](./api.md) + +## See Also + +- [Getting Started](./getting-started.md) - Quick start guide +- [CLI Reference](./cli.md) - Command-line usage +- [API Reference](./api.md) - REST API documentation +- [Development](./development.md) - Contributing guide diff --git a/docs/installation.zh-CN.md b/docs/installation.zh-CN.md new file mode 100644 index 0000000..03ae3a2 --- /dev/null +++ b/docs/installation.zh-CN.md @@ -0,0 +1,419 @@ +# 安装指南 + +本指南介绍如何在您的系统上安装和设置 AHA。 + +## 目录 + +- [前置要求](#前置要求) +- [安装方法](#安装方法) +- [平台特定说明](#平台特定说明) +- [功能特性](#功能特性) +- [验证安装](#验证安装) +- [故障排除](#故障排除) + +## 前置要求 + +### 必需 + +- **Rust 工具链**：Rust 1.85 或更高版本（edition 2024） + ```bash + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + ``` + +- **Git**：用于克隆仓库 + ```bash + # Ubuntu/Debian + sudo apt-get install git + + # macOS + brew install git + + # Windows + # 从 https://git-scm.com/download/win 下载 + ``` + +### 可选（用于 FFmpeg 功能） + +- **FFmpeg 开发库**：音频/视频处理所需 + +## 安装方法 + +### 方法 1：从源码构建 + +克隆仓库并构建： + +```bash +git clone https://github.com/jhqxxx/aha.git +cd aha + +# 构建发布版本 +cargo build --release + +# 二进制文件位于 target/release/aha +``` + +### 方法 2：从 Crates.io 安装（当可用时） + +```bash +cargo install aha +``` + +### 方法 3：启用特定功能构建 + +使用特定功能构建： + +```bash +# 启用 CUDA 支持（NVIDIA GPU） +cargo build --release --features cuda + +# 启用 Metal 支持（Apple Silicon） +cargo build --release --features metal + +# 启用 Flash Attention +cargo build --release --features "cuda,flash-attn" + +# 启用 FFmpeg 支持 +cargo build --release --features ffmpeg +``` + +## 平台特定说明 + +### Linux + +#### Ubuntu/Debian + +```bash +# 安装构建依赖 +sudo apt-get update +sudo apt-get install -y build-essential pkg-config git clang + +# FFmpeg 功能所需 +sudo apt-get install -y ffmpeg libavutil-dev libavcodec-dev \ + libavformat-dev libavfilter-dev libavdevice-dev \ + libswresample-dev libswscale-dev + +# CUDA 支持，从 https://developer.nvidia.com/cuda-downloads 安装 CUDA toolkit +``` + +#### Fedora/RHEL + +```bash +# 安装构建依赖 +sudo dnf install gcc gcc-c++ make git clang pkg-config + +# FFmpeg 功能所需 +sudo dnf install ffmpeg-devel + +# CUDA 支持 +sudo dnf install cuda-devel +``` + +### macOS + +#### Apple Silicon (M1/M2/M3/M4) + +```bash +# 安装 Rust（如果尚未安装） +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# 安装命令行工具 +xcode-select --install + +# FFmpeg 功能所需 +brew install ffmpeg + +# 启用 Metal 支持以进行 GPU 加速 +cargo build --release --features metal +``` + +#### Intel Mac + +```bash +# 安装 Rust（如果尚未安装） +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# 安装命令行工具 +xcode-select --install + +# FFmpeg 功能所需 +brew install ffmpeg + +# CUDA 支持（如果您有 NVIDIA GPU） +# 从 https://developer.nvidia.com/cuda-downloads 安装 CUDA +cargo build --release --features cuda +``` + +### Windows + +#### 使用 MSVC + +```bash +# 从 https://rustup.rs/ 安装 Rust +# 从 https://visualstudio.microsoft.com/downloads/ 安装 Visual Studio Build Tools + +# FFmpeg 功能 +# 从 https://ffmpeg.org/download.html 下载 FFmpeg +# 设置 FFMPEG_DIR 环境变量指向您的 FFmpeg 安装目录 + +# 构建 +cargo build --release +``` + +#### 使用 WSL2（推荐） + +```bash +# 在 WSL2 中按照 Linux 说明操作 +wsl +sudo apt-get update +sudo apt-get install -y build-essential pkg-config git clang +``` + +## 功能特性 + +AHA 支持多个可选功能： + +### cuda + +启用 NVIDIA GPU 加速的 CUDA 支持。 + +```bash +cargo build --release --features cuda +``` + +**要求**： +- NVIDIA GPU +- CUDA Toolkit 11.0 或更高版本 +- cuDNN 库 + +**优势**： +- 推理速度提升 10-50 倍 +- 支持更大的模型 +- 降低 CPU 使用率 + +### metal + +启用 Apple Silicon GPU 加速的 Metal 支持。 + +```bash +cargo build --release --features metal +``` + +**要求**： +- Apple Silicon (M1/M2/M3/M4) +- macOS 11.0 或更高版本 + +**优势**： +- 推理速度提升 5-20 倍 +- 更低的功耗 +- 支持更大的模型 + +### flash-attn + +启用 Flash Attention 以优化长序列处理。 + +```bash +cargo build --release --features "cuda,flash-attn" +``` + +**要求**： +- 启用 CUDA 功能 +- 支持的 GPU 架构（计算能力 7.0+） + +**优势**： +- 减少内存使用 +- 长序列推理更快 +- 对视觉模型特别有益 + +**注意**：必须与 `cuda` 功能一起使用。 + +### ffmpeg + +启用 FFmpeg 支持以进行音频/视频处理。 + +```bash +cargo build --release --features ffmpeg +``` + +**要求**： +- FFmpeg 开发库 +- 特定平台（见上文） + +**优势**： +- 扩展的音频格式支持（MP3、AAC 等） +- 视频处理能力 +- 更好的音频重采样 + +### 组合功能 + +您可以组合多个功能： + +```bash +# NVIDIA GPU 上的最佳性能 +cargo build --release --features "cuda,flash-attn" + +# 带音频支持的 Apple Silicon +cargo build --release --features "metal,ffmpeg" + +# 启用所有功能 +cargo build --release --features "cuda,flash-attn,ffmpeg" +``` + +## 验证安装 + +安装后，验证 AHA 是否正常工作： + +```bash +# 检查版本 +./target/release/aha --version + +# 列出支持的模型 +./target/release/aha list + +# （如果已安装到 PATH） +aha --version +aha list +``` + +`aha list` 的预期输出： + +```shell +#Supported models: + + Available models: + +Model Name ModelScope ID +----------------------------------------------------------- +minicpm4-0.5b OpenBMB/MiniCPM4-0.5B +qwen2.5vl-3b Qwen/Qwen2.5-VL-3B-Instruct +qwen2.5vl-7b Qwen/Qwen2.5-VL-7B-Instruct +qwen3-0.6b Qwen/Qwen3-0.6B +qwen3asr-0.6b Qwen/Qwen3-ASR-0.6B +qwen3asr-1.7b Qwen/Qwen3-ASR-1.7B +qwen3vl-4b Qwen/Qwen3-VL-2B-Instruct +qwen3vl-4b Qwen/Qwen3-VL-4B-Instruct +qwen3vl-8b Qwen/Qwen3-VL-8B-Instruct +qwen3vl-32b Qwen/Qwen3-VL-32B-Instruct +deepseek-ocr deepseek-ai/DeepSeek-OCR +hunyuan-ocr Tencent-Hunyuan/HunyuanOCR +paddleocr-vl PaddlePaddle/PaddleOCR-VL +rmbg2.0 AI-ModelScope/RMBG-2.0 +voxcpm OpenBMB/VoxCPM-0.5B +voxcpm1.5 OpenBMB/VoxCPM1.5 +glm-asr-nano-2512 ZhipuAI/GLM-ASR-Nano-2512 +fun-asr-nano-2512 FunAudioLLM/Fun-ASR-Nano-2512 + +``` + +## 故障排除 + +### 构建错误 + +#### "error: linking with cc failed" + +通常表示缺少系统依赖。 + +**解决方案**：为您的平台安装所需的构建工具（见平台特定说明）。 + +#### "error: CUDA not found" + +启用了 CUDA 功能但未安装 CUDA toolkit。 + +**解决方案**： +- 从 https://developer.nvidia.com/cuda-downloads 安装 CUDA toolkit +- 或不使用 CUDA 构建：`cargo build --release` + +#### "error: Metal not available" + +启用了 Metal 功能但硬件不支持。 + +**解决方案**： +- 确保您在 Apple Silicon 上 +- 或不使用 Metal 构建：`cargo build --release` + +### 运行时错误 + +#### "error while loading shared libraries" + +缺少运行时库。 + +**解决方案**：安装所需的库（见平台特定说明）。 + +#### "Out of memory" + +模型对于可用 RAM/VRAM 太大。 + +**解决方案**： +- 使用更小的模型 +- 关闭其他应用程序 +- 启用 GPU 加速以获得更好的内存效率 + +#### "Model download failed" + +网络问题或磁盘空间不足。 + +**解决方案**： +- 检查互联网连接 +- 确保 `~/.aha/` 中有足够的磁盘空间 +- 重试：如果中断，下载将恢复 + +### 性能问题 + +#### 推理速度慢 + +**解决方案**： +1. 启用 GPU 加速：`--features cuda` 或 `--features metal` +2. 启用 Flash Attention：`--features "cuda,flash-attn"` +3. 使用更小的模型 +4. 检查是否正在使用 GPU（应在监控工具中看到 GPU 使用率） + +#### CPU 使用率高 + +**解决方案**： +1. 启用 GPU 加速 +2. 减少批处理大小 +3. 使用低精度模型 + +## 系统要求 + +*注：模型不同需求不同的硬件和软件，供参考* +### 最低要求 + +- **CPU**：x86_64 或 ARM64 +- **RAM**：8 GB（推荐 16 GB） +- **磁盘**：10 GB 用于模型（因模型而异） +- **OS**：Linux、macOS 或 Windows + +### 推荐要求 + +- **CPU**：现代多核处理器 +- **RAM**：24 GB 或更多 +- **GPU**：NVIDIA GPU（带 CUDA）或 Apple Silicon +- **磁盘**：具有 50+ GB 可用空间的 SSD +- **OS**：Linux (Ubuntu 22.04+) 或 macOS (Monterey+) + +## 模型大小 + +流行模型的**大致**下载大小： + +| 模型 | 大小 | RAM 使用 | +|------|------|----------| +| qwen3-0.6b | ~1.2 GB | ~2 GB | +| qwen3vl-2b | ~4 GB | ~6 GB | +| qwen3vl-8b | ~16 GB | ~20 GB | +| qwen3vl-32b | ~64 GB | ~70 GB | + +## 后续步骤 + +成功安装后： + +1. 阅读[快速入门指南](./getting-started.zh-CN.md) +2. 下载您的第一个模型：`aha download -m qwen3-0.6b` +3. 启动服务：`aha cli -m qwen3-0.6b` +4. 探索 [API 参考](./api.zh-CN.md) + +## 另见 + +- [快速入门](./getting-started.zh-CN.md) - 快速入门指南 +- [CLI 参考](./cli.zh-CN.md) - 命令行使用 +- [API 参考](./api.zh-CN.md) - REST API 文档 +- [开发指南](./development.zh-CN.md) - 贡献指南 diff --git a/docs/supported-models.md b/docs/supported-models.md new file mode 100644 index 0000000..01765d5 --- /dev/null +++ b/docs/supported-models.md @@ -0,0 +1,92 @@ +# Supported Models + +aha supports a growing collection of state-of-the-art AI models across multiple domains. + +## Text Generation + +| Model | Parameters | Description | Use Case | +|-------|-----------|-------------|----------| +| **Qwen2.5-7B** | 7B | General-purpose LLM | Chat, reasoning, code | +| **Qwen3** | Various | Latest generation | Advanced reasoning | +| **MiniCPM4** | 4B | Efficient lightweight | Edge deployment | + +## Vision & Multimodal + +| Model | Type | Description | Resolution | +|-------|------|-------------|------------| +| **Qwen2.5-VL** | Vision-Language | Image understanding | Up to 1024x1024 | +| **Qwen3-VL** | Vision-Language | Enhanced multimodal | Up to 1536x1536 | +| **MiniCPM-V** | Vision-Language | Lightweight vision | Up to 768x768 | + +## Speech Recognition (ASR) + +| Model | Language | Real-time | Speed | +|-------|----------|-----------|-------| +| **FunASR-Nano** | Chinese/English | Yes | 16x realtime | +| **GLM-ASR-Nano** | Chinese/English | Yes | 32x realtime | + +## OCR + +| Model | Languages | Type | Strength | +|-------|-----------|------|----------| +| **PaddleOCR-VL** | 80+ | Lightweight | General documents | +| **Hunyuan-OCR** | Chinese | Deep learning | Complex layouts | +| **DeepSeek-OCR** | Multi | Scene text | Natural images | + +## Audio Processing + +| Model | Type | Description | +|-------|------|-------------| +| **VoxCPM** | Voice Codec | Neural audio codec | +| **RMBG-2.0** | Background Removal | Voice isolation | + +## Model Formats + +All models are served in optimized ONNX format for: + +- **Cross-platform compatibility** - Windows, macOS, Linux +- **CPU acceleration** - AVX2, NEON, SIMD +- **Edge deployment** - No GPU required +- **Fast inference** - Optimized runtime + +## Model Selection + +aha automatically selects the best model for each task. To override: + +```bash +aha chat "Hello" --model qwen2.5-7b +aha vision --model qwen2.5-vl "Describe this" --image img.jpg +aha asr --model fun-asr-nano audio.wav +``` + +## Model Sources + +Models are sourced from: + +- [Hugging Face](https://huggingface.co) - Primary model hub +- [ModelScope](https://modelscope.cn) - Chinese model hub +- [GitHub Releases](https://github.com) - Backup releases + +## Adding New Models + +See [Development Guide](./development.md) for instructions on adding new model integrations. + +## Model Updates + +Models are regularly updated. Check the [releases](https://github.com/yourusername/aha/releases) for the latest versions. + +## License + +Each model has its own license. Please review the model's license before use in production. + +## Performance Benchmarks + +Approximate inference speeds on CPU (M1 Pro): + +| Model | Task | Tokens/sec | +|-------|------|------------| +| Qwen2.5-7B | Text | 25-35 | +| Qwen2.5-VL | Vision | 20-30 | +| FunASR-Nano | ASR | 200-500x | + +*Benchmarks vary by hardware and input size.* diff --git a/docs/supported-models.zh-CN.md b/docs/supported-models.zh-CN.md new file mode 100644 index 0000000..0827760 --- /dev/null +++ b/docs/supported-models.zh-CN.md @@ -0,0 +1,92 @@ +# 支持的模型 + +aha 支持多个领域的最先进 AI 模型集合。 + +## 文本生成 + +| 模型 | 参数量 | 描述 | 使用场景 | +|------|--------|------|----------| +| **Qwen2.5-7B** | 7B | 通用大语言模型 | 对话、推理、代码 | +| **Qwen3** | 多种 | 最新一代 | 高级推理 | +| **MiniCPM4** | 4B | 高效轻量级 | 边缘部署 | + +## 视觉与多模态 + +| 模型 | 类型 | 描述 | 分辨率 | +|------|------|------|--------| +| **Qwen2.5-VL** | 视觉语言 | 图像理解 | 最高 1024x1024 | +| **Qwen3-VL** | 视觉语言 | 增强多模态 | 最高 1536x1536 | +| **MiniCPM-V** | 视觉语言 | 轻量视觉 | 最高 768x768 | + +## 语音识别 (ASR) + +| 模型 | 语言 | 实时 | 速度 | +|------|------|------|------| +| **FunASR-Nano** | 中/英 | 是 | 16x 实时 | +| **GLM-ASR-Nano** | 中/英 | 是 | 32x 实时 | + +## OCR + +| 模型 | 语言 | 类型 | 优势 | +|------|------|------|------| +| **PaddleOCR-VL** | 80+ | 轻量级 | 通用文档 | +| **Hunyuan-OCR** | 中文 | 深度学习 | 复杂布局 | +| **DeepSeek-OCR** | 多语言 | 场景文字 | 自然图像 | + +## 音频处理 + +| 模型 | 类型 | 描述 | +|------|------|------| +| **VoxCPM** | 语音编解码器 | 神经音频编解码 | +| **RMBG-2.0** | 背景移除 | 语音隔离 | + +## 模型格式 + +所有模型均采用优化的 ONNX 格式，提供： + +- **跨平台兼容** - Windows、macOS、Linux +- **CPU 加速** - AVX2、NEON、SIMD +- **边缘部署** - 无需 GPU +- **快速推理** - 优化运行时 + +## 模型选择 + +aha 会自动为每个任务选择最佳模型。如需覆盖： + +```bash +aha chat "你好" --model qwen2.5-7b +aha vision --model qwen2.5-vl "描述这个" --image img.jpg +aha asr --model fun-asr-nano audio.wav +``` + +## 模型来源 + +模型来源： + +- [Hugging Face](https://huggingface.co) - 主模型中心 +- [ModelScope](https://modelscope.cn) - 中文模型中心 +- [GitHub Releases](https://github.com) - 备份发布 + +## 添加新模型 + +参见 [开发指南](./development.zh-CN.md) 了解添加新模型集成的说明。 + +## 模型更新 + +模型定期更新。查看 [releases](https://github.com/jhqxxx/aha/releases) 获取最新版本。 + +## 许可证 + +每个模型都有自己的许可证。在生产中使用前请查看模型许可证。 + +## 性能基准 + +CPU (M1 Pro) 上的近似推理速度： + +| 模型 | 任务 | Tokens/秒 | +|------|------|-----------| +| Qwen2.5-7B | 文本 | 25-35 | +| Qwen2.5-VL | 视觉 | 20-30 | +| FunASR-Nano | ASR | 200-500x | + +*基准测试因硬件和输入大小而异。* diff --git a/download_and_run.sh b/scripts/download_and_run.sh similarity index 100% rename from download_and_run.sh rename to scripts/download_and_run.sh