jhqxxx
290 lines
7.6 KiB
Markdown
290 lines
7.6 KiB
Markdown
<p align="center">
|
||
<img src="assets/img/logo.png" alt="aha logo" width="120"/>
|
||
</p>
|
||
|
||
<p align="center">
|
||
<a href="https://github.com/jhqxxx/aha/stargazers">
|
||
<img src="https://img.shields.io/github/stars/jhqxxx/aha" alt="GitHub Stars">
|
||
</a>
|
||
<a href="https://github.com/jhqxxx/aha/issues">
|
||
<img src="https://img.shields.io/github/issues/jhqxxx/aha" alt="GitHub Issues">
|
||
</a>
|
||
<a href="https://github.com/jhqxxx/aha/blob/main/LICENSE">
|
||
<img src="https://img.shields.io/github/license/jhqxxx/aha" alt="GitHub License">
|
||
</a>
|
||
</p>
|
||
|
||
<p align="center">
|
||
<a href="README.zh-CN.md">简体中文</a> | <strong>English</strong>
|
||
</p>
|
||
|
||
# aha
|
||
|
||
**Lightweight AI Inference Engine — All-in-one Solution for Text, Vision, Speech, and OCR**
|
||
|
||
aha is a high-performance, cross-platform AI inference engine built with Rust and the Candle framework. It brings state-of-the-art AI models to your local machine—no API keys, no cloud dependencies, just pure, fast AI running directly on your hardware.
|
||
|
||
|
||
### Supported Models
|
||
|
||
| Category | Models |
|
||
|----------|--------|
|
||
| **Text** | Qwen3, MiniCPM4, MiniCPM5, LFM2, LFM2.5 |
|
||
| **Vision** | Qwen2.5-VL, Qwen3-VL, Qwen3.5, <br> LFM2.5-VL, LFM2-VL |
|
||
| **OCR** | DeepSeek-OCR, DeepSeek-OCR-2 , PaddleOCR-VL <br> PaddleOCR-VL1.5, Hunyuan-OCR, GLM-OCR |
|
||
| **ASR** | GLM-ASR-Nano, Fun-ASR-Nano, Qwen3-ASR |
|
||
| **TTS** | VoxCPM, VoxCPM1.5, VoxCPM2, Moss-TTS-Nano |
|
||
| **Image** | RMBG-2.0 (background removal) |
|
||
| **Embedding** | Qwen3-Embedding, all-MiniLM-L6-v2 |
|
||
| **Reranker** | Qwen3-Reranker |
|
||
|
||
## Changelog
|
||
### 2026-05-29
|
||
- generate code refactored
|
||
|
||
### 2026-05-28
|
||
- generate code refactoring progress 1/3
|
||
|
||
### 2026-05-27
|
||
- add MiniCPM5
|
||
|
||
### 2026-05-24
|
||
- update doc
|
||
|
||
### 2026-05-11
|
||
- add Moss-TTS-Nano，its performance is worse than the original Python version
|
||
|
||
### 2026-05-09
|
||
- merge pr/eastgold15/46, add aha-ui
|
||
|
||
**[View full changelog](docs/changelog.md)** →
|
||
|
||
## Why aha?
|
||
- **🚀 High-Performance Inference** - Powered by Candle framework for efficient tensor computation and model inference
|
||
- **🔧 Unified Interface** — One tool for text, vision, speech, and OCR
|
||
- **📦 Local-First** — All processing runs locally, no data leaves your machine
|
||
- **🎯 Cross-Platform** — Works on Linux, macOS, and Windows
|
||
- **⚡ GPU Accelerated** — Optional CUDA support for faster inference
|
||
- **🛡️ Memory Safe** — Built with Rust for reliability
|
||
- **🧠 Attention Optimization** - Optional Flash Attention support for optimized long sequence processing
|
||
|
||
## Quick Start
|
||
|
||
### Installation
|
||
|
||
```bash
|
||
git clone https://github.com/jhqxxx/aha.git
|
||
cd aha
|
||
cargo build --release
|
||
```
|
||
|
||
**Optional Features:**
|
||
|
||
```bash
|
||
# CUDA (NVIDIA GPU acceleration)
|
||
cargo build --release --features cuda
|
||
|
||
# Metal (Apple GPU acceleration for macOS)
|
||
cargo build --release --features metal
|
||
|
||
# Flash Attention (faster inference)
|
||
cargo build --release --features cuda,flash-attn
|
||
|
||
# FFmpeg (multimedia processing)
|
||
cargo build --release --features ffmpeg
|
||
```
|
||
|
||
### CLI Quick Reference
|
||
|
||
```bash
|
||
|
||
# List all supported models
|
||
aha list
|
||
|
||
# Download model only
|
||
aha download -m Qwen/Qwen3-ASR-0.6B
|
||
|
||
# Download model and start service
|
||
aha cli -m Qwen/Qwen3-ASR-0.6B
|
||
|
||
# Run inference directly (without starting service)
|
||
aha run -m Qwen/Qwen3-ASR-0.6B -i "audio.wav"
|
||
|
||
# Run local all-MiniLM-L6-v2 embedding (native safetensors)
|
||
aha run -m all-minilm-l6-v2 -i "Rust embedding test" --weight-path D:\model_download\all-MiniLM-L6-v2
|
||
|
||
# Start service only (model already downloaded)
|
||
aha serv -m Qwen/Qwen3-ASR-0.6B -p 10100
|
||
```
|
||
|
||
### Chat
|
||
|
||
```bash
|
||
aha serv -m Qwen/Qwen3-0.6B -p 10100
|
||
```
|
||
|
||
Then use the unified (OpenAI-compatible) API:
|
||
|
||
```bash
|
||
curl http://localhost:10100/v1/chat/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "Qwen/Qwen3-0.6B",
|
||
"messages": [{"role": "user", "content": "Hello!"}],
|
||
"stream": false
|
||
}
|
||
'
|
||
```
|
||
|
||
### aha-ui
|
||
```bash
|
||
cd aha-ui
|
||
```
|
||
|
||
#### use npm
|
||
##### install npm
|
||
refer to https://nodejs.org/en/download
|
||
```bash
|
||
# Download and install nvm:
|
||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.4/install.sh | bash
|
||
# in lieu of restarting the shell
|
||
\. "$HOME/.nvm/nvm.sh"
|
||
# Download and install Node.js:
|
||
nvm install 24
|
||
# Verify the Node.js version:
|
||
node -v # Should print "v24.15.0".
|
||
# Verify npm version:
|
||
npm -v # Should print "11.12.1".
|
||
```
|
||
|
||
##### npm run aha-ui
|
||
```bash
|
||
# Make sure in the aha-ui directory
|
||
# and make sure that aha has been compiled
|
||
npm install
|
||
npm run tauri dev
|
||
```
|
||
|
||
##### npm build & install & run
|
||
```bash
|
||
npm run tauri build
|
||
# target in
|
||
# -- aha-ui/src-tauri/target/release/bundle/deb/aha-ui_0.1.0_amd64.deb
|
||
# -- aha-ui/src-tauri/target/release/bundle/rpm/aha-ui-0.1.0-1.x86_64.rpm
|
||
# -- aha-ui/src-tauri/target/release/bundle/appimage/aha-ui_0.1.0_amd64.AppImage
|
||
```
|
||
|
||
#### use pnpm
|
||
##### install pnpm
|
||
```bash
|
||
curl -fsSL https://get.pnpm.io/install.sh | sh -
|
||
```
|
||
|
||
##### pnpm run aha-ui
|
||
```bash
|
||
# Make sure in the aha-ui directory
|
||
# and make sure that aha has been compiled
|
||
pnpm run tauri dev
|
||
```
|
||
|
||
##### pnpm build & install & run
|
||
```bash
|
||
pnpm run tauri build
|
||
# target in
|
||
# -- aha-ui/src-tauri/target/release/bundle/deb/aha-ui_0.1.0_amd64.deb
|
||
# -- aha-ui/src-tauri/target/release/bundle/rpm/aha-ui-0.1.0-1.x86_64.rpm
|
||
# -- aha-ui/src-tauri/target/release/bundle/appimage/aha-ui_0.1.0_amd64.AppImage
|
||
```
|
||
|
||
## Documentation
|
||
|
||
| Document | Description |
|
||
|----------|-------------|
|
||
| [Getting Started](docs/getting-started.md) | First steps with aha |
|
||
| [Installation](docs/installation.md) | Detailed installation guide |
|
||
| [CLI Reference](docs/cli.md) | Command-line interface |
|
||
| [API Documentation](docs/api.md) | Library & REST API |
|
||
| [Supported Models](docs/supported-models.md) | Available AI models |
|
||
| [Concepts](docs/concepts.md) | Architecture & design |
|
||
| [Development](docs/development.md) | Contributing guide |
|
||
| [Changelog](docs/changelog.md) | Version history |
|
||
|
||
|
||
## Development
|
||
|
||
### Using aha as a Library
|
||
> cargo add aha
|
||
|
||
```rust
|
||
// VoxCPM example
|
||
use aha::models::voxcpm::generate::VoxCPMGenerate;
|
||
use aha::utils::audio_utils::save_wav_mono;
|
||
use anyhow::Result;
|
||
|
||
fn main() -> Result<()> {
|
||
let model_path = "xxx/OpenBMB/VoxCPM2/";
|
||
|
||
let mut voxcpm_generate = VoxCPMGenerate::init(model_path, None, None)?;
|
||
let generate = voxcpm_generate.inference(
|
||
"aha是一个基于Rust和Candle框架的本地AI推理引擎，支持多模态模型（文本、视觉、语音、OCR）。".to_string(),
|
||
None,
|
||
None,
|
||
2,
|
||
1000,
|
||
10,
|
||
2.0,
|
||
6.0,
|
||
)?;
|
||
|
||
save_wav_mono(&generate, "voxcpm2.wav", voxcpm_generate.sample_rate() as u32)?;
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
### Extending New Models
|
||
|
||
- Create new model file in src/models/
|
||
- Export in src/models/mod.rs
|
||
- Add support for CLI model inference in src/exec/
|
||
- Add tests and examples in tests/
|
||
|
||
## Features
|
||
|
||
- High-performance inference via Candle framework
|
||
- Multi-modal model support (vision, language, speech)
|
||
- Clean, easy-to-use API design
|
||
- Minimal dependencies, compact binaries
|
||
- Flash Attention support for long sequences
|
||
- FFmpeg support for multimedia processing
|
||
|
||
## License
|
||
|
||
Apache-2.0 — See [LICENSE](LICENSE) for details.
|
||
|
||
## Acknowledgments
|
||
|
||
- [Candle](https://github.com/huggingface/candle) - Excellent Rust ML framework
|
||
- All model authors and contributors
|
||
|
||
## Wechat & Donate
|
||
<div align="center">
|
||
|
||
| Wechat Group | Donate |
|
||
|--------------|--------|
|
||
|  |  |
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
<p align="center">
|
||
<sub>Built with ❤️ by the aha team</sub>
|
||
</p>
|
||
|
||
<p align="center">
|
||
<sub>We're continuously expanding our model support. Contributions are welcome!</sub>
|
||
</p>
|
||
<p align="center">
|
||
<sub>If this project helps you, please consider giving us a ⭐ Star!</sub>
|
||
</p>
|