Huang Qi
de602365e9
- Move all code into src/mouse_control/ following the src-layout convention, with model/collect/train/visualize/inference as separate modules and SimpleNet extracted into model.py - Add hatchling-based pyproject.toml with three CLI entry points (mouse-collect, mouse-train, mouse-visualize) and bundle the trained ONNX model as a package resource under assets/ - Move training data to data/, delete superseded show.py, remove dead imports (numpy, onnxruntime, onnx, onnxsim) and the unused visualize_path() helper - Fix crashes in the collector: guard against destroyed-widget access after the n==100 destroy(), and skip save_to_csv when recording is off or the path is too short to derive 10 keypoints - Switch ONNX export to external_data=False so the model ships as a single self-contained 19KB file - Bytes-load the bundled model via importlib.resources so packaging remains correct under zip-safe distributions - Rewrite README around the new layout, commands, and public API
149 lines
4.2 KiB
Markdown
149 lines
4.2 KiB
Markdown
# mouse_control
|
||
|
||
一种基于神经网络来模拟人手移动鼠标的方法。
|
||
|
||
## 简介
|
||
|
||
本项目源于该死的轨迹检测。我们用一个简单的三层全连接神经网络去拟合真人鼠标移动轨迹。
|
||
|
||
<img src="./imgs/Figure_1.png">
|
||
|
||
上图为真人轨迹移动得到的散点图——用 10 个关键点拟合一条轨迹。可以看到，人手移动和 PID、贝塞尔等曲线有明显区别，因此用神经网络来拟合更合适。
|
||
|
||
**网络结构**：
|
||
|
||
- 输入：`(dx, dy)` —— 目标点相对当前位置的偏移
|
||
- 三层全连接：`Linear(2 → 64) → ReLU → Linear(64 → 32) → ReLU → Linear(32 → 20)`
|
||
- 输出：`(10, 2)` —— 10 个轨迹点
|
||
|
||
仓库内含 `data/` 下的 400 条采集数据和 `src/mouse_control/assets/mouse.onnx` 预训练模型，clone 之后可以直接 `mouse-visualize` 看效果。
|
||
|
||
## 安装
|
||
|
||
使用 [uv](https://docs.astral.sh/uv/) 管理：
|
||
|
||
```bash
|
||
uv sync
|
||
```
|
||
|
||
或从构建产物安装：
|
||
|
||
```bash
|
||
uv pip install dist/mouse_control-0.1.0-py3-none-any.whl
|
||
```
|
||
|
||
要求 Python ≥ 3.12。
|
||
|
||
## 项目结构
|
||
|
||
```
|
||
mouse_control/
|
||
├── pyproject.toml
|
||
├── README.md
|
||
├── data/ # 训练 / 测试数据 (380 + 20 条)
|
||
│ ├── mouse_data.csv
|
||
│ └── mouse_data_test.csv
|
||
├── imgs/ # README 图片
|
||
└── src/mouse_control/ # 包源码 (src layout)
|
||
├── __init__.py # 公共 API
|
||
├── model.py # SimpleNet 网络定义
|
||
├── collect.py # 数据采集 GUI
|
||
├── train.py # 训练 + ONNX 导出
|
||
├── visualize.py # 轨迹可视化
|
||
├── inference.py # 模型加载 / 推理辅助
|
||
└── assets/
|
||
└── mouse.onnx # 随包发布的预训练模型
|
||
```
|
||
|
||
## 命令行用法
|
||
|
||
安装后可以直接调用三个命令（也可以用 `uv run <cmd>`）：
|
||
|
||
### 1. 采集真人鼠标轨迹
|
||
|
||
```bash
|
||
uv run mouse-collect -o data/mouse_data.csv -n 100
|
||
```
|
||
|
||
<img src="./imgs/collect.png">
|
||
|
||
全屏界面：点**红球**开始记录、移动到**蓝球**点击结束记录，每条轨迹保存 10 个关键点到 CSV。
|
||
|
||
- 默认采集 100 条后自动退出，按 `Esc` 提前退出
|
||
- 推荐采集 ≥ 300 条以获得较好的训练效果
|
||
|
||
### 2. 划分训练 / 测试集
|
||
|
||
把 `mouse_data.csv` 里**剪切**若干行（10-20 条）到 `mouse_data_test.csv` 即可。
|
||
|
||
### 3. 训练模型
|
||
|
||
```bash
|
||
uv run mouse-train \
|
||
--train-csv data/mouse_data.csv \
|
||
--test-csv data/mouse_data_test.csv \
|
||
--output mouse.onnx \
|
||
--epochs 1000
|
||
```
|
||
|
||
训练结束后在当前目录生成 `mouse.onnx`。其它参数 `--batch-size` `--lr` 可调。
|
||
|
||
### 4. 可视化效果
|
||
|
||
```bash
|
||
uv run mouse-visualize --n-trajectories 10 --seed 42
|
||
```
|
||
|
||
<img src="./imgs/Figure_2.png">
|
||
|
||
随机生成若干目标点，每条轨迹包含 10 个模型输出点 + 80 个三次样条插值点。`--no-show` 跳过 GUI 窗口、只写 PNG；`--model` 指定其它 ONNX 文件。
|
||
|
||
## Python API
|
||
|
||
```python
|
||
import mouse_control as mc
|
||
|
||
# 加载模型：优先 ./mouse.onnx，否则回退到包内打包的预训练模型
|
||
session = mc.load_model()
|
||
|
||
# 推理：输入 (dx, dy)，输出 (10, 2) 轨迹点
|
||
pts = mc.predict(session, 120, -80)
|
||
# -> ndarray, shape (10, 2), dtype float32
|
||
|
||
# 直接使用模型类（用于自训练 / 微调）
|
||
from mouse_control import SimpleNet
|
||
```
|
||
|
||
## ONNX 推理（其它语言）
|
||
|
||
### Python (onnxruntime)
|
||
|
||
```python
|
||
import numpy as np
|
||
import onnxruntime as ort
|
||
|
||
session = ort.InferenceSession("mouse.onnx")
|
||
inp = np.array([[[100.0, 200.0]]], dtype=np.float32) # dx=100, dy=200
|
||
output = session.run(None, {"input": inp})[0] # shape (1, 10, 2)
|
||
```
|
||
|
||
### C++ (OpenCV DNN)
|
||
|
||
```cpp
|
||
cv::dnn::Net net = cv::dnn::readNetFromONNX("mouse.onnx");
|
||
cv::Mat blob(1, 2, CV_32F, cv::Scalar(100, 100)); // 输入 dx=100, dy=100
|
||
net.setInput(blob, "input");
|
||
cv::Mat output = net.forward("output"); // 输出 1*10*2
|
||
std::cout << output.at<float>(0, 8, 0) << std::endl;
|
||
```
|
||
|
||
## 打包发布
|
||
|
||
```bash
|
||
uv build
|
||
# -> dist/mouse_control-0.1.0-py3-none-any.whl
|
||
# -> dist/mouse_control-0.1.0.tar.gz
|
||
```
|
||
|
||
wheel 内含 `assets/mouse.onnx` 预训练模型，安装后直接 `mc.load_model()` 即可使用。
|