Server Environment Configuration

Only after configuring the server environment can the project run smoothly, including model training, inference, and animal behavior analysis processes. Paper Figure Reproduction🔗 does not require server environment configuration, only local environment configuration is needed.

Server Environment Configuration Notice

The deep learning environment is the core foundation of this project. We understand that environment setup can be a complex and challenging process, which is a common problem throughout the AI field. Although we are continuously optimizing the deployment process, the current project still has some version compatibility issues. We recommend that you be patient when configuring the environment and have a certain spirit of technical research to deal with possible challenges.

Hardware Configuration

Server hardware configuration includes:

  • CPU: Intel Xeon 4314
  • Memory: 256 GB
  • GPU: 4 NVIDIA GeForce RTX 3090 cards
  • Storage: 1 TB SSD + 8 TB HDD

Software Environment

Software environment includes:

  • Operating System: Ubuntu 18.04
  • Deep Learning Framework: CUDA 11.4, cuDNN 8.8, TensorRT 8.6
  • Development Tools: Python 3.7.16, OpenCV-Python 4.8.0, TensorFlow 2.7.0, PyTorch 1.11.0

Used for deep learning model training and data analysis.

Important Warning

Currently only supports Ubuntu system and NVIDIA graphics cards. Please refer to relevant materials for other systems.

Important Warning

Since the software was configured in 2019, some software versions may be outdated. It is recommended to choose the latest versions according to actual needs. Most Python libraries can be found in pip and conda, but some libraries may be difficult to install compatibly. This document cannot accommodate all software's latest versions. Please refer to relevant materials.

Software Name Local Test Version (2022) 2025 Version, Untested Notes
Ubuntu 18.04 24.04 Operating system, recommended
Python 3.7.16 3.12 Development language, required
CUDA 11.4 12.9 Deep learning framework, required
cuDNN 8.8 8.9 Deep learning framework, required
TensorRT 8.6 10.13 Inference acceleration, required
OpenCV-Python 4.8.0 4.12.0 Image processing, required
TensorFlow 2.7.0 2.16.0 DANNCE model, required
PyTorch 1.11.0 2.7.1 Deep learning framework, required
Docker 24.0.6 28.3.0 Containerization, for Mediamtx
FFmpeg 4.x 6.x Video reading and writing
Mediamtx - - Video stream server, for closed-loop behavioral intervention

Test Installation Environment

After all software is installed, testing is required to ensure successful installation. Testing methods are as follows:

Test nvidia cuda driver

$ nvidia-smi

Test Nvidia Cuda Compiler

$ nvcc --version

Test TensorRT Command Line Tools

$ trtexec --onnx=/path/to/model.onnx --saveEngine=/path/to/model.engine

Test polygraphy (NVIDIA Official Tool) Command Line Tools

$ polygraphy inspect model --onnx=/path/to/model.onnx

Test FFmpeg Command Line Tools

# ffmpeg generate a 640x480 video, 30 fps, 1 minute
$ ffmpeg -f lavfi -i testsrc=duration=60:size=640x480:rate=30 -c:v libx264 -pix_fmt yuv420p -f mp4 /path/to/test.mp4

# convert it to hevc by NVIDIA HEVC encoder
$ ffmpeg -i /path/to/test.mp4 -c:v hevc_nvenc -pix_fmt yuv420p -f mp4 /path/to/test_hevc.mp4

Test Docker Command Line Tools

$ docker run hello-world
$ docker run --rm -it -e MTX_PROTOCOLS=tcp -p 8554:8554 -p 1935:1935 bluenviron/mediamtx

Third-party Deep Learning Toolkits

1. OPEN-MMLAB/MMDETECTION Model

This is an open-source project that provides MaskRCNN for object detection and segmentation. We forked it (in 2022) and added custom models and configuration files to achieve multi-rat segmentation.

Please refer to MMdetection's official documentation for installation instructions.

  • Code address: https://github.com/chenxinfeng4/mmdetection.git

Important Warning

We forked MMDET 1.x version, which currently supports python3.7; the latest MMDET on the official website is 3.x version, which may have serious compatibility issues. Please choose the appropriate version according to actual needs.

2. OPEN-MMLAB/MMPOSE Model

This is an open-source project that provides MaskRCNN for human pose keypoint detection. We forked it (in 2022) and added custom models and configuration files to achieve ball detection.

Please refer to MMpose's official documentation for installation instructions.

  • Code address: https://github.com/chenxinfeng4/mmpose.git

Important Warning

We forked MMPOSE 0.x version. New versions and environments may have compatibility issues. Please choose the appropriate version according to actual needs.

3. YOLO-v8 Model

This is a lightweight object detection and instance segmentation open-source project. We forked it (in 2024) and added custom models and configuration files to achieve real-time multi-rat segmentation.

  • Code address: https://github.com/chenxinfeng4/ultralytics.git

Important Warning

Our forked version may have compatibility issues with the latest version. Please choose the appropriate version according to actual needs.

4. DANNCE Model

This is a library for multi-view animal pose estimation. We forked it (in 2022) and added custom models and configuration files to achieve real-time multi-rat 3D keypoint recognition.

  • Code address: https://github.com/chenxinfeng4/dannce.git

Important Warning

Our forked version may have compatibility issues with the latest version. Moreover, this library has high prediction accuracy but slow speed. We performed speed optimization, which caused some differences from the original code and more compatibility issues. Installation is relatively troublesome, and we are optimizing the installation documentation.

5. SmoothNet Model

This is a library for pose time smoothing. We forked it (in 2022) and added custom models and configuration files to achieve real-time multi-rat 3D keypoint smoothing.

  • Code address: https://github.com/chenxinfeng4/SmoothNet.git

Important Warning

Our forked version may have compatibility issues with the latest version. Please choose the appropriate version according to actual needs.

Install Python Packages Created and Modified by This Project

1. LILAB Analysis Toolkit

Contains common analysis tools, such as camera calibration process, rat segmentation and pose estimation process, behavior recognition process, big data mining related tools, etc.

  • Code address: https://github.com/chenxinfeng4/LILAB-py.git

Installation method:

git clone https://github.com/chenxinfeng4/LILAB-py.git
cd LILAB-py
pip install -e .

Important Warning

Contains many process tools with overly large dependencies, so automatic dependency installation is not performed. If import errors occur, dependencies need to be installed manually.

2. FFMPEGCV Toolkit

GPU-accelerated video processing toolkit, convenient for multi-camera video reading and writing, which is 9 times faster than OpenCV in certain scenarios. After I open-sourced it, the annual download volume is 70,000 times. See corresponding documentation.

  • Code address: https://github.com/chenxinfeng4/ffmpegcv.git

Installation method:

pip install ffmpegcv

Important Warning

To use GPU acceleration, NVIDIA-driver is required.

3. multiview_calib Toolkit

Camera extrinsic calibration (ball calibration) toolkit. Based on third-party libraries, it has been optimized and custom calibration processes have been added. See corresponding documentation.

  • Code address: https://github.com/chenxinfeng4/multiview_calib.git

Installation method:

pip install git+https://github.com/chenxinfeng4/multiview_calib.git

# Simple ball calibration tutorial
git clone https://github.com/chenxinfeng4/multiview_ball_calib

4. Other Modified Toolkits

torch2trt: torch2trt modification, supporting direct loading of engine files.

pip install git+https://github.com/chenxinfeng4/torch2trt

cocoapi: Multi-threading version of cocoapi to improve speed.

pip install git+https://github.com/chenxinfeng4/cocoapi

mmdetection-to-tensorrt: Converting Mask-RCNN model to tensorrt model.

pip install git+https://github.com/chenxinfeng4/mmdetection-to-tensorrt