What is SocialSeq Pipeline Playground?

The SocialSeq pipeline playground is a Docker packaged code environment designed to help users understand the functionality and logic of the SocialSeq framework through the demonstration data processing process.

Use the web browser to explore the social-seq playground.

System Requirements

SocialSeq has been tested on the following configurations:

• Windows 10 (with WSL2) + x86_64 + NVIDIA RTX4090
• Debian 11 + x86_64 + NVIDIA A100
• Ubuntu 20.04 + x86_64 + NVIDIA RTX3090

Docker and NVIDIA Driver need to be installed:

$ docker --version  # Docker version should be >= 24.0.6
$ nvidia-smi        # NVIDIA driver version should be >= 550.54

Installation Guide

You need to install Docker and download the model and data files🔗:

$ cd SOCIAL_SEQ_DOWNLOAD_PATH
$ docker load -i ./lilab_socialseq_pipeline_cuda_vscode_amd64_2025*.tar
$ docker images | grep lilab      # Result will be an image file named 'lilab*'

$ unzip lilab_socialseq_pipeline_code_with_data_2025*.zip -d ./pipeline  # Unzip file
$ ls ./pipeline/1-Ball_Calibration   # Check path

$ docker run --rm -it -p 8080:8080 \
    --gpus all \
    --name lilab_socialseq \
    -e PASSWORD=2025cxf  \
    -v ./pipeline:/root/Downloads/pipeline  \
    lilab/socialseq:codeserver-cuda-tensorrt-torch-tensorflow-mmdet-mmpose-dannce-yolo-20250903 \
    /app/code-server/bin/code-server

Navigate to http://localhost:8080 and enter the PASSWORD 2025cxf to access the VS Code server environment where you can view, edit, and run SocialSeq code.

If you're running Docker on a remote server, you should forward port 8080 to your local machine. For more information, please refer to bug fix.

Or enter the container through the Docker terminal:

$ docker exec -it lilab_socialseq bash

Environment Verification

In the Docker VS Code server environment at http://localhost:8080:

$ nvidia-smi  # In the Docker container, should display GPU information

Model Initialization

Convert deep learning models to TensorRT for improved performance. Since TensorRT models are bound to specific systems and hardware, this step must be performed on each new machine.

bash /root/Downloads/pipeline/model_dannce/convert_dannce_hdf5_to_tensorrt.sh
bash /root/Downloads/pipeline/model_mask_rcnn_r101_fpn_2x_coco_bwrat_816x512_cam9/convert_mmdet_model_to_tensorrt.sh
bash /root/Downloads/pipeline/model_mmpose/convert_mmpose_model_to_tensorrt.sh
bash /root/Downloads/pipeline/model_YOLOv8/convert_yolov8seg_to_tensorrt.sh

# Patch the LILAB-pkg package
cp -r /root/Downloads/pipeline/LILAB-pkg-patch-for-docker/* ~/LILAB-pkg/

After running these commands, you will generate the following files:

• model_mmpose/latest.full.engine
• model_dannce/DANNCE/train_results/MAX/latest.engine
• model_YOLOv8/last.full.engine
• model_mask_rcnn_r101_fpn_2x_coco_bwrat_816x512_cam9/latest.trt

All set! You're now ready to run the SocialSeq pipeline playground with demo data.

🚀 Quick Start

After completing the above installation steps, you can start running the playground.

1. Ball Calibration

Using a ball as a calibration target provides better visibility than a chessboard, enabling faster extrinsic calibration of multi-camera systems. For detailed information, refer to the documentation.

bash /root/Downloads/pipeline/1-Ball_Calibration/run_task.sh

2. Social 3D Pose (SOCIAL)

Processing multi-camera animal videos through Mask R-CNN for animal identification and segmentation, followed by DANNCE for 3D pose keypoint inference, and SmoothNet for motion keypoint smoothing. For detailed information, refer to the documentation.

bash /root/Downloads/pipeline/2-Social_3D_Pose/run_task_segmentation.sh   # Mask R-CNN for ID segmentation
bash /root/Downloads/pipeline/2-Social_3D_Pose/run_task_keypoint.sh       # DANNCE and SmoothNet for 3D pose reconstruction

3. Sequence Labeling (SEQ)

Extracting social-related features from 3D poses and segmenting them (0.8 seconds per segment). Using a Seq2seq-FCN model to classify each video segment, resulting in consistent social sequence labels. For detailed information, refer to the documentation.

bash /root/Downloads/pipeline/3-Sequence_Labeling/run_task.sh

4. Closed-Loop Control (LIVE)

Establishing a real-time behavior analysis system with closed-loop feedback. This will involve multi-camera perception and recognition of animal social behaviors to establish optogenetic conditional stimulation.

Not yet added to the playground, preparation in progress