How to analyze classroom behavior using computer vision and pose estimation?

One possible approach is to combine object detection, pose estimation and behavior analysis.

A typical pipeline could look like this:

1. Use an object detection model such as YOLOv8 to detect students in the classroom.

2. Apply pose estimation (for example MediaPipe Pose or OpenPose) to extract skeleton keypoints.

3. Analyze interactions using spatial distance or temporal patterns.

4. Aggregate the results into behavioral indicators such as attention level, activity level and social interaction.

I implemented a prototype system using this idea in an open-source project:

https://github.com/KEYUJIN881129/AI-Based-Five-Capability-Smart-Kindergarten-System

The project demonstrates how computer vision and multimodal AI can be used to estimate behavioral indicators such as attention, participation and activity level in a classroom environment.

It may provide a useful reference for building a similar system.

I don't see YOLO nor MediaPipe in project on GitHub. There is only basic code for titanic.csv

This is a very challenging project considering the practical constraints involved. Before thinking about models, it is important to consider the image acquisition and processing challenges, such as camera placement, viewing angles, distance to students, and the number of individuals in the same scene. In a real classroom environment these factors can significantly affect the quality of the visual signals available for analysis.

For a production system, this would likely require careful camera positioning and potentially significant hardware investment to ensure that students are captured with sufficient spatial resolution.

Given the current state of computer vision, what tends to work more reliably is body behavior analysis rather than attempting to infer complex internal states directly. From pose and movement patterns it is possible to infer observable behaviors such as:

• activity level (hand, head, or torso movement)

• focused behavior (writing, reading, head orientation toward the front)

• social interaction (turning toward or leaning toward other students)

These signals generally cannot be inferred from a single frame. Instead, they need to be extracted from temporal sequences of frames, where the system can detect events such as:

• a student leaning toward another student

• writing activity over a short time window

• sustained head orientation

• posture changes

These events can then be aggregated into higher-level behavioral indicators.

A practical pipeline for this type of system could look like:

person detection
↓
tracking (DeepSORT / ByteTrack)
↓
pose estimation
↓
temporal feature extraction
↓
behavior classification

Tracking is important because it allows the system to maintain individual identities across frames, enabling the analysis of behavioral patterns over time.

Finally, any attempt to derive higher-level metrics (e.g., engagement or participation) would likely require aggregating temporal observations per individual, potentially maintaining a per-student behavioral profile over time.

In practice, most successful systems focus on observable behaviors rather than attempting to directly infer emotional or cognitive states.