Live video in, 543 landmark coordinates per frame out — pose, face mesh and both hands, tracked by MediaPipe Holistic. That is the whole system, and saying so plainly is the substance of this case study.
Each frame goes through MediaPipe Holistic, which runs four models and returns landmark sets. The program draws them and moves to the next frame.
| Group | Landmarks | What each one is |
|---|---|---|
| Pose | 33 | Body joint positions, with visibility scores |
| Face mesh | 468 | Dense facial surface points |
| Left hand | 21 | Finger and palm joints |
| Right hand | 21 | Finger and palm joints |
| Total | 543 | Per frame, when all groups are visible |
The directory is called Micro-Behavior-Analysis-for-Predicting-Future-Actions. The code does not predict future actions, and never did. There is no temporal model, no cross-frame feature aggregation, no behaviour classifier, no labels, no training code and no evaluation. The project has always been a single file containing the extraction stage.
That gap was left open deliberately during the repair. The missing components are exactly the interesting ones — a temporal model over landmark sequences is a real piece of research work — and writing one now would produce something that had never existed, presented under a project name implying it always had.
Build an LSTM over the landmark stream, label a small dataset, and publish the project as a working behaviour classifier. Faster to make impressive. Dishonest about provenance.
Option BRepair the extraction stage so it runs on current MediaPipe, then state precisely which stages exist and which do not.
ChosenB. The pipeline diagram above marks stages 05 and 06 as never built, in red, on the public page. If the classifier gets written later it will be dated to when it was written.
The script was written against a MediaPipe API that has since been removed twice over. On a fresh install it failed before finishing the first frame.
| # | Defect | Effect |
|---|---|---|
| 1 | mp_holistic.FACE_CONNECTIONS | Removed in MediaPipe 0.8.9. AttributeError on frame 1 — the program never reached frame 2. Replaced with FACEMESH_CONTOURS. |
| 2 | No MediaPipe upper bound | Release 1.0.0 deleted mp.solutions entirely, so the import fails. Now pinned <1.0. |
| 3 | VideoStream(src=0).start() then immediate .read() | Returns None until the first frame arrives; that None went straight into cv2.cvtColor, which raises. Replaced with cv2.VideoCapture plus an explicit frame check. |
| 4 | draw_landmarks called unconditionally | MediaPipe returns None for groups it cannot see — a frame with no hands in it crashed the program. Now guarded per group. |
| 5 | No camera → unhandled crash | Now reports the cause and accepts --source <file> so it runs headless. |
Defect 1 is a one-line fix. Defect 2 is the one that keeps the project alive: MediaPipe 1.0 removed the entire solutions namespace this code is built on. Without an upper bound, a routine pip install -r requirements.txt a year from now would fail at import with no obvious cause. The pin is documented in the requirements file with the reason attached.
| Check | Result |
|---|---|
| MediaPipe 0.10.21 imports, Holistic initialises | Pass |
Video source opens headless (--no-window) | Pass |
| Frames processed without exception | 30 / 30 |
| Pose landmarks recovered | 33 in 30/30 frames |
| Face landmarks recovered | 468 in 30/30 frames |
| Absent hand groups handled without crash | Pass |
Environment: Python 3.12, OpenCV 4.11, CPU. No throughput figure is quoted. On a clip this short the elapsed time is dominated by model initialisation, so any frames-per-second number derived from it would misrepresent steady-state performance.
python -m venv .venv && .venv\Scripts\activate pip install -r requirements.txt python main.py # webcam python main.py --source clip.mp4 # video file python main.py --source clip.mp4 --no-window --max-frames 100
This code measures body, face and hand landmark geometry. It infers nothing about mental state, emotion, personality, deception, criminal intent, or any medical or psychological condition. Any claim of that kind would be unsupported by this code, and the wider research literature does not support it either — landmark geometry is not a channel into someone's mind.
Behavioural computer vision attracts overclaiming, and the label "micro-behaviour" attracts it more than most. The scope statement above is reproduced verbatim in the project's own README so the constraint travels with the source rather than living only on this page.
The extraction stage works and is verified. The temporal and classification stages do not exist. Both facts are on this page because a portfolio that only shows finished work is not a record of how research actually goes.