Why 2D Beat 3D on the Angles That Matter

I once had a working monocular 3D pipeline in this system: a neural network that took one video frame and returned a full parametric body, from which you could read any joint angle you liked in three dimensions. I deleted it and replaced it with school trigonometry on pixel coordinates. The numbers got better. This post is why, and it is also the post where I mark the exact line past which the trade stops working.

#1. The trade that looks impossible to lose

A knee angle is the angle between the thigh and the shank. The thigh and the shank are three-dimensional objects, so the honest place to measure that angle is in three dimensions. A camera flattens the world onto a sensor, and measuring in that flattened picture is obviously an approximation: if the athlete’s thigh is angled toward the lens, the picture shows a short thigh and the angle you read off is not the angle the knee is actually holding.

So the obvious move is to undo the flattening. Take the single camera, run a network that predicts where each joint sits in space, and read the angle from the recovered 3D positions. That is what a monocular 3D lifter does. In the precise version: the shipped path in part 2 computes each angle from two 2D vectors in the image plane, while the lifter fits a parametric body model (an SMPL-X mesh, via HybrIK-X in this system’s case) and reads Euler angles off the fitted skeleton. One of those two representations strictly contains the other. The 3D one knows everything the 2D one knows plus a depth coordinate per joint.

The angles this system reports are sagittal flexion: a joint folding in the plane you see when you stand side-on to a runner, so knee bend, hip bend, elbow bend. On exactly those angles the poorer representation wins. That is the counterintuitive result, and it is not a fluke of my implementation. It shows up in the published literature and it shows up again, independently, in this repository’s own benchmarks. The mechanism is one sentence long, and everything else in this post is an unpacking of it: the extra coordinate is not measured, it is inferred, and nothing in the image can check the inference.

#2. What the published comparison says

The design decision rested on a paper other people wrote, and I want to be exact about the boundary between their numbers and mine.

Rode et al. (Scientific Reports, 2025) assessed 11 open-source monocular markerless pose estimators, spanning 18 two-dimensional and 26 three-dimensional model configurations, against a 27-camera Vicon system, on 2.2 million RGB frames from 25 participants performing physiotherapy exercises. The lowest knee-flexion mean absolute error, averaged over participants, exercises and camera angles, was 9.3 degrees among the 2D estimators and 14.1 degrees among the 3D ones. Elbow flexion was worse in both families. Those two figures are theirs, not mine, and they are the most directly comparable published reference point I could find for a monocular sagittal knee angle.

Two caveats govern any comparison with it, and the repository records both rather than quoting the headline alone. Their movements are planar clinical exercises performed to instruction; my evaluation set is free outdoor sport, where limbs leave the sagittal plane even when the trunk does not. And both their figures and my aggregate average over camera angle, which part 5 shows is the single variable that dominates monocular error. The right reading is that the two sets of numbers bracket the same regime. They are not a league table.

Here is the same comparison drawn as a pipeline, with the costs that this repository measured attached to the stages they belong to.

  shipped 2D path                      removed monocular-3D path
  ───────────────                      ─────────────────────────
  detector pixels                      detector pixels
     │  7.38° on COCO val2017             │  same detector, same cost
     ▼                                    ▼
  facing sign                          inferred depth + body shape
     │  100/100 correct, n=919            │  no residual can check it
     ▼                                    ▼
  angle in the image plane             angle in 3D space
     │  out-of-plane motion is            │  loses nothing in
     │  silently lost                     │  principle
     ▼                                    ▼
  9.3° best published, planar          14.1° best published, same
  clinical exercises                   frames, same protocol

The 7.38 degrees is mine: it is the mean error, in the product’s own clinical units, between angles computed from human COCO val2017 annotations (a public photograph set whose joint positions were marked by hand) and angles computed from the detector’s keypoints on the same image and the same box (n=2000, validation/README.md section 2). The 100/100 facing figure is mine too (COCO val2017, n=919) and is the subject of part 4. The 9.3 and 14.1 are Rode et al.

The important structural point about that diagram is that the two columns share their first stage. Both paths pay the detector’s pixel error in full. The 3D column then adds a stage that the 2D column does not have, and the question is only whether that stage gives back more than it costs.

#3. Why the depth estimate cannot be checked

Point a camera at a knee. The detector gives you a pixel. Every point along the ray from the lens through that pixel projects back to exactly the same pixel. The image contains no information about where along the ray the knee is.

 ONE camera, one pixel, many worlds

   cam        image plane
    ⊙ ─────────┬── × ──┬───────── ○ ──────── ○ ──────── ○
               │       │  d =   2.0 m      2.5 m      3.0 m
               └───────┘

   The pixel × is measured. The distance d along the ray is not.
   All three candidates reproject to the SAME pixel, so no image
   residual can prefer one over another. Something else has to
   choose, and that something is a prior learned from training
   data, not an observation of this athlete on this day.

A monocular lifter resolves this the only way it can: with a prior. It has learned what human bodies look like and how limbs are proportioned, and it picks the depth that makes the observed pixels most plausible under that learned model. This works remarkably well as computer vision. It is very weak as measurement, for a reason that has nothing to do with how good the network is: there is no residual. In multi-view triangulation, a second camera’s ray constrains the depth, and the disagreement between rays leaves a reprojection error you can at least look at: push the reconstructed point back out through each lens and measure, in pixels, how far it lands from the pixel the detector actually gave you. From one camera, every depth fits the evidence perfectly. The error term you would need in order to notice a bad depth estimate is identically zero by construction.

Two consequences follow, and they push in opposite directions.

The first is that for motion already in the image plane, the depth coordinate contributes nothing to the answer but contributes its full error. A knee flexing in a sagittal-plane squat sweeps across the sensor. The angle is visible in pixels. Reconstructing it in space means computing the same angle from coordinates two-thirds of which were observed and one-third of which was guessed. The guess cannot help, because there was nothing left to learn, and it can hurt.

The second is that for motion out of the image plane, the depth coordinate is the whole answer, and a guessed answer beats no answer at all. That is section 6, and it is where the literature’s elbow result and my own worst joint meet.

#What the repository measured on the same idea

I could not reproduce Rode et al.’s 2D-versus-3D comparison, because I removed the lifter rather than benchmarking it side by side. What I can show is the size of the effect they are describing, measured on my own code. Against ASPset-510’s marker-based 3D ground truth (n=1986, validation/README.md section 3):

Configuration	MAE
Angle code on noise-free projections of the ground-truth 3D	11.72°
Same code, real detector	19.44°
Implied detector contribution	about 7.7°

That 7.7 degrees agrees with the 7.38 degrees measured on COCO by a completely different harness, and two unrelated routes landing within half a degree of each other is the only reason I trust either. The interesting row is the first one. Even with perfect pixels, the monocular image-plane path carries a floor of nearly 12 degrees against 3D truth. That floor is the out-of-plane motion the image plane throws away, and it is the thing a working 3D estimate would buy back.

It is worth knowing why 19.44 is so much larger than Rode et al.’s 9.3. The repository’s own analysis notes put ASPset’s median camera obliquity at 46 degrees (paper/athleteos-markerless-rom.md). Obliquity is how far the camera sits from side-on, 0 degrees being a perfect profile and 90 degrees face-on, so the aggregate is dominated by viewpoints the monocular method is not suited to. Band the same readings by the system’s own obliquity estimate and the near-side-on band comes in at 7.69 degrees (n=153) against 12.79 degrees when banded by the mocap-derived truth. Both numbers are in validation/README.md section 3b, and why the estimate outperforms the truth is part 5’s whole subject.

#4. What the 2D path computes instead

The replacement is not clever. That is the point of it. A hinge joint’s flexion is the signed rotation from the proximal segment, continued through the joint, to the distal segment, computed from the two pixel vectors:

$θ = atan2 (v_{1} \times v_{2}, v_{1} \cdot v_{2})$

In pose-service/src/pose_service/pipeline/angles_from_2d.py that is seven lines:

def _signed_angle(v1: np.ndarray, v2: np.ndarray) -> float:
    """Signed angle v1 -> v2 in degrees, positive counter-clockwise (y-up)."""
    cross = float(v1[0] * v2[1] - v1[1] * v2[0])
    dot = float(v1[0] * v2[0] + v1[1] * v2[1])
    if abs(cross) < 1e-12 and abs(dot) < 1e-12:
        return 0.0
    return float(np.degrees(np.arctan2(cross, dot)))

atan2 of the cross over the dot rather than arccos, because arccos loses about half its significant digits near 0 and 180 degrees, which is exactly where a straight knee sits. The knee itself is then:

seg_prox = _up(middle) - _up(proximal)
seg_dist = _up(distal) - _up(middle)
if float(np.linalg.norm(seg_prox)) < 1e-6 or float(np.linalg.norm(seg_dist)) < 1e-6:
    return 0.0
bend = _signed_angle(seg_prox, seg_dist)
sense = 1.0 if flex_ccw_when_facing_right else -1.0
return sense * facing * bend

Three keypoints, one atan2, one sign. _up flips image y so the maths runs in a conventional y-up frame. facing is the athlete’s left/right orientation, and flex_ccw_when_facing_right encodes the anatomy: the knee folds the shank backward while the elbow folds the forearm forward, so the two joints take opposite rotation senses for the same facing. Hyperextension comes out negative on its own, with no special case.

What it costs. Two things, both of them real. The sign depends entirely on facing, which is part 4. And the magnitude depends on the camera being near side-on, which is part 5. Get either wrong and the reading is not slightly wrong, it is mirrored or foreshortened. The 2D path does not remove uncertainty from the problem, it relocates it from an invisible depth prior to two things that can be estimated, reported and argued about.

What it also bought, which I should not pretend was incidental. Accuracy was one of three reasons the lifter went, and the other two were not scientific. SMPL-X carries a non-commercial licence, which is a problem for a commercial product. And dropping the lifter removed a source-built CUDA extension, an import pip-at-module-scope build hack, a maze of working-directory-relative symlinks and roughly 9 GB of Docker image (pose-service/README.md). If the accuracy had gone the other way I would have paid all of that and kept the mesh. It did not, so all three arrows pointed the same way, and a decision where every reason agrees deserves less credit than one where they conflict.

The lifter is still wired in behind enable_mesh_overlay, which defaults to False (pose-service/src/pose_service/config.py), and even when it is on it feeds a 3D viewer and not the metrics. The clinical numbers come from the 2D path either way, so enabling the mesh cannot silently change a measurement. That separation is deliberate: a display feature that can alter a recorded number is a governance bug wearing a graphics hat.

#5. The boundary, stated as a refusal

Here is what 2D genuinely cannot do, and this is not a matter of accuracy. It is a matter of the quantity not being present in the data.

Axial rotation is a twist about a limb’s own long axis. Internal and external rotation at the hip, which the clinical workbook aggregates into Hip Total ARC; the same at the shoulder, which becomes Shoulder Total ROM. The way you observe a twist is by watching the segment below it act as a pointer: bend the knee to 90 degrees and the shank swings around the femur’s axis, and how far it has swung is the hip’s rotation.

  sagittal flexion                    axial rotation
  ────────────────                    ──────────────
   ● hip                               ● hip
   │                                   │  u = femur long axis
   ● knee ──▶ swings ACROSS            ● knee
   │          the sensor                ╲ v = shank, the pointer
   ● ankle    every degree               ● ankle
              moves pixels
                                      Rotating about u sweeps the
  One camera sees all of it.          pointer TOWARD and AWAY from
  The angle is in the pixels.         the lens. One camera sees a
                                      foreshortened chord, and the
                                      part it cannot see IS the answer.

From one camera, that pointer’s swing is almost entirely along the viewing direction. There is no degraded version of this measurement to report. So the monocular path reports nothing. From the module’s own docstring, on what it deliberately does not compute: axial rotations, knee valgus, ankle inversion and pelvis angles are “structurally unobservable or clinically indefensible from one camera”, and they stay at 0.0 so the schema is unchanged and the aggregator reads them as not measured.

The same contract is stated on the Go side, where the API’s callers see it (backend/internal/domain/pose.go):

// PoseModeMonocular: a single phone/camera. Runs the 2D detector and
// computes clinical joint angles directly from the image-plane
// geometry of the smoothed keypoints, no parametric body fit.
// [...] Axial rotation, and therefore Hip Total ARC and Shoulder
// Total ROM, is NOT recoverable from one camera and is reported as
// "not measured" rather than estimated. Use multiview for those.
PoseModeMonocular PoseMode = "monocular"

Reporting not-measured is cheap to write and expensive to hold. Every product review asks why the monocular mode cannot produce a hip rotation number when a competitor’s demo appears to. The answer is that the competitor’s demo has a prior where mine would need an observation. Part 7 is entirely about the discipline of refusing, and about how many of this system’s apparent accuracy gains are really refusals in disguise.

#Even multi-view only partly recovers it

I should not oversell the alternative. With three real calibrated cameras on ASPset-510, on frames where sagittal angles measured 8.9 degrees, axial rotation measured 41.3 degrees MAE (validation/README.md section 4). That is the same code and the same detector that reach 6.9 degrees on CMU Panoptic with all 21 views, and 4.27 degrees against a marker-based reference on the LBMC dataset’s nine-camera 360-degree rig (section 8e). That last one is an aligned figure, 32.09 degrees raw, with a constant offset and a sign convention taken out; section 6 comes back to what alignment does and does not excuse.

The difference is rig geometry, not the estimator. Panoptic is a dome surrounding the subject and LBMC is a genuine surround (measured azimuths 34/78/132/157/196/214/277/322/356 degrees, largest gap 62.8 degrees). ASPset is three cameras in a shallow arc about 10 m out, which barely constrains the transverse plane at all. The product’s canonical rig surrounds the athlete, so it resembles Panoptic and LBMC far more than ASPset. Both numbers are reported and neither is quoted alone, and until an in-vivo study reports limits of agreement on a real rig, hip ARC and shoulder ROM stay Tier C (“screening only”) with their asymmetry flags gated. Part 6 is about how much of that is the rig and how much is the solver.

The 3D path applies the same refusal logic inside itself. From pose-service/src/pose_service/pipeline/angles_from_3d.py, when the pointer is too short to have a meaningful direction the rotation is not reported:

_MIN_ROTATION_PERP_RATIO = 0.30

v_perp = v - float(np.dot(v, u)) * u
n_perp = float(np.linalg.norm(v_perp))
if n_perp < max(1e-6, _MIN_ROTATION_PERP_RATIO * nv):
    # Limb too straight: pointer direction is noise, not rotation.
    return 0.0

The ratio $r = ∣ v_{⊥} ∣/∣ v ∣$ is the sine of the angle the distal segment makes with the proximal segment’s long axis, so 0.30 is roughly a 17-degree bend at the knee. Below that, a straight leg cannot tell you which way the femur is twisted, and triangulation noise will happily pretend otherwise. The sibling constant _MIN_FLEX_PERP_RATIO = 0.45 does the same job for ball-joint flexion, and its measured justification is one of the sharpest tables in the repository: shoulder-flexion MAE by sagittal-projection ratio runs 4.69 degrees at a ratio above 0.80 (93.8% of readings) and 122.38 degrees below 0.15. Refusing below 0.45 costs 2.1% of readings and moves MAE from 6.21 to 5.10. The estimator did not get better at anything. It stopped emitting flexion angles the geometry cannot support.

#6. The elbow is the same effect with its sign reversed

In part 2 the elbow was the worst of the four joints, and I left it there as a curiosity. It is not a curiosity, it is the boundary showing through.

Angles from human annotations against angles from the detector’s keypoints, same image, same box (COCO val2017, n=2000, validation/README.md section 2):

Joint	MAE	Bias	Within 10°
Shoulder flexion	5.11°	−0.33°	88 / 100
Hip flexion	6.60°	−0.25°	83 / 100
Knee flexion	7.31°	−0.78°	83 / 100
Elbow flexion	10.49°	−0.28°	77 / 100
Mean	7.38°		83 / 100

The elbow is worst by a clear margin, and Rode et al. found elbow flexion worse than knee in both their 2D and their 3D families. The reason is anatomical rather than algorithmic: a forearm goes where it likes. A knee in sport is mostly doing sagittal work because that is what legs are for in running and jumping, while an arm crosses the body, reaches behind, points at the camera. The share of an elbow’s motion that lies in the image plane is simply lower, so the share the image plane discards is higher.

And then the mirror image, from the one dataset here with a genuine surround rig. LBMC gait, participant_02, aligned MAE over 10 random camera subsets at each count (section 8e):

Cameras	Elbow	Knee	Shoulder axial rotation
3	3.13°	2.80°	7.27°
4	2.16°	2.40°	5.07°
5	1.71°	2.33°	7.16°
6	1.66°	2.29°	6.72°

With all nine cameras on the full trial, elbow flexion reaches 1.73 degrees, and the six-camera subsets average 1.66 degrees: the lowest error this project has measured for any joint against an independent reference. The joint that suffers most from being flattened benefits most from not being flattened. Note also that the rotation column does not fall monotonically, 5.07 degrees at four cameras against 7.16 at five, which is what a barely-observable axis looks like when you resample the rig under it. That is the same physics as the 2D-beats-3D result running the other way, and it is why “use 2D” is a statement about a joint and a plane, never a statement about the method.

Two honest limits on that table. It is one participant, one trial and one task, and gait is not sport: an elbow in level walking barely moves, so a small aligned MAE there is not a promise about a bowling action. And the alignment itself removes a constant offset and, where the correlation is strongly negative, the sign, because two biomechanical models define neutral differently. The sign flip is reported and never silently applied. A harness that quietly flipped a sign would look exactly like one hiding a real bug.

#7. The lesson, stated for engineers who will never touch a camera

Strip the biomechanics out and the shape is general.

You have two representations of the same quantity. Representation B contains representation A plus one extra parameter. B is therefore capable of expressing everything A can express and more, which feels like a proof that B cannot be worse. It is not a proof, because capability is not accuracy. What decides the comparison is whether the extra parameter is observable in your data.

If it is observable, B wins and should. If it is not observable, B’s extra parameter has to be filled in from a prior, and now three things are true at once. The parameter carries error. That error propagates into the answer. And your evaluation cannot see it, because the parameter was never constrained by an observation, so there is no residual for it to leave.

I hit the same shape twice more in this project, from different directions. Confidence-weighted triangulation, which is standard practice, made the reconstruction 23% worse in millimetres (40.5 mm to 49.8 mm PA-MPJPE, the mean per-joint position error after a best-fit rigid alignment) while making its own reprojection error look better (3.21 px to 2.97 px), because RTMPose confidence reports how peaked the heatmap was, not how close the peak is to the joint, so a confidently mislocated keypoint gets more say. And the rig refinement cuts reprojection error by 89% to 99.9% on rigs whose geometry stays wrong. Both are section 4b and section 8c of validation/README.md, and both taught the same sentence: in multi-view pose, reprojection error is an objective, not a validation.

The general form of that sentence is: a quantity your model optimised cannot also be the quantity that validates it. A free parameter fitted against a residual will always drive that residual down. It tells you the fit converged. It tells you nothing about whether the fit is true. And a monocular depth estimate is the limiting case, a free parameter with no residual at all.

#8. What this does not license

The above is a design argument supported by measurement, not a clinical claim.

There is no in-vivo validation anywhere in this programme. Every figure I have quoted is agreement with a public dataset’s own reconstruction, not with a goniometer (the hinged protractor a clinician holds against a limb) on a real athlete. Test-retest reliability and minimal detectable change are unmeasured, so “did this athlete actually change?” has no defensible threshold yet. And the gap that bears directly on this post: monocular limb foreshortening is not gated. Section 4b’s projection-ratio gate works in the 3D path because the ratio is directly computable there. The monocular path has the identical failure mode, a thigh pointing at the camera projecting to a handful of pixels whose direction is noise, and it currently rejects only a zero-length limb. The obliquity estimate partly covers it, since foreshortening and high obliquity coincide, but a per-limb gate against anatomically expected length would be tighter. Identified, not implemented.

The study that would settle it is written down: 15 to 20 athletes, a guided protocol per metric, reference goniometry by two raters, reporting MAE, bias, Bland-Altman limits of agreement and test-retest MDC per metric per side. Until that exists, this system supports a clinical decision and does not make one. It is not a medical device.

#The short version

A monocular 3D lifter contains everything the 2D image-plane path contains plus a depth coordinate per joint, and on sagittal flexion it still loses. The published comparison it rested on (Rode et al., Scientific Reports 2025, 11 estimators against 27-camera Vicon) found 9.3 degrees best-case knee-flexion MAE among 2D configurations against 14.1 degrees among 3D.
The mechanism: from one camera, every depth along the ray reprojects to the same pixel, so the depth is supplied by a prior and leaves no residual. For motion already in the image plane it can only add error.
My own numbers on the shipped 2D path: 19.44 degrees against ASPset-510’s 3D truth (n=1986), 11.72 degrees with noise-free projections, and 7.69 degrees on the near-side-on band. The detector’s own contribution is 7.38 degrees, measured independently on COCO val2017.
The boundary is not accuracy, it is presence. Axial rotation is a twist about the limb’s own axis and one camera cannot see it, so hip ARC and shoulder ROM report not-measured rather than an estimate.
Multi-view only partly fixes it, and how much depends on the rig, not the code: 41.3 degrees axial MAE on ASPset’s shallow arc, 6.9 on Panoptic’s 21-view dome, 4.27 on LBMC’s nine-camera surround.
The elbow being the worst joint in part 2 (10.49 degrees against 7.31 for the knee) is the same effect showing through, and it reverses with a real rig: elbow flexion reaches 1.73 degrees aligned MAE on LBMC with nine cameras, the lowest error this project has measured for any joint against an independent reference.
The generalisable lesson: a richer representation whose extra parameter is unobservable in your data can lose to a poorer one that never claims it. And the parameter your solver optimised cannot be the number that validates it.

Next in this series: facing and sign, where a single left-or-right decision sets the sign of every sagittal angle at once, which means getting it wrong produces a mirrored measurement rather than a slightly wrong one.