Rendering Mediapipe Hands in Three JS for a Physics Simulation

Question

I'm looking to use Handsfree.js and Three to create a game where you can pick up and move around blocks with your hand. I'm wondering where to start: I have a basic understanding of Three but don't know how I should approach this project. I'm trying to render and animate a hand dynamically based on the coordinates that Mediapipe outputs. I've looked into Three's Skinned Mesh, but I'm not sure if it would be feasible to control and entire hand using it. I've also tried rigging a hand in Blender and then importing it in Three js, but I couldn't find any documentation on how to control imported rigs. How should I go about dynamically animating a hand? Also, how would I add physics to a hand that I import in? Is Three the right tool for the job? I've also tried using unity, but can't find any libraries that output 3d coordinates for the hands.

(for reference, mediapipe outputs an array of Vector3s with the positions of each joint on the hand)