Hand & Finger Tracking [Windows]

Framework for Finger Joint Detection

The Hand & Finger Tracking package for Unity provides a system designed for high-performance detection of hand movements with low latency. The underlying AI is capable of tracking 22 specific finger joints per hand in real time. This tracking includes the wrist, palm, and the specific joints of the thumb, index, middle, ring, and pinky fingers. By identifying these specific landmarks, the system allows for detailed gesture recognition and interaction within a digital environment.

One of the core functional components of this tool is its ability to automatically distinguish between left and right hands. This automatic recognition simplifies the logic required for two-handed interactions, ensuring that the software can correctly assign tracking data to the appropriate hand model or input handler without manual intervention by the user.

Implementation Paths for Developers and Designers

The package is structured to accommodate different levels of development expertise through two distinct implementation methods. For those who prefer a programmatic approach, the creator provides a C# API described as simple and robust. This API allows for deep integration into existing codebases, giving developers direct access to the tracking data for custom gameplay mechanics or specialized UI controls.

Alternatively, the package includes no-code support intended for rapid prototyping or for users who prefer visual assembly. This is achieved through drag-and-drop prefabs that can be placed directly into a Unity scene to enable finger-tracking capabilities immediately. These prefabs handle the heavy lifting of connecting the camera feed or media source to the tracking logic, allowing for the creation of finger-tracking applications without writing custom scripts.

Visualization and Spatial Mapping

Data gathered by the AI can be visualized in two primary ways depending on the needs of the project. The 2D visualization mode is optimized for overlay applications, such as drawing joint landmarks directly over a live video stream or a static image. This is particularly useful for diagnostic tools or augmented reality interfaces where the user needs to see the alignment of the tracking data against the source media.

For projects requiring spatial awareness, the 3D visualization mode maps the detected landmarks into three-dimensional space. This allows the hand data to interact with other 3D objects within the Unity world, supporting spatial understanding and more complex physical interactions. The 3D samples provided in the package have been specifically updated to adjust to phone orientation, ensuring consistent spatial mapping even when the input device is rotated.

Input Source Flexibility and Media Support

The tracking system is not limited to live camera feeds. While it fully supports real-time tracking via webcams—including laptop cameras and connected phone cameras—it also processes pre-recorded media. This flexibility allows developers to test their systems against consistent data or build applications that analyze existing archives.

Webcam Input: Tracks landmarks from a live feed and can display the results on a standard Unity Canvas or within the 3D world space.
Video Files: The system can process various video formats including MP4, MOV, and AVI. This is useful for offline analysis or for applications that require processing recorded performances.
Static Images: Landmarks can be extracted from still pictures in PNG or JPEG formats, allowing for precise finger positioning analysis on individual frames.
Runtime Platforms: The asset is built specifically for Windows 10 64-bit or later and functions within the Unity Editor.

Technical Specifications and Compatibility

The package is designed to be compatible with a wide range of Unity setups. It supports Unity 2020.3 or later and was originally developed in version 2022.3.31. One of its strengths is its broad support for various render pipelines. It is compatible with the Built-in Render Pipeline, the High Definition Render Pipeline (HDRP), the Universal Render Pipeline (URP), and Custom Scriptable Render Pipelines (SRP).

The asset package includes 103 individual assets, encompassing the core tracking logic, the C# API, and several built-in demos. These demos are organized to show users how to implement tracking for each supported input type—webcam, video, and picture—providing a functional baseline that can be used to jumpstart project development. The setup process is designed to be efficient, with the creator noting that the built-in demos can be operational within seconds of importing the package.

Practical Application in the Unity Environment

When working within the Unity Editor, the plugin operates as a tool for both development and runtime execution on Windows x64 systems. Because it covers both 2D and 3D landmark tracking, it serves as a versatile utility for projects ranging from simple gesture-based menu navigation to complex 3D hand simulations. The inclusion of well-documented C# samples and the specific focus on low-latency performance makes it a viable option for real-time interactive installations and desktop-based software that requires precise hand input without the need for specialized wearable hardware.