LogIn
The method enables the team to render high-fidelity personalized hands that can be animated in real-time under novel illumination.
A team of researchers from Meta's Reality Labs Research has presented RelightableHands, a new neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination.
Leveraging the teacher-student framework, with the teacher learning appearance under a single point light from images captured in a light-stage, allowing the team to synthesize hands in arbitrary illuminations albeit with heavy computing, and the student model using the teacher model as training data and directly predicting appearance under natural illuminations in real-time, the team was able to outperform expectations and achieve photo-realistic relighting of two interacting hands at real-time speeds.
"Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage, allowing us to synthesize hands in arbitrary illuminations but with heavy compute. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time," writes the team. "To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead. Our key insight is that these features have strong correlation with subsequent global light transport effects, which proves sufficient as conditioning data for the neural relighting network."
You can read the original research paper here. And don't forget to join our 80 Level Talent platform, our Reddit page, and our Telegram channel, follow us on Instagram and Twitter, where we share breakdowns, the latest news, awesome artworks, and more.