Humans are social animals, and our clothing and accessories are key elements of self-expression and communication in our everyday lives. As social networking has gradually expanded into the online world, so has the need for digitization of clothing and accessories.

In the research paper titled MEGANE: Morphable Eyeglass and Avatar Network, researchers from the Australian National University and Meta focused on modeling eyeglasses, an everyday accessory for billions of people around the world. In particular, they argue that to achieve realism, it is not enough to model the glasses alone, and the interaction of the glasses with the face must also be considered.

From a geometric point of view, the glasses and the human face deform each other at the point of contact. Therefore, the shape of the glasses and the face cannot be determined independently. Similarly, their appearance is coupled through global light transport, and shadows and mutual reflections may appear and affect radiation. Computational methods to model such interactions are therefore necessary to achieve realism.

Original methods do not faithfully recreate all geometric and photometric interactions that exist in the real world, or animation results often suffer from view and time inconsistencies. Although emerging neural rendering methods achieve photorealistic rendering of human heads and general objects in a 3D consistent manner, related methods do not consider the interactions between objects and lead to untrustworthy object combinations.

In contrast to existing methods, the team aims to model the geometric and photometric interactions between spectacle frames and the human face in a data-driven manner based on image observations. To this end, they launched MEGANE.

This one-of-a-kind deformable and relightable glasses model represents the shape and appearance of the frame and its interaction with the human face. To support changes in topology and rendering efficiency, the researchers use a hybrid representation that combines surface geometry and volumetric 3D representations.

Since the hybrid representation provides an explicit correspondence between glasses, its structure can be subtly deformed according to the head shape. Most importantly, the model is conditioned by a high-fidelity generative model of the human head, allowing it to specifically target the deformation and appearance changes of the wearer.