Art style

This technology can turn the real world into living art

Researchers can take an image and use it as a reference point to create a virtual world, object or person.

As companies consider having a presence in the metaverse via a digital twin, the ability to quickly and easily create stylized 3D content and virtual worlds will become increasingly important in the future.

A recent publication Cornell University The paper explored this growing trend and developed a solution to produce stylized neural radiation fields (SNeRF) that can be used to create a wide range of dynamic virtual scenes at higher speeds than traditional methods.

Using various reference images, the research team of Thu Nguyen-Phuoc, Feng Liuand Lei Xiao were able to generate stylized 3D scenes that could be used in a variety of virtual environments. For example, imagine putting on a VR headset and seeing what the real world would look like through a stylized lens such as a painting by Pablo Picasso.

This process allows the team not only to quickly create virtual objects, but also to use their real environment as part of the virtual world with 3D object detection.

Importantly, the research team was also able to observe the same object through different view directions at the same vantage point, otherwise known as cross-view coherence. This creates an immersive 3D effect when viewed in VR.

By alternating stages of NeRF optimization and stylization, the research team was able to take an image and use it as a reference style to then recreate a real-world environment, object, or person in a way that adapts the stylization of this image, speeding up the creative process.

“We introduce a new training method to solve this problem by alternating NeRF optimization and stylization steps”, says the team. “Such a method allows us to fully utilize the capacity of our hardware memory to both generate higher resolution images and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor, and dynamic scenes, and synthesizes new high-quality views with cross-view consistency.

Due to NeRF’s memory limitations, the researchers also had to solve another problem, namely how to render more high-definition 3D images at a speed that was more like real time. The solution was to create a loop of rendered views that, with each iteration, was able to target stylization points more consistently on each pass, and then reconstruct the image with more detail.

Technology has also improved avatars. The research team’s stylized SNeRF approach allowed them to create a more expressive avatar during conversations. The result is dynamic 4D avatars that can realistically convey emotions such as anger, fear, excitement, and confusion, all without having to use an emoji or press a button on a VR controller.

The research work is still ongoing, but for the moment the team has been able to develop a method for stylizing 3D scenes using implicit neural representations that impact their environment and their avatars. Additionally, their approach of using an alternate stylization method allowed them to take full advantage of their hardware memory capacity to stylize both static and dynamic 3D scenes, allowing the team to generate higher resolution imagery and to adopt a more expressive picture style transfer. virtual reality methods.

If you want to go deeper into the details, you can access their report here.

Image credit: Cornell University