Visualisation
The holographic HMD provided by Creal is made of a light field display projecting 32 images with micro-parallax to each eye, i.e. each image shows a slightly different perspective view compared to its adjacent ones, ensuring altogether Holographic Vision (cf. the prefix “Hovi” in HoviTron) and eye accommodation to any perceived object in the scene.
The system therefore exhibits two levels of parallax: one that is creating stereoscopic views by an advanced depth-based interpolation (aka virtual view synthesis) between camera views (at large inter-camera distance/baseline), and another one that is creating the 32 micro-parallax views within each eye.
The large baseline view synthesis
computes a virtual view from a couple of camera views with depth images having grey-scale pixel values that allow to push the 2D camera colour pixels into 3D space. Any virtual viewpoint to the scene can then be rendered by perspective reprojection of this 3D scene. Special precautions are, however, needed to avoid cracks and holes in the virtual viewpoint image:
-
Adjacent input pixels are interconnected with triangles, creating an implicit 3D mesh of the scene, for crack-free 3D perspective rendering in OpenGL.
-
Multiple input images on both sides of the virtual viewpoint fill large disocclusion holes.
The corresponding software tool has been developed within the MPEG Immersive Video (MIV) standardisation activities (where MPEG stands for “Moving Picture Expert Group”) and is available as Reference View Synthesiser (RVS) at this link, with an interactive website here. We have further developed a proprietary Real-time Accelerated View Synthesizer (RaViS) version of RVS for stereoscopic Head Mounted Displays (HMD).
This tool may also be used for the micro-parallax view synthesis that supports Holographic Vision (cf. Hovi in HoviTron), but since 32 times more images must be rendered at high framerates in the Light Field HMD, a proprietary memory-optimized version was developed: the Spatio-Temporally Amortized Light-Fields (STALF) renderer. It delivers images that are projected through a miniature hardware projector at very high framerate, virtually creating light wavefronts (perpendicular to the light rays) that enable Holographic Vision.
Further useful links:
MPEG MIV: https://mpeg-miv.org/
Reference View Synthesizer (RVS): https://gitlab.com/mpeg-i-visual/rvs
RVS Interactive Website: https://lisaserver.ulb.ac.be/rvs/
Capture
