geofront - Early Projects : MPEG Z/Alpha, Relief Texture, Gstreamer 3D Video (gstsert)

This page has been created to document MPEG Z/Alpha, an MPEG2 video file extension for the storage of depth and transparency information.

Presentation, Examination talk

MSc thesis

For an overview of the format's inner workings, please read Gernot Ziegler's diploma thesis on this subject.

Videos

These movies, created with the TexMPEG library, show possible applications of the file format.

Video grab of realtime MPEG Z/Alpha viewer

Video grab of realtime MPEG Z/Alpha viewer (shows standard OpenGL rendering limitations for depth maps)

Relief texture merger of several images with depth maps (movie shows only merging of still RPF images)

Relief texture merger of several images with depth maps (movie shows only merging of still RPF images)

Overlaying of/Intersection with realtime graphics (movie shows only still images)

Background music by Lackluster (Can'O'lard: Jugglers and Can'o'Lard: Tardy) and Dennis Gustavsson aka Void Main (Artificial Intelligence). Thank you for the permission!

If you have any questions on MPEG Z/Alpha, feel free to mail to info@geofront.eu.

Recently, there has been increased interest in the usage of streaming video textures in 3D applications. Streaming video textures are textures that continously change, mostly based on media data stored in a common video format.

The demo movies below demonstrate how the media component framework GStreamer and OpenGL 1.5 for Nvidia cards can be used to create surprising real-time video effects in Linux.

Figure 1. Video rendered as 3D pixels, each pixel is offset from its 2D video position, based on its luminance value.
Video footage from Singularity by Satori and Lackluster.
Conceptual idea based on Justin Manor's Key Grip system. See also his diploma thesis on Realtime Video Performance.

Technical background

Gstreamer provides YUV planes of the video frames that are uploaded as 3 luminance only textures (using Pixel Buffer Objects, PBO).
Instead of drawing a simple quad with these textures, an array of point primitives is being generated, one for each video pixel.

A Vertex Buffer Object (VBO), bound to the same memory area as the PBO, holds the original 2D positions of the points, and grants access to the Y (luminance) values.

A vertex shader reads the point array, and displaces them according to the provided Y values. The following fragment shader reconstructs the RGB pixel colors from the 3 given luminance textures, using a commonly known YUV->RGB conversion formula.

It is important to note that this video effect is not the only one achievable with this technology.
The software is able to provide PAL resolution (720x576) video textures at full framerate (25 Hz) on a decent PC. Only the GPU resources and the programmer's OpenGL capabilities set limits for what can be done, as soon as the texture is available in the 3D environment.

The sourcecode is available on request.

Comments and ideas are very welcome!

TexMPEG is an advanced MPEG2 software decoder for IRIX and Linux systems, able to utilize multiple CPUs and able to play several MPEG files simultaneously (see picture).


A simple OpenGL demo, 150 lines

It fits perfectly as an MPEG2 player for all applications where it is neccessary to retain control over the video output, such as:

This page has been created to document MPEG Z/Alpha, an MPEG2 video file extension for the storage of depth and transparency information.

Presentation, Examination talk

MSc thesis

For an overview of the format's inner workings, please read Gernot Ziegler's diploma thesis on this subject.

Videos

These movies, created with the TexMPEG library, show possible applications of the file format.

Video grab of realtime MPEG Z/Alpha viewer

Video grab of realtime MPEG Z/Alpha viewer (shows standard OpenGL rendering limitations for depth maps)

Relief texture merger of several images with depth maps (movie shows only merging of still RPF images)

Relief texture merger of several images with depth maps (movie shows only merging of still RPF images)

Overlaying of/Intersection with realtime graphics (movie shows only still images)

Background music by Lackluster (Can'O'lard: Jugglers and Can'o'Lard: Tardy) and Dennis Gustavsson aka Void Main (Artificial Intelligence). Thank you for the permission!

If you have any questions on MPEG Z/Alpha, feel free to mail to info@geofront.eu. A TexMPEG is an advanced MPEG2 software decoder for IRIX and Linux systems, able to utilize multiple CPUs and able to play several MPEG files simultaneously (see picture).


A simple OpenGL demo, 150 lines

It fits perfectly as an MPEG2 player for all applications where it is neccessary to retain control over the video output, such as:

Documentation

For an overview of the software's inner workings, please read Gernot Ziegler's diploma thesis on this subject.

You can also take a look at the Presentation Paper in the MPEG Z/Alpha section (to the left).

The developer documentation has been assembled from the Doxygen style comments in the sourcecode.

Support

TexMPEG is starting to age, but could be remodeled to new architectures if there is sufficient interest from new users/developers. If you run Linux and own an NVidia graphics card, then we can instead offer you the source code a nice tool called ff3d which is able to stream video textures from any video format, and which is based on ffplay and thus comparably easy to understand. Please contact us for further information.

Availability

TexMPEG is OpenSource software, and is thus freely available. However, future development (and debugging) and the implementation of features are dependent on available time. We therefore kindly ask you to consider assisting with development and/or documentation time to advance this project if you use this software in a production environment (both academic and commercial). Please contact Geofront for more information.

ViewCVS: Visit http://cvs.lysator.liu.se/viewcvs/viewcvs.cgi/?cvsroot=aquamarijn

Download over CVS:
Anonymous access possible.
Use :pserver:anonymous@cvs.lysator.liu.se:/cvsroot/aquamarijn as CVSROOT and press enter when asked for a password. Then, run "cvs checkout texmpeg".

More info on CVS: CVS-Manual Further, you will require the following libraries:

Lib/developing withLinux tarIRIX tar
LibPNG 1.2.3 1.2.4
LibMAD 0.14.2 0.14.2
OpenPerformer (only for test apps) 2.5.1 2.4
libao2 (audio output) included in source package included in source package
libac3 (AC3 decoder) included in source package included in source package
pthreads included in all dists included
libmpeg2 (MPEG2 video codec) 0.3.1 0.3.1 (?)

Geofront - The *sert appliations

Recently, there has been increased interest in the usage of streaming video textures in 3D applications. Streaming video textures are textures that continously change, mostly based on media data stored in a common video format.

The demo movies below demonstrate how the media component framework GStreamer and OpenGL 1.5 for Nvidia cards can be used to create surprising real-time video effects in Linux.

Figure 1. Video rendered as 3D pixels, each pixel is offset from its 2D video position, based on its luminance value.
Video footage from Singularity by Satori and Lackluster.
Conceptual idea based on Justin Manor's Key Grip system. See also his diploma thesis on Realtime Video Performance.

Technical background

Gstreamer provides YUV planes of the video frames that are uploaded as 3 luminance only textures (using Pixel Buffer Objects, PBO).
Instead of drawing a simple quad with these textures, an array of point primitives is being generated, one for each video pixel.

A Vertex Buffer Object (VBO), bound to the same memory area as the PBO, holds the original 2D positions of the points, and grants access to the Y (luminance) values.

A vertex shader reads the point array, and displaces them according to the provided Y values. The following fragment shader reconstructs the RGB pixel colors from the 3 given luminance textures, using a commonly known YUV->RGB conversion formula.

It is important to note that this video effect is not the only one achievable with this technology.
The software is able to provide PAL resolution (720x576) video textures at full framerate (25 Hz) on a decent PC. Only the GPU resources and the programmer's OpenGL capabilities set limits for what can be done, as soon as the texture is available in the 3D environment.

The sourcecode is available on request.

Comments and ideas are very welcome!