ViVoNets Lab

Welcome to the Video and Voice over Networks (ViVoNets) Group at UCSB

High Dynamic Range (HDR) Video

Loading video ...	Sunlight from a nearby window brightens half of the user's face. HDR removes most of these oversaturated pixels, while maintaining good color and texture in the darker regions.
Loading video ...	Bright pixels cause standard auto-exposure to produce a severely underexposed face. HDR allows the exposure time to be increased, without increasing the number of saturated pixels within the face (note that the nose appears at the same brightness in the low dynamic range and HDR frames).
Loading video ...

 

S. Mangiat and J. D. Gibson, Inexpensive High Dynamic Range Video for Large Scale Security and Surveillance,
MILCOM, Baltimore, MD, Nov 2011. [pdf]

S. Mangiat and J. D. Gibson, Spatially Adaptive Filtering for Registration Artifact Removal in HDR Video,
IEEE International Conference on Image Processing (ICIP), Brussels, Sept 2011. [pdf]

S. Mangiat and J. Gibson, High Dynamic Range Video with Ghost Removal,
SPIE Optical Engineering & Applications, August 1-5, 2010. [pdf]

Loading video ...	Loading video ...
Loading video ...

 

Most consumer cameras capture only a small fraction of the brightness variation that occurs in everyday life. A typical 24-bit color sensor can reproduce 256 levels per channel, whereas an outdoor sunlit scene may require on the order of 10,000 levels. The result of this limited dynamic range is saturated pixels and poorly exposed images. Auto-exposure mechanisms try to minimize the number of saturated pixels or correctly expose a region of interest such as a person's face, yet they fail to correctly expose the entire frame. This commonly leads to white skies and shadowy foregrounds.

Low Dynamic Range: Long Exposure	Low Dynamic Range: Short Exposure	High Dynamic Range

 

High Dynamic Range Imaging (HDRI) uses hardware or software methods to produce images with expanded dynamic range. The most common method for HDR still photography combines multiple exposures of a scene. In this way, the bright regions will be captured in the shorter exposures while the dark regions are captured in the longer exposures. Using pixel values, their corresponding shutter times, and the camera response function, one can estimate a scene-referred, high dynamic range radiance map. This HDR radiance map can then be mapped back to displayable range on low dynamic range media using "tone mapping".

The method described above works well for still photography, but is not directly applicable to video because pixels must correspond exactly between the different exposures. Any motion will lead to ghosting artifacts. To address this, we have created a new method to produce HDR video using a camera that captures frames with alternating short and long exposures. Motion estimation is used to map adjacent frames onto the current frame. However, due to occlusions and non-overlapping regions, motion estimation will fail to eliminite all ghosting. We therefore use simple block-based motion estimation, and enhance these results using color information in areas with good correspondence and the edge information of the current frame to find and fix poorly registered regions. The result is improved dynamic range for videos with fast local motion and/or occlusions.

A sample video is shown below, with the alternating exposure input video on the left and our high dynamic range output on the right. In this video, a camera sits in a shadow as two cars pass (one in the shadow and the other in direct sun). Using a single exposure, it is impossible to correctly expose both cars. However, combining the information from a short and long exposure not only correctly exposes both cars simultaneously, but it brightens the yellow car with respect to the long exposure. The output has vibrant colors throughout the frame, despite the poor lighting conditions.

Loading video ...

HDR Video Results

HDR for Video Conferencing

High dynamic range is particularly useful for video conferencing, which often uses inexpensive camera sensors. Sunlight from nearby windows or bright indoor lights can overexpose a portion of the user's face. For mobile video conferencing on a handheld, this situation is worsened by direct sunlight. Pixel saturation is analogous to audio clipping in a phone call, and severely detracts from the sense of immersion and the overall quality of interaction. High dynamic range video may help lead to a significant increase in the adoption of video conferencing as the standard means of communication. Another issue common in video conferencing is that the exposure can change simply due to the motion of the user, as shown in the video below captured using auto exposure. Using an HDR method, the lighting becomes consistent despite significant motion.

Loading video ...	Loading video ...
Auto Exposure Video Conferencing	HDR Video Conferencing

Through a series of complexity reductions and parallelizations, we have made a real-time implementation of our algorithm on an 8-core server. This provides a solution to cloud-based applications such as the HDR video communications between a smartphone and a telepresence room. A video below visually compares the resulting quality with that of our original MATLAB implementation.

Loading video ...