Real-time Compression and Streaming of 4D Performances
SessionCapturing 4D Performances
Authors
Event Type
Technical Papers
TimeFriday, 7 December 20189:26am - 9:52am
LocationHall B5(2) (5F, B Block)
DescriptionWe introduce a realtime compression architecture for 4D performance capture that is two orders of magnitude faster than current state-of-the-art techniques, yet achieves comparable visual quality and bitrate. We note how much of the algorithmic complexity in traditional 4D compression arises from the necessity to encode geometry in a explicit model (i.e. a triangle mesh). In contrast, we propose an encoder that leverages implicit representations (namely a Signed Distance Function) to represent the observed geometry, as well as its changes through time. We demonstrate how SDFs, when defined in a local support (i.e. a block), admit a low-dimensional embedding due the innate geometric redundancies in their representation. We
then propose an optimization that takes Truncated SDF (i.e. TSDF) found in most rigid/non-rigid reconstruction pipelines, and efficiently project each TSDF block in the SDF latent space. This results in a collection of low entropy tuples that can be effectively quantized and symbolically encoded. On the decoder side, to avoid the typical artifacts of block-based coding, we also
propose a variational optimization that recovers the quantization residuals in order to penalize unsightly discontinuities in the decompressed signal. This optimization is expressed in the SDF latent embedding, and hence can also be performed efficiently. We demonstrate our compression/decompression architecture by realizing, to the best of our knowledge, the first system for streaming a real-time captured 4D performance on consumer-level networks.
then propose an optimization that takes Truncated SDF (i.e. TSDF) found in most rigid/non-rigid reconstruction pipelines, and efficiently project each TSDF block in the SDF latent space. This results in a collection of low entropy tuples that can be effectively quantized and symbolically encoded. On the decoder side, to avoid the typical artifacts of block-based coding, we also
propose a variational optimization that recovers the quantization residuals in order to penalize unsightly discontinuities in the decompressed signal. This optimization is expressed in the SDF latent embedding, and hence can also be performed efficiently. We demonstrate our compression/decompression architecture by realizing, to the best of our knowledge, the first system for streaming a real-time captured 4D performance on consumer-level networks.