Video-based services involved in tactile internet (vehicle teleoperation, remote surgery…) require data exchange with an end-to-end latency of a few milliseconds. While 5G provides significantly reduced latency for the access network, video acquisition, coding, long-range transmission, and decoding delays become prevailing. ZL-LVC aims at proposing low-to-Zero Latency (ZL) video coding and delivery schemes, giving an end-user the feeling to watch pictures at the time they are remotely acquired.
For that purpose, latency will be compensated thanks to temporal frame extrapolation at the coder or/and at the decoder. At the encoder, future frames are obtained via extrapolation, encoded and transmitted. They are then decoded and displayed while the corresponding frames are acquired at encoder. At the decoder, frames are extrapolated from previously received frames to compensate latency.
With time-varying channels, encoding rate adaptation induces delivery delay jitter. Linear Video Coding (LVC) only relies on linear operators and delivers video to the decoder with a quality commensurate to the transmission channel’s. Moreover, LVC does not require any adaptation of the coding parameters to the channel characteristics, nor retransmission of corrupted data, which is beneficial in terms of latency.
ZL-LVC will combine LVC and extrapolation schemes. However, this will require reducing the encoding latency of LVC systems, which exploit temporal redundancy via 2D+T DCT over a group of pictures (GoP). This leads to a latency of the duration of the GoP and requires extrapolation on a relatively long horizon with a reduced overall quality.
The considered use case involves sport cars driven on tracks and streaming video to base stations around the track. The aim is to propose a ZL video coding scheme compatible with vehicle teleoperation.

 

In work package (WP) 1, baseline ZL schemes will be developed. The conventional video encoder design choices are explored that would best fit with a ZL scheme. Extrapolation at the encoder and/or at the decoder will be compared in point-to-point communication. To reduce the encoding latency of LVC schemes, while preserving their efficiency, image extrapolation combined with source coding techniques with side information at the decoder will be considered.
WP2 addresses the design of extrapolation techniques accounting for compression and channel noise. A joint end-to-end optimization of the LVC video codec and frame extrapolation will be performed by representing them with an auto-encoder trained using perceptual loss measures. This approach will benefit from our previous experience on video compression using deep learning techniques.
WP3 considers the design of an hybrid ZL scheme benefiting from the best of both conventional and LVC worlds. The adaptation of coding parameters of ZL schemes will be considered in response to variation of channel characteristics, so as to optimize the latency-rate-distortion trade-off.
WP4 is dedicated to the ZL-LVC software library and demonstrators. ZL schemes will be compared, based on conventional video codecs, involving a 4G/5G access network, or involving an LVC implemented on a software-defined radio platform. WP0 will specify the simulation and demonstration conditions.
Finally, WP5 is devoted to results dissemination & consortium management.
ZL-LVC will provide original solutions for ZL video coding and transmission. This will be an ideal solution for applications where delays and robustness are critical, such as vehicle teleoperation. Results of ZL-LVC will be published on relevant scientific venues, presented in MPEG meetings, and patented. They will also lead to the development of solutions by the industrial partner to better address the market of low-delay and high reliability video delivery. Experience acquired within ZL-LVC will provide us the opportunity to initiate new collaborative projects of broader scope.