NYU Tandon
Applications

Coding and Streaming of Point Cloud Video

Yong Liu · Yao Wang · R. Luke DuBois · Todd Bryant

Overview

Point cloud video captures a scene in 3D, so a viewer can move around a performer and watch from any angle. It needs far more bandwidth than 2D video and lower latency, and decoding and rendering it is computationally heavy for both the sender and the viewer. This NSF-funded project designs coding, streaming and edge processing together so that point cloud video can be streamed to ordinary Internet-connected devices.

The project has four thrusts: learned point cloud video coding that adapts to the available rate and the viewer’s field of view; progressive streaming that refines each region of the predicted field of view as its playback time approaches; edge caching that works with edge-based post-processing; and a streaming testbed used for modern dance education. The team also released DanceNet3D, a multi-view 3D dance dataset.

Point cloud videoVolumetric video streaming3D Gaussian splattingLearned compressionField-of-view predictionDance education
A dancer on a multi-camera volumetric capture stage, with the live 3D capture shown on the monitor in front.
Point cloud video is coded as a geometry octree with scalable latent features, which a network decodes into 3D Gaussians for rendering (NYU Video Lab).

Coding and rendering point cloud video

Each frame is coded as an octree for geometry plus scalable latent features, so a receiver can decode at the quality its bandwidth allows. A feed-forward network, P2ENet, turns the decoded points into 3D Gaussians that render in real time on a GPU (Hu et al., CVPRW 2024). Bits-to-Photon trains compression and rendering end to end, so bits go where they most improve the rendered image (Hu et al., ICIP 2025). Follow-on work adds temporal prediction between frames (U-Motion), textured surfel octrees (TeSO) and multi-resolution Gaussian decoding (SplatTree).

Point cloud video is coded as a geometry octree with scalable latent features, which a network decodes into 3D Gaussians for rendering (NYU Video Lab).

Frames from two DanceNet3D sequences, rendered from 3D Gaussian splatting reconstructions (DanceNet3D).

DanceNet3D dataset

DanceNet3D is a 3D dance dataset with multi-view video of each performance and a 3D Gaussian splatting reconstruction of every frame, for research on 3D human rendering, compression and dance analysis. The dataset paper received the Best Paper Award at the 3DMV Workshop at CVPR 2026 (Wei, Li et al., CVPRW 2026).

Frames from two DanceNet3D sequences, rendered from 3D Gaussian splatting reconstructions (DanceNet3D).

Dancers in motion-capture suits during a capture session.

Streaming to dance students

The project is building a point cloud video streaming testbed and will use it in modern dance education, streaming captures of professional dancers to dance students both on demand and live. It is a collaboration with the Mark Morris Dance Group and NYU Tisch (NYU Tandon news).

Dancers in motion-capture suits during a capture session.

Team

Faculty

Yong LiuYao WangR. Luke DuBoisTodd Bryant

Students

Tingyu FanRan GongYueyu HuChen LiTongyu ZongMingjian LiShihang WeiReese AnspaughMoira ZhangAdi SomaiahClaudia Cerdas VegaMarilyn Fang

Selected publications

DanceNet3D: A 3D Dance Dataset with Multi-View Videos and 3DGS Reconstructions S. Wei, M. Li, R. Gong, Y. Hu, Y. Wang CVPR Workshops (3DMV), 2026 (Best Paper Award) SplatTree: Voxel-Based Multi-Resolution Decoding for Feed-Forward 3D Gaussian Generation R. Gong, Z. Ren, T. Fan, Y. Hu, Y. Wang CVPR Workshops (3DMV), 2026 TeSO: Representing and Compressing 3D Point Cloud Scenes with Textured Surfel Octree Y. Hu, R. Gong, T. Fan, Y. Wang arXiv:2508.07083, 2025 Progressive Frame Patching for FoV-based Point Cloud Video Streaming T. Zong, Y. Mao, C. Li, Y. Liu, Y. Wang IEEE Transactions on Multimedia, 2025 Spatial Visibility and Temporal Dynamics: Rethinking Field of View Prediction in Adaptive Point Cloud Video Streaming C. Li, T. Zong, Y. Hu, Y. Liu, Y. Wang ACM Multimedia Systems Conference (MMSys), 2025 Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering Y. Hu, R. Gong, Y. Wang IEEE ICIP, 2025 U-Motion: Learned Point Cloud Video Compression with U-Structured Temporal Context Generation T. Fan, Y. Hu, R. Gong, Y. Wang arXiv:2411.14501, 2024 Low Latency Point Cloud Rendering with Learned Splatting Y. Hu, R. Gong, Q. Sun, Y. Wang CVPR Workshops (AIS), 2024