Customized Integration#

You can prototype a new RGB-D volumetric reconstruction algorithm with additional properties (e.g. semantic labels) while maintaining a reasonable performance. An example can be found at examples/python/t_reconstruction_system/integrate_custom.py.

Activation#

The frustum block selection remains the same, but then we manually activate these blocks and obtain their buffer indices in the Hash map:

60# examples/python/t_reconstruction_system/integrate_custom.py
61        depth = o3d.t.io.read_image(depth_file_names[i]).to(device)
62        extrinsic = extrinsics[i]
63
64        start = time.time()
65        # Get active frustum block coordinates from input
66        frustum_block_coords = vbg.compute_unique_block_coordinates(
67            depth, intrinsic, extrinsic, config.depth_scale, config.depth_max)
68        # Activate them in the underlying hash map (may have been inserted)

Voxel Indices#

We can then unroll voxel indices in these blocks into a flattened array, along with their corresponding voxel coordinates.

72# examples/python/t_reconstruction_system/integrate_custom.py
73        synchronize()
74        end = time.time()

Up to now we have finished preparation. Then we can perform customized geometry transformation in the Tensor interface, with the same fashion as we conduct in numpy or pytorch.

Geometry transformation#

We first transform the voxel coordinates to the frame’s coordinate system, project them to the image space, and filter out-of-bound correspondences:

80# examples/python/t_reconstruction_system/integrate_custom.py
81
82        # Now project them to the depth and find association
83        # (3, N) -> (2, N)
84        start = time.time()
85        extrinsic_dev = extrinsic.to(device, o3c.float32)
86        xyz = extrinsic_dev[:3, :3] @ voxel_coords.T() + extrinsic_dev[:3, 3:]
87
88        intrinsic_dev = intrinsic.to(device, o3c.float32)
89        uvd = intrinsic_dev @ xyz
90        d = uvd[2]
91        u = (uvd[0] / d).round().to(o3c.int64)
92        v = (uvd[1] / d).round().to(o3c.int64)
93        synchronize()
94        end = time.time()
95
96        start = time.time()
97        mask_proj = (d > 0) & (u >= 0) & (v >= 0) & (u < depth.columns) & (
98            v < depth.rows)

Customized integration#

With the data association, we are able to conduct integration. In this example, we show the conventional TSDF integration written in vectorized Python code:

  • Read the associated RGB-D properties from the color/depth images at the associated u, v indices;

  • Read the voxels from the voxel buffer arrays (vbg.attribute) at masked voxel_indices;

  • Perform in-place modification

 98# examples/python/t_reconstruction_system/integrate_custom.py
 99
100        v_proj = v[mask_proj]
101        u_proj = u[mask_proj]
102        d_proj = d[mask_proj]
103        depth_readings = depth.as_tensor()[v_proj, u_proj, 0].to(
104            o3c.float32) / config.depth_scale
105        sdf = depth_readings - d_proj
106
107        mask_inlier = (depth_readings > 0) \
108            & (depth_readings < config.depth_max) \
109        synchronize()
110        end = time.time()
111
112        start = time.time()
113        weight = vbg.attribute('weight').reshape((-1, 1))
114        tsdf = vbg.attribute('tsdf').reshape((-1, 1))
115
116        valid_voxel_indices = voxel_indices[mask_proj][mask_inlier]
117        w = weight[valid_voxel_indices]
118        wp = w + 1
119
120        tsdf[valid_voxel_indices] \
121            = (tsdf[valid_voxel_indices] * w +
122               sdf[mask_inlier].reshape(w.shape)) / (wp)
123        if config.integrate_color:
124            color = o3d.t.io.read_image(color_file_names[i]).to(device)
125            color_readings = color.as_tensor()[v_proj, u_proj].to(o3c.float32)
126
127            color = vbg.attribute('color').reshape((-1, 3))
128            color[valid_voxel_indices] \

You may follow the example and adapt it to your customized properties. Open3D supports conversion from and to PyTorch tensors without memory any copy, see PyTorch I/O with DLPack memory map. This can be use to leverage PyTorch’s capabilities such as automatic differentiation and other operators.