Open3D 0.15: more than 500+ issues addressed

Open3D 0.15 Release Notes

We are excited to bring you the best Open3D yet - version 0.15. Take a look below:

Starting from this release, we adopt a "tick-tock" model for balancing resolving issues vs. adding new features. In a nutshell, the "tick" releases are focused on resolving existing issues and eliminating bugs, while the "tock" releases mainly focus on developing new features. Open3D 0.15 is a "tick" release. We resolved over 500 issues for Open3D and Open3D-ML, as the infographic below illustrates.

Issue_Stats

Google Summer of Code 2022

Open3D has applied for the Google Summer of Code 2022 to increase community participation. Check out details and our project ideas here. Please help in making Open3D better for all.

Build System

  • [New] We now provide Open3D binary packages for C++ users. No need to “Build from Source” - just download a binary package for your Operating System and use it “out of the box”. See GitHub releases for v0.15 and getting started guides for the latest development package.
  • [New] Docker build tools to build ARM64 Linux wheels and multiple Python versions. See ARM64 build guide for more details.
  • [New] Pre-compiled Open3D wheel for ARM64 Linux and macOS. Improved Apple Silicon support. Install Open3D on ARM64 Linux and macOS with pip install open3d.
  • [Update] Open3D now builds with the new CXX11 ABI by default on Linux. Set -DGLIBCXX_USE_CXX11_ABI=OFF in cmake if you need the old ABI, e.g. to work with PyTorch / TensorFlow libraries.
  • [Update] Starting with version 0.15, Open3D Conda packages are no longer supported. Install Open3D with pip install open3d inside a Conda virtual environment.

Core

Datasets

  • [New] Dataset module for automatically downloading and managing example data. The following example demonstrates how to create a Dataset object, extract its path, and display it in the Open3D Visualizer:
    Open3D 0.15 Dataset Demo

    import open3d as o3d
    
    if __name__ == "__main__":
        dataset = o3d.data.EaglePointCloud()
        pcd = o3d.io.read_point_cloud(dataset.path)
        o3d.visualization.draw(pcd)
    #include 
    #include 
    #include "open3d/Open3D.h"
    
    int main() {
        using namespace open3d;
    
        data::EaglePointCloud dataset;
        auto pcd = io::CreatePointCloudFromFile(dataset.GetPath());
        visualization::Draw({pcd});
    
        return 0;
    }

Command Line Interface (CLI)

  • [New] Open3D-dedicated Command Line Interface (CLI) for visualization and running Python examples. Below is a code snippet to get started with Open3D and its examples.

    # Install Open3D pip package
    pip install open3d
    
    # Print help
    open3d --help
    
    # List all runnable examples
    open3d example --list
    
    # Print source code of an example
    open3d example --show [category]/[example_name]
    
    # Run an example
    open3d example [category]/[example_name]
    
    # Run Open3D Viewer
    open3d draw
    
    # Open a mesh or point cloud file in Open3D Viewer
    open3d draw [filename]

    open3d_015_cli

  • [Update] Python examples directory has been refactored for better namespace consistency and new examples have been added.

Neighbor search

  • [Update] Updated neighbor search module. The neighbor search module is now equipped with highly optimized built-in implementations of all search methods (Knn, Radius, and Hybrid) supporting both CPU and GPU. Faiss build dependency is removed.

Visualization and GUI

  • [New] Introducing raw mode visualization. The raw mode automatically simplifies the lighting environment and object materials to make it easier to inspect the underlying geometry of point clouds and triangle meshes.
    Open3D 0.15 Raw Mode Demo
  • [New] Open3D new visualizer now features CPU Rendering based on Mesa’s LLVMpipe software OpenGL implementation. Interactive applications, demos, and Python scripts are all supported as well as off-screen rendering. This feature can also be used within Docker.

    • Method 1: LD_PRELOAD from the command line
      LD_PRELOAD=/home/open3d/development/mesa-21.3.4/libGL.so python examples/python/visualization/draw.py
    • Method 2: Preload library in Python

      import ctypes
      ctypes.cdll.LoadLibrary('/home/open3d/development/mesa-21.3.4/libGL.so')
      import open3d as o3d
      
      mesh = o3d.io.read_triangle_model('/home/open3d/development/FlightHelmet/FlightHelmet.gltf')
      o3d.visualization.draw(mesh)

      open3d_015_cpu_render

  • [New] WidgetProxy and WidgetStack widgets allow the creation of user interfaces on the fly (contributed by @forrestjgq).
  • [New] Background color option for button widgets (contributed by @forrestjgq).
  • [New] Set maximum visible items in a list widget to prevent lists from growing too large (contributed by @forrestjgq).
  • [New] Function keys are now bindable (contributed by @forrestjgq).
  • [New] Support for specifying intrinsic projection matrix in the new visualizer.
  • [New] Add support for scaling 3D labels.
  • [Fix] Open3D for TensorBoard plugin does not need Open3D-ML now.
  • [Fix] Point picking, text input, and UI layout (contributed by @forrestjgq).

Geometry

  • [Fix] Oriented Bounding Box
    • Fixed an issue where the orientation of the OrientedBoundingBox was mirrored.
    • [New] added a new parameter for robust oriented bounding box computation for degenerated point clouds.
  • [Fix] Convex hull meshes created from point clouds now have outward-pointing triangles.
  • [Update] Added a new parameter for robust convex hull computation.
  • [Update] TriangleMesh GetSelfIntersectingTriangles() and related functions like IsWatertight(), GetVolume(), etc. are now more than 4 times faster.
  • [Fix] Corrected an issue with io::AddTrianglesByEarClipping() where the algorithm could fail for concave polygons.
  • [New] New Python examples for reconstruction and voxelization.
  • [Fix] Improved logger performance.

Open3D-ML

  • [New] MIT-licensed implementation of RandLANet.
    open3D_015_randlanet
  • [New] Intel OpenVINO inference backend (contributed by @dkurt).
  • [Fix] Fixed an issue with S3DIS where the loss gets NaN after a few epochs.
  • [Fix] Fixed an issue with IoU calculation which fails for large point clouds while running inference in patches.
  • [Fix] Fixed an issue where the labels are not correctly ordered in the visualizer.
  • [New] Support for Images in Dataset Visualizer (contributed by @ajinkyakhoche).
    open3D_015_ml_image_vis

Acknowledgment

We would like to thank all of our community contributors for their true labor of love for this release!

@ajinkyakhoche @ceroytres @chunibyo-wly @dkurt @forrestjgq @Fuhrmann-sep @jeertmans @junha-l @mag-sruehl @maxim0815 @Nicholas-Mitchell @nigels-com @NobuoTsukamoto @ntw-au @roehling @theNded

Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Open3D 0.14 is full of new features

Open3D 0.14 Release Notes

We are excited to present the new Open3D version 0.14!

In this release, you will find:

  • TensorBoard visualization
  • Upgraded GUI API
  • New RayCastingScene class
  • Improved Tensor performance
  • Improved I/O performance
  • New 3D learning models in Open3D-ML: Point Transformer and PVCNN
  • Improved interoperability with Numpy

Changes in Installation and Build system

  • Open3D now works with Python 3.9. We release Open3D pre-compiled Python packages in Python 3.6, 3.7 3.8, and 3.9.
  • Open3D 0.14 is the last version that supports conda installation. Starting from version 0.15, users will need to install Open3D with pip install open3d. We recommend installing Open3D with pip inside a conda virtual environment.
  • Git submodules are no longer required in Open3D. You can now clone Open3D with git clone https://github.com/isl-org/Open3D.git without the --recursive flag. Also please note the updated Github URL.
  • Open3D will now build in Release mode by default if CMAKE_BUILD_TYPE is not specified. Python is no longer required for building Open3D for C++ users.
  • Open3D-ML is now recommended to be used along with PyTorch 1.8.2 and/or Tensorflow 2.5.2. Checkout Open3D-ML for more information.

Tensorboard Integration

Now you can use Open3D within Tensorboard for interactive 3D visualization! At a glance, you can:

  • Save and visualize geometry sequences and their properties. This enables interactive visualization and debugging of 3D data and 3DML model training.
  • Visualize 3D semantic segmentation and object detection with input data, ground truth, and predictions. In addition, any custom properties for a PointCloud, from scalar to vector, can be easily visualized.
  • Synchronize time steps and viewpoints during different runs. This helps debug and monitor the effect of parameter tuning.

Rich PBR materials

tensorboard_demo_scene

Object detection

tensorboard_objdet_full_2_vp9 webm

Semantic segmentation

tensorboard_sync_view_vp9

To get started, write some sample geometry data to a TensorBoard summary with this snippet:

from torch.utils.tensorboard import SummaryWriter  # TensorFlow also works, see docs.
import open3d as o3d
from open3d.visualization.tensorboard_plugin import summary
from open3d.visualization.tensorboard_plugin.util import to_dict_batch
writer = SummaryWriter("demo_logs/")
cube = o3d.geometry.TriangleMesh.create_box(1, 2, 4)
cube.compute_vertex_normals()
colors = [(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0)]
for step in range(3):
    cube.paint_uniform_color(colors[step])
    writer.add_3d('cube', to_dict_batch([cube]), step=step)

Now you can visualize this in TensorBoard with tensorboard --logdir demo_logs. For more details on how to use TensorBoard with Open3D, check out this tutorial.

Visualizer

Further enhancements have been added to the GUI viewer. Now you can:

  • Directly visualize tensor-based geometry classes including PointCloud, TriangleMesh, and LineSet.
  • Use physically based rendering (PBR) materials that deliver appealing appearance.
  • New default lighting environment and skybox improves visual appeal
  • Use all the functionality in Tensorboard!

    img

import open3d as o3d
import open3d.visualization as vis
a_sphere = o3d.geometry.TriangleMesh.create_sphere(2.5, create_uv_map=True)
a_sphere.compute_vertex_normals()
a_sphere = o3d.t.geometry.TriangleMesh.from_legacy(a_sphere)
# Compare this...
vis.draw(a_sphere)
a_sphere.material = vis.Material('defaultLit')
a_sphere.material.texture_maps['albedo'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Color.jpg')
a_sphere.material.texture_maps['roughness'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Roughness.jpg')
a_sphere.material.texture_maps['normal'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_NormalDX.jpg')
# With this!
vis.draw(a_sphere)

A complete, complex demo scene can be found at examples/python/gui/demo-scene.py

Core

  • The Open3D Tensor class received a major performance boost with the help of Intel ISPC compiler and optimization for the contiguous code path.
    img
    (See python/benchmarks/core for the benchmark scripts. For each operation, the geometric mean of run times with different data types is reported. The time is measured with an Intel i9-10980XE CPU.)
  • A major upgrade of Parallel HashMap is done. Now you can choose from multi-valued HashMap and HashSet depending your value types. A comprehensive tutorial is also available.
  • Linear Algebra performance have been optimized for small matrices, especially on 4x4 transformations.
  • Semantics for tensor and tensor-based geometry have been improved, especially on device selection.
  • Functions expecting a Tensor now accept Numpy arrays and Python lists. For example:

    import open3d as o3d
    import numpy as np
    
    mesh = o3d.t.geometry.TriangleMesh()
    mesh.vertex['positions'] = np.array([[0, 0, 0], [1, 0, 0], [1, 1, 0]], dtype=np.float32)
    mesh.vertex['colors'] = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.float32)
    mesh.triangle['indices'] = [[0, 1, 2]]
    o3d.visualization.draw(mesh)

I/O

  • We now support I/O from/to Numpy compatible .npz .npy formats for Open3D tensors and tensor maps. It is now easier to convert between Open3D geometry classes and Numpy properties.
  • We have improved I/O performance for tensor-based point cloud and triangle-mesh file-formats, including .ply, .pcd, .pts. Geometry loading time is hence improved for the stand-alone visualizer app.
  • We added support for material data to the MessagePack serialization format of the RPC module.

Geometry

  • We introduce a new class RaycastingScene with basic ray intersections functions and distance transform for meshes, utilizing the award-winning Intel Embree library.

    Example code for rendering a depth map:

    import open3d as o3d
    import matplotlib.pyplot as plt
    
    # Create scene and add a cube
    cube = o3d.t.geometry.TriangleMesh.from_legacy(o3d.geometry.TriangleMesh.create_box())
    scene = o3d.t.geometry.RaycastingScene()
    scene.add_triangles(cube)
    
    # Use a helper function to create rays for a pinhole camera.
    rays = scene.create_rays_pinhole(fov_deg=60, center=[0.5,0.5,0.5], eye=[-1,-1,-1], up=[0,0,1],
                                   width_px=320, height_px=240)
    
    # Compute the ray intersections and visualize the hit distance (depth)
    ans = scene.cast_rays(rays)
    plt.imshow(ans['t_hit'].numpy())

    Distance transform generated with RaycastingScene:

    See the tutorials for more information (Ray casting, Distance queries).

  • Normal estimation for tensor PointCloud is supported with the tensor-compatible nearest neighbor search modules.

  • Customizable tensor-based TriangleMesh, VoxelBlockGrid, and LineSet are implemented that allow user-defined properties. For example:

    import open3d as o3d
    import open3d.core as o3c
    
    mesh = o3d.t.geometry.TriangleMesh()
    mesh.vertex["positions"] = o3c.Tensor([[0.0, 0.0, 1.0],
                                         [0.0, 1.0, 0.0],
                                         [1.0, 0.0, 0.0],
                                         [1.0, 1.0, 1.0]], dtype=o3c.float32)
    mesh.vertex["my_custom_labels"] = o3c.Tensor([0, 1, 2, 4], dtype=o3c.int32)
    mesh.triangle["indices"] = o3c.Tensor([[0, 1, 2], 
                                         [1, 2, 3]], dtype=o3c.int32)

Pipelines

  • We have enhanced point cloud registration (ICP) with a tensor interface:
    • Float64 (double) precision point cloud is supported for a higher numerical stability
    • Robust Kernels, including Huber, Tukey, and GM losses are supported for robust registration.
    • Colored-ICP is now supported in the unified tensor-based API.
    • See this tutorial for more details.
  • We also provide an initial tensor-based reconstruction system in Python, including
    • Customizable volumetric RGB-D integration;
    • Dense RGB-D SLAM with a GUI;
    • See this tutorial for more details.

Open3D-ML

The Open3D-ML library welcomes more state-of-the-art models and operators that are ready to use for advanced 3D perception, especially semantic segmentation, including

  • New state-of-the-art Point Transformer for Semantic Segmentation.
    img
  • Highly Efficient Point-Voxel Convolution for Semantic Segmentation.
    img
  • RaggedTensor integration that enables batch SparseConvolution and SparseConvolutionTranspose along with PyTorch.
  • Batched voxelization for fast point-voxel conversions.

Refer to the tutorial for training and inference on new models. (PyTorch TensorFlow).

Expreimental ARM64 builds


We hope you find Open3D 0.14.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

Acknowledgment

We thank all the community contributors for this release!

(alphabetical order)
@cclauss
@chrockey
@chunibyo-wly
@cosama
@forrestjgq
@gsakkis
@junha-l
@ktsujister
@leomariga
@li6in9muyou
@marcov868
@michaelbeale-IL
@muskie82
@nachovizzo
@NobuoTsukamoto
@plusk01
@reyanshsolis
@ShreyanshDarshan
@ShubhamAgarwal12
@SoftwareApe
@stanleyshly
@stotko
@theNded
@zhengminxu

Open3D better than ever in our newest 0.13 release

Open3D 0.13.0 Release Notes

We welcome you to the 0.13.0 release of Open3D. This release is full of exciting new features with a strong emphasis in real-time pipelines, but also full of bug fixes and usability improvements. The big highlights of this release are as follows:

  • New real-time 3D reconstruction pipeline, featuring GPU and CPU support based on VoxelHashing.
  • New real-time point cloud registration algorithm, featuring a high-performance version of Iterative Closest Point (ICP).
  • New Neighbor Search module, introducing your favorite search algorithms such as KNN and RadiusSearch, with support for GPU and CPU devices through a common interface.
  • New web visualizer, which enables users to access the advanced rendering and visualization features of Open3D in your favourite web environments (remote and locally!), including Jupyter notebooks, Jupyter lab, and standalone web applications.
  • New 3D machine learning models and datasets, featuring PointRCNN for 3D object detection, SparseConvNets for point cloud semantic segmentation, and support for ScanNet and SunRGBD.
  • Upgraded GUI module, providing improved and more versatile versions of existing widgets, and new ones: ImageWidget and ToggleSwitch.
  • Upgraded build system, adding support for CUDA 11.

Real-time 3D reconstruction

We introduce a new CUDA accelerated pipeline including RGBD odometry, frame-to-model tracking, and volumetric integration.

image
Figure 1. Example of 3D reconstruction from an RGB-D sensor.

Odometry

We introduce the tensor based real-time RGBD Odometry pipeline. In addition to the legacy Hybrid and Intensity based methods, we support the popular point-to-plane method.

TSDFVoxelGrid

We further accelerate volumetric integration and introduce fast ray casting for rendering.

VoxelHashing

Based on the accelerated RGBD odometry and raycasting, we present the fully functional VoxelHashing system. It performs dense volumetric reconstruction with fast frame-to-model tracking. We present an easy-to-use GUI that also shows real-time interactable surface reconstruction.

SLAC

We have further enhanced our legacy offline reconstruction system by introducing the Simultaneous Localization and Calibration (SLAC) algorithm. This algorithm applies advanced dense multi-way registration along with non-rigid deformation to create highly-accurate reconstructions.

Real-time point cloud registration

We present a high-performance implementation of ICP using Open3D’ Tensor library. This module is one of the first on leveraging the new Neighbor search module and the newly crafted parallel kernels. This implementation brings support for multi-scale ICP, which allows us to do iterations on different resolutions in order to accelerate convergence while keeping computation low.

image
Figure 2. ICP registration of multiple point clouds from a driving dataset.

New Neighbor Search module

Neighbor search is at the core of many 3D algorithms. Therefore, it is critical to have access to a fast implementation able to execute a large number of queries in a fraction of a second. After months of development, the Open3D team is proud to present the new Neighbor Search module!

This module brings support for core search algorithms, such as KNN, Radius search, and Hybrid search. All these algorithms are provided with support for both CPU and GPU, through a common and easy-to-use interface. Write your code once and support multiple devices! Moreover, we have not sacrificed a single flop of computation, making this module one of the fastest neighbor search libraries ever created.

Web visualizer

The need for visualizing complex 3D data in web environments has surged considerably in the past few years, in part thanks to the proliferation of sensors like LIDAR and RGBD cameras. New use cases, such as online dataset inspection and remote visualization are now an integral part of many tasks, requiring the crafting of ad-hoc tools, which often are cumbersome to use.

image
Figure 3. Standalone visualization of a semantic segmentation model in a browser.

In order to improve this situation, we introduce our new web-based visualization module, which enables 3D visualization from any browsers and any location. This module lets users run advanced rendering and visualization pipelines, both remote and locally through your web browser. All the power of Open3D’ rendering engine --including support for PBR materials, multiple lighting systems, 3D ML visualization, and many other features--, are now supported in your browser. This module also includes a Jupyter extension for interactive web-based visualization! This new feature allows you to run compute-intensive 3D processing in a dedicated server while visualizing the results remotely on any device through your browser.

image
Figure 4. Visualization of a 3D model on a Jupyter notebook.

3D Machine Learning

Architectures

In this release, we introduce a new point cloud semantic segmentation architecture based on a Sparse Convolution-based UNet model. This architecture leverages the new sparse convolution operators provided by Open3D, and achieves state of art performance for Semantic Segmentation on the ScanNet dataset. We have also added support for PointRCNN for the task of 3D object detection. To enable PointRCNN we have added new operators like furthest_point_sampling, three_interpolate, ball_query, which are available through Open3D for TensorFlow and Pytorch.

image
Figure 5. Example of 3D semantic segmentation using a SparseConvUNet model on ScanNet.

image
Figure 6. Example of 3D object detection using a PointRCNN on KITTI.

All these models are provided with their respective training and inference pipelines with support for TensorFlow and Pytorch. Pre-trained models are also provided (check out the following link).

Datasets

This release brings the following datasets: Scannet and SunRGBD.

Frameworks

We now support all models on newer versions of TensorFlow (2.4.1) and PyTorch (1.7.1), on CUDA 11.0.

GUI and rendering modules

Open3D 0.13.0 brings a cascade of improvements and fixes to the renderer and GUI modules.

  • The camera can now be controlled with respect to a target object, and the pan-in/pan-out actions are smoother.
  • The render supports render targets.
  • The black screen issue in MacOS systems is now solved.
  • Support for a video widget
  • The background of the 3D scene widget can be updated with an image.
  • Camera objects can be visualized in the 3D scene.
  • Camera objects support intrinsic and extrinsic parameters.
  • Support for multiple fonts.
  • Support for a new ToggleSwitch widget.
  • New GUI/rendering examples for common tasks, such as data acquisition from RGBD sensors, 3D reconstruction, and registration, among others.

Build System

Our pip packages now include support for CUDA 11.0, PyTorch 1.7.1, and TensorFlow 2.4.1 to enable RTX 3000 series devices. Please, notice that we provide custom PyTorch wheels for Linux to work around an incompatibility between CUDA 11, PyTorch, and extension modules such as Open3D-ML.

This release also brings new improved support for CUDA on Windows. Users can now build CUDA accelerated Python wheels for Windows. Open3D is now built with security options enabled by default.


We hope you find Open3D 0.13.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

The Open3D team


Changelog:

Deprecating

  • All visualization tools, such as draw_geometries will be deprecated in Open3D 0.14 in favor of the new O3DViewer.

Changes to Open3D

  • add ml deps for conda build (#2826)
  • tensorkey refactoring: pimpl and optional arguments (#2813)
  • Add file and line for error message (#2829)
  • color map optimization refactoring (#2832)
  • remove openmp macro guards for color map optimization (#2835)
  • easy way of initializing c++ tensor (#2796)
  • Add numerical operations to Tensor (#2848)
  • Implement np style arange for Tensor (#2849)
  • Refactor TSDF kernels (#2854)
  • core::Scalar class and refactor arange (#2857)
  • octree updates (#2833)
  • numpy-compatible tensor IO (#2859)
  • Upgrade zlib and libpng, use ExteralProject (#2878)
  • update mkl download links (#2882)
  • Python RSBagReader.open() crash on Ubuntu (#2876)
  • refactoring: add pipelines/kernel folder (#2875)
  • Standarize To() and Copy() (#2877)
  • compile 3rd-party library in a separate step for arm64 CI (#2891)
  • Hardened security flags for building Open3D (#2883)
  • fix incompatible CUB include dir with CUDA 11 and later (#2913)
  • Jsoncpp with ExternalProject (#2907)
  • Python example of using a texture on a Material (#2935)
  • Add rendering for VoxelGrid and Octree to new renderer (#2932)
  • Enable RSUSB backend only on Linux. (#2923)
  • Added C++ example OffscreenRendering (#2914)
  • Lazy load renderer when creating window in case Python crashes before window becomes visible (#2819)
  • Fixed trackpad scrolling to be buttery smooth at slow speeds on macOS (instead of being very discrete-seeming) (#2964)
  • add tunable parameter to filter outliers in surface extraction (#2931)
  • Added close callback to window (#2958)
  • Mode For Light To Follow Camera (#2944)
  • Image processing Ops sample with IPP and NPP (#2898)
  • Added ability to set camera in open3d.visualization.draw() (#2936)
  • Mouse movement by spherical coordinates (#2962)
  • Refactored RPC interface to be more amenable to variation in visualizers (fixes add not updating) (#2970)
  • Optimised kernel for ICP (#2917)
  • Add support for FPS-style mouselook to visualizer (#2551)
  • maximize build space for ubuntu (#2986)
  • Minor typo in repr for OBB (#2987)
  • function for casting the tensor data pointer to a datatype (#2934)
  • Undo lazy creation of renderer on Windows; it fixed the hang on exit when a python exception is generated before the run() happens, but broke normal windows (#2996)
  • Add Image IO (#2955)
  • Implement ray casting for TSDF voxel grids (#2941)
  • Procedural Ground Plane (#2980)
  • Mousewheel dollies based on distance from center of rotation rather than size of model (#3011)
  • Update Mac build to use latest 1.9.11 Filament to fix illegal instruction error (#3013)
  • Fix arm64 CI out-of-time (#3020)
  • Add Ground Plane support to O3DViewer (#3014)
  • Recenter rotation to point on double-click (#2905)
  • Implement depth buffer read back (#3028)
  • Remove unnecessary include causing unwanted compile dependencies (#3030)
  • Update faiss_build.cmake (#2979)
  • Added python bindings for rendering::Camera (#3033)
  • PointRCNN ops (#3021)
  • add int16 dtype and tests (#3035)
  • enable ccache for cuda with CMAKE_CUDA_COMPILER_LAUNCHER (#3043)
  • Support updating tpointcloud with the same or fewer vertices as original (#3048)
  • Use RenderTargets For Scene Caching (#3031)
  • Add init pybind for Label3D (#3045)
  • Fix linking issue of Torch/TF Ops (#2957)
  • GCE CUDA Python tests (#3063)
  • Rearrange constructor so input missing "points" attribute reaches LogError (#3062
  • Make sure bounding box is non-empty (#3061)
  • Replace CMake GLOB and GLOB_RECURSE with file listing (#3059)
  • Add pybind param names to docstring for 'TSDFVoxelGrid.integrate' (#3058)
  • Support USE_SYSTEM_LIBREALSENSE flag. (#3057)
  • Faster FixedRadiusSearch and HybridSearch (#2821)
  • IPP/NPP image operators (#2972)
  • Add function to randomly downsample from pointcloud (#3050)
  • maximize build space (#3071)
  • Add 'd' suffix to non-system libpng compiled library discovery for non-MSVC Debug builds (#3067)
  • disable incompatible clang flags for now (#3080)
  • [FileSystem] Fix GetPathComponents and unit tests on Windows. (#3070)
  • Minor spell fix (#3081)
  • Add disconnection function to AzureKinectSensor (#3044) (#3082)
  • PointRCNN ops tf (#3075)
  • Set post-processing coniguration when copying View settings (#3083)
  • Fix crash if posted Window callback executes after Window is destroyed (#3084)
  • style-guide link update (#3086)
  • Update contribution_recipes.rst (#3088)
  • changed Registration unit-test to use RegistrationPermuteDevices (#3095)
  • LU, LU_with_ipiv, Det, Triu, Tril, Triul Ops (#2879)
  • Set position in constructor (#3096)
  • memory efficient hybrid search (#3073)
  • Implemented ability to render to a bitmap window and interact with UI (#3072)
  • Better error message if you don't initialize gui::Application before creating a window (#3103)
  • Added pybind for Camera::CopyFrom() (#3104)
  • Enable builds on Fedora. (#3102)
  • [CUDA] Fix memory leaks. (#3099)
  • Use UnownedPointer so Python doesn't retain a valid shared_ptr (#3106)
  • Tensor nested initializer list with recursion (#3055)
  • Fix window resize when Scene caching is enabled. (#3107)
  • delete some ssh keys to keep the profiles small (#3110)
  • Fixed crash when referencing submenus (#3114)
  • Enable builds on Fedora (FAISS, MKL, TBB) and fix output result of CMAKE. (#3118)
  • Added the double-click code from Window.cpp in #2905 that got clobbered by the WindowSystem refactor (#3119)
  • TransformConverter supports Float32 and Float64 (#3098)
  • Fix one frame blinking in Open3DViewer when layout changed or dialog box was closed (#3121)
  • benchmark links 3rd-party library (#3124)
  • Set byproducts for external projects. (#3138)
  • Open3D C++ example projects as separete repos (#3128)
  • gce disk size set to 48GB (#3169)
  • download test datasets from intel-isl/open3d_downloads (#3105)
  • Add ability to set center of rotation on SceneWidget, which is necessary for copying from a camera. (#3147)
  • Minor(?) fix (#3175)
  • Clone from specific commit instead of release branch (#3179)
  • fix docstrings for ML modules (#3160)
  • Add multi-window C++ and Python examples (#3167)
  • Expose rendering to a depth image to Python, makes resulting depth image range from 0 (near plane) to 1 (far plane) instead of simply returning the raw values from Filament's buffer (which are reversed), fixes widgets that aren't SceneWidget or a Layout not working if they are a top-level child. (#3178)
  • Fix incorrect coordinates for mouse events on Linux (#3177)
  • Added gui::ToggleSwitch (#3182)
  • Give Material::shader default value of "defaultUnlit" (#3184)
  • Add support for textures in TriangleMesh::operator+= (#3170)
  • TensorToEigenMatrix Ops. (#3163)
  • assigning scalar to r-value tensor of arbitrary shape (#3137)
  • t::PointCloud from RGBD image (#3113)
  • openblas x86_64 docker (#3195)
  • Updated copyrights to 2021 for gui (#3189)
  • Added an example for line width (#3194)
  • disable CreateFromRGBDImage for arm64 test (#3206)
  • Don't allow empty geometry to be added to scene (#3203)
  • upgrade scipy and cmake version (#3199)
  • Added pybinds for color grading enums, also changed property open3d.visualization.rendering.Open3DScene.get_view to view (it is a property, not a function) (#3193)
  • Fix for CreateFromRGBDImage test (#3207)
  • speed up arm64 build with docker and ccache (#3205)
  • [Windows] Fix linker errors and warnings. (#3151)
  • Fix ModuleNotFoundError when build with GCC10. (#3142)
  • Tensor-based RGBD Odometry (#3085)
  • t-ICP ComputeTransform major performance improvements. (#3019)
  • UV-Map for Box, Tetrahedron, Octahedron, Icosahedron (#3097)
  • Init voxeldownsample (#3144)
  • Fix compilation on Windows when building with CUDA. (#3036)
  • Added mouse events + convert point to world coordinates example (#3192)
  • Optimising Tensor RGBD Odometry. (#3213)
  • handle the invalid tetra warning (#3201)
  • SplitString returns tokens (#3223)
  • clean up ci_utils for arm64 (#3224)
  • less verbose tar print for arm docker (#3226)
  • optimised hybrid search, clear tree from master (#3218)
  • Added 3D labels to O3DVisualizer, added example (#3233)
  • Add ability to set layout minor axis size, moved render target drawing from Open3DScene to SceneWidget which is a little more straightforward and also puts the image in the proper z-level so that SceneWidget can be used in layouts (#3212)
  • Improves progress bar in Open3DViewer app, also fixes unlit material instead of lit (#3238)
  • refactor logging (#3241)
  • Adds ImageWidget and ability to efficiently change Scene background image (#3002)
  • Update to filament v1.9.19 for mac for multi-window fixes (#3251)
  • Fix spurious warning if a triangle mesh with no normals uses "default Unlit". Happens if you pick points (#3256)
  • Fix camera matrix getting pybinds not returning something that can be converted to a Python object (#3257)
  • Hashmap refactor (#3211)
  • Added camera visualization to LineSet (#3255)
  • Added header files for TriangleMeshModel to Open3D.h.in (#3261)
  • Update stdgpu dependency to latest upstream (#3265)
  • Use more sensible default color for point clouds with no per-vertex color (#3278)
  • T-Registration pybinds and python-tests. (#3112)
  • Fix memory leak in stdgpu hashmap backend (#3281)
  • Upgrade Tensor-based RGBD odometry (#3229)
  • UV Maps for Cone, Cylinder, Sphere added. (#3275)
  • get_is_open() for CollapsableVert widget (#3286)
  • Remove use of CMAKE_BUILD_TYPE. (#3164)
  • Support TGeometry Image At Low Level and In Image Widget (#3277)
  • simplify MANIFEST.in with graft (#3288)
  • Optimised CopyCPU and CopyCUDA for case when dst is contiguous and src has single element. [Fill] (#3136)
  • Added polygon picking (#3252)
  • Add post_redraw() to O3DVisualizer so that it can be used non-blocking (#3262)
  • Fixes crash/hang on exit if rendered to an image (#3273)
  • voxel_down_sample pybind for t::pointcloud (#3293)
  • Added italic, bold, and bold italic styles (#3298)
  • Tensor IsInf, IsNan, IsFinite (#3302)
  • Add remaining fixed-size integer dtypes (#3320)
  • Port to CMAKE_CUDA_ARCHITECTURES (#3316)
  • Tensor Multi Scale ICP (#3285)
  • Fix many minor C++ documentation issues (#3326)
  • update docs-build dependencies (#3335)
  • Added tooltips (#3334)
  • Defer layout of dialog to ensure ImGUI context is valid for CalcPreferredSize (#3339)
  • Optimize TSDF-related operations (#3314)
  • Unify dispatch macros from CoreUtil.h and Dispatch.h (#3295)
  • Fix error message when no correct arguments are provided (#3292)
  • Add material property for supplying vertex colors in sRGB (#3340)
  • CUDA, TF, Pytorch version upgrade (#3049)
  • fix header include (#3344)
  • pass cuda inc dir to stdgpu to find thrust if cuda is not in a standard (#3343)
  • fix conditions for showing warning. drop pytorch version test. (#3355)
  • Remove attributes from pointcloud and trianglemesh (#3299)
  • fixed int64_t issue on mac (#3291)
  • Adjusted odd characters which were giving unicode charmap error (#3336)
  • Improve performance of stdgpu hashmap backend (#3346)
  • Add pybind for T PointCloud and fix associated bugs (#3345)
  • Shubhamagarwal12/tensor mesh io (#3221)
  • Update Linux and Windows platforms to Filament v1.9.19 (#3271)
  • Fix multiple VectorEdits in a UI not working (#3360)
  • Properly scale UInt8 colors when converting to Legacy Point Cloud (#3350)
  • Make tooltips prettier (#3348)
  • fix scalar boolean handling in pybind (#3362)
  • Fixed identically named Checkboxes and ToogleSwitches not working, removed use of stringstream in the rest (#3369)
  • Unqueue dead window events (#3370)
  • Make sure ResizeRenderer gets called on BitmapWindow (#3374)
  • Further improve performance of stdgpu hashmap backend (#3377)
  • simplify jsoncpp cmake (#3174)
  • [WebRTC] WebRTC library build scripts (#3349)
  • Move vertexmap/normalmap related functions to t::Image (#3318)
  • Revert "simplify jsoncpp cmake (#3174)" (#3386)
  • Don't add menus in Jupyter that are not meaningful (#3385)
  • Added ability to setup camera with extrinsic and intrinsic in new visualizer (#3338)
  • Increase warning level (#3357)
  • Clarify C++ standard in documentation (#3384)
  • Optimize RGBD odometry (#3323)
  • Python docs fix (#3373)
  • Add Docker minimal example to getting_started docs. Fix #3388 (#3396)
  • Introducing SLAC (#3108)
  • Voxel hashing GUI (#3363)
  • Fix screen turning black on resize in standalone viewer (#3394)
  • Tensor Registration Update (#3352)
  • SLAC Pybinds, Unit Tests, Documentation. (#3393)
  • add template parameters for input and output tensors in the cconv op (#3353)
  • RGBD Odometry Pybinds (#3359)
  • Tensor::Clip() simple implementation (#3266)
  • Fix filament byproducts and build from source. (#3375)
  • Fix CUDA builds on Windows. (#3380)
  • Replace templates with Scalar class (#3391)

Open3D 0.8.0 is here!

Open3D Version 0.8.0 (released 2019-09-06)

The Open3D team and the Open Source Vision Foundation are excited to share with the community Open3D 0.8.0!

Starting from this release, we'll be using the open3d package name instead of open3d-python on PyPI. You can now install Open3D with:

# Pip
pip install open3d

# Conda
conda install -c open3d-admin open3d

In this release, we bring support for the brand new Azure Kinect sensor. Users can enjoy the benefits of this RGB-D sensor through the simple Python and C++ APIs provided by Open3D.

We bring in a new geometry type, the Tetrahedral Mesh, which supports Delaunay triangulation from PointCloud, isosurface extraction to TriangleMesh and visualization. We also bring in two algorithm implementations, check out Ball Pivoting and DBSCAN.

As usual, we received many contributions from the community. Thank you all for your contribution and support! We also worked extensively on fixing issues brought up by the community. Please, have a look at our documentation Open3D docs to see all the details, and send us feedback at info@open3d.org. You can also join our Discord network to participate in the development discussions.

Enjoy!
The Open3D team


 

Legend:

  • [Added]: Used to indicate the addition of new features
  • [Changed]: Updates of existing functionalities
  • [Deprecated]: Functionalities / features removed in future releases
  • [Removed]: Functionalities/features removed in this release
  • [Fixed]: For any bug fixes
  • [Breaking] This functionality breaks the previous API and you need to check your code

Installation and project structure

  • [Added] Make include directories in Open3DConfig.cmake relative again #1025 (Thanks pezcode!)
  • [Changed] Replace libjpeg with libjpeg-turbo #1097
  • [Deprecated] libjpeg dependency #1097
  • [Fixed] empty bdist_wheel #1012  #970 #780 #905
  • [Fixed] master build error, fix yapf==0.28.0 #1071
  • [Fixed] Warning compile flags and fixing warning #1077

CORE features and applications

  • [Added] Expose CaptureDepthPointCloud to python interface #1014 (Thanks JackLangerman)
  • [Added] FileOBJ for TriangleMesh #1021
  • [Added] Option to remove NaN points in ReadPointCloud#1022
  • [Added] FileOFF for TriangleMesh #1035 (Thanks kongsgard)
  • [Added] DBSCAN clustering algorithm #1038
  • [Added] VoxelGrid Carving #1048
  • [Added] Add flags to the write triangle mesh methods #1051 (Thanks kongsgard)
  • [Added] function is_watertight() to TriangleMesh #1053  (Thanks kongsgard)
  • [Added] Triangulation by ear clipping #1060  (Thanks kongsgard)
  • [Added] FileGLTF for TriangleMesh #1082 (Thanks kongsgard)
  • [Added] Kinect recorder #1094
  • [Added] visualization of RGBD images #1107
  • [Added] Azure Kinect support with python interface #1118
  • [Changed] TriangleMesh Filter improvements, GetBounds improvements #1010
  • [Changed] Logging/Print #1069
  • [Removed] unary_function from hash_eigen #1023
  • [Removed] remove TSDFVoxelGrid class #1079
  • [Fixed] PointCloud EstimateNormals fix #1011
  • [Fixed] Check for mesh normals before saving STL file #1041 (Thanks kongsgard)
  • [Fixed] Set z_near_ close to 0 #1080
  • [Fixed] internal texture format in RGBDImageShader #1111
  • [Fixed] Control flows and improve performance #1112

 

Documentation, tutorials, examples and tests

  • [Added] Documentation to build docs #1036 (Thanks RohanRathi)
  • [Added] Support multiple documentation version in open3d.org/docs #1055
  • [Changed] Update file_io.rst #1020 (Thanks kongsgard)
  • [Changed] Python and C++ docs for Camera PinholeCameraIntrinsic #1039 (Thanks RohanRathi)
  • [Changed] getting_started.rst #1113
  • [Fixed] python unit test #1031
  • [Fixed] correct typo #1088 (Thanks JackLangerman)

It is here: Open3D 0.7.0!

Open3D Version 0.7.0 (released 2019-06-05)

The Open3D team and the Open Source Vision Foundation are proud to present the 0.7.0 release of the Open3D library.

This release is focused on extending the functionality of Open3D data types such as Octree, VoxelGrid, and Mesh. Among the main novelties we would like to highlight:

  • New generic Octree data type with arbitrary leaves
  • Upgraded VectorGrid data type
  • Conversion functionalities from Point Cloud to VectorGrid and Octree
  • Upgraded Mesh data type
  • Visualization support for Octree and VectorGrid
  • New sampling methods for Triangle mesh
  • New Mesh filter functionalities: Sharpen, Smooth, Taubin
  • Upgraded TSDFVolume generation

We also tackled a set of issues brought up by the community, including support of finer control over the geometry of the Visualizer. Now it is possible to add and remove geometries dynamically!

Please, have a look at our documentation (Open3D docs) to see all the details, and send us feedback at info@open3d.org. You can also join our Discord network to participate in the development discussions.

Full list of changes below.

Enjoy!

The Open3D team


Legend:

  • [Added]: Used to indicate the addition of new features
  • [Changed]: Updates of existing functionalities
  • [Deprecated]: Functionalities / features removed in future releases
  • [Removed]: Functionalities/features removed in this release
  • [Fixed]: For any bug fixes
  • [Breaking] This functionality breaks the previous API and you need to check your code

Installation and project structure

  • [Added] Googletest as a submodule #981
  • [Changed] Standardize headers #902
  • [Changed] Remove dummy semicolon after empty functions #908
  • [Changed]  appveyor build badge to master branch #925
  • [Changed] Avoid import star (from open3d import *) #982
  • [Changed] Open3D-3rdparty repository into the main repository #967
  • [Removed] Duplicated sort includes #930
  • [Fixed] Eigen::floor compiler error issue #935 #952
  • [Fixed] Build docs on make -j #960

CORE features and applications

  • [Added] Support for Octree, OctreeNode, OctreeLeafNode, and OctreeNodeInfo #903 #946 #959
  • [Added] Octree python bindings #947
  • [Added] Visualizer remove geometry #904
  • [Added] Uniform point sampling from triangle mesh #907
  • [Added] Octree I/O support for json #911
  • [Added] Octree to VoxelGrid conversion #914
  • [Added] Voxel grid to octree #983
  • [Added] Remove non-manifold edges from triangle mesh #985
  • [Added] CuboidShader for rendering VoxelGrid and Octree in OpenGL3.1+ #918
  • [Added] Octree visualization support and refactor cuboid shader #924
  • [Added] Mesh filtering functionalities (Sharpen, Smooth, Laplacian, Taubin) #926
  • [Added] Method to check if TriangleMesh is watertight #929
  • [Added] Mesh Simplification method #945
  • [Added] Loop Subdivision method #954
  • [Added] Poisson Disk Sampling of TriangleMesh #955
  • [Added] Basic Transformation Methods (Translate, Scale, Rotate) to Geometry3D #956
  • [Added] Functionality to compute convex hulls from TriangleMesh and PointCloud #965
  • [Added] Extension of mesh properties to check if a triangle mesh is orientable,  #977
  • [Added] TUM trajectory format #953 [Thanks martinruenz]
  • [Added] Center parameter to rotate method #994
  • [Changed] Refactor UniformTSDFVolume to use VoxelGrid #971
  • [Changed] Small AttributeError problem in pointcloud.py #987 [Thanks #janfelixklein]
  • [Breaking]  Update of VoxelGrid data type #933

Documentation, tutorials, and examples

  • [Fixed] Point cloud dist function docs #899
  • [Fixed] doc string of PinholeCameraTrajectory #923
  • [Fixed] misc fixes for reconstruction docs #997
  • [Changed] Improve doc for TriangleMesh, LineSet and PointCloud #980
  • [Changed] Docs update on unit test and python style #989

 

Open3D is Joining 2019 Google Season of Docs

Open3D is Joining 2019 Google Season of Docs

News: We're proud to announce that Open3D is selected as one of the participating open-source projects for 2019 Google Season of Docs (GSoD) program. We're recruiting 1-2 technical writers for this project via the program. By participating in GSoD with Open3D, you'll spend three months working closely with the Open3D team, learn about this state-of-the-art 3D data processing library, build up your open-source contribution, and receive stipends. Star Open3D on GitHub and apply today! Please read this blog post for project ideas. To apply, please (1) fill this application survey form and (2) visit GSoD official site for GSoD application instructions. Contact us if you have any questions! 

 

Links:
(1) Google Season of Docs (GSoD): https://developers.google.com/season-of-docs
(2) Timelines: https://developers.google.com/season-of-docs/docs/timeline
(3) Stipends ($6,000 USD equivalent): https://developers.google.com/season-of-docs/docs/tech-writer-stipends
(4) Open3D blog post for project ideas: http://www.open3d.org/index.php/2019/04/11/season-of-docs-for-open3d
(5) Fill out this survey from: https://docs.google.com/forms/d/e/1FAIpQLSdYZJuYTsBV8C0Ta6cXPJgWqTsnd0l19rIAR9Xr7dBPt0Ftqw/viewform?usp=sf_link . You'll also need to apply through the GSoD official site as well.

 

 

1. Project description

Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization. Open3D was developed from a clean slate with a small and carefully considered set of dependencies. It can be set up on different platforms and compiled from source with minimal effort. The code is clean, consistently styled, and maintained via a clear code review mechanism. Open3D has been used in a number of published research projects and is actively deployed in the cloud. We welcome contributions from the open-source community.

You can see the progress of Open3D through the different video releases we have published:

 

2. Organization

Open3D lives under the umbrella of the non-profit Open Source Vision Foundation (OSVF.org). OSVF provides support with the administration and coordination of the resources needed for the development of Open3D. The Open3D project has originally created within the Intelligent Systems Lab (ISL) at Intel, and the development of Open3D is still done in collaboration with ISL.

The main contributors of Open3D are:

The management of the project is done by German Ros and Vladlen Koltun.

3. Relevant links

  • Github repository [link]
  • Documentation page [link]
  • Contact e-mail [link]
  • Chat with us via Discord [link]
  • YouTube channel [link]

4. Project ideas

Below you can find our proposed ideas to improve Open3D documentation. Please, feel free to contact us to propose alternative ideas or changes to the current ideas. During the 3-month period, you're expected to complete 3 or more of the following projects.

  • Project 1 name: Upgrade and extend Open3D contributor’s guide
  • Description: Contributor’s guides are the main entry point to collaborate in an open source project. This project requires the creation of a new contributor’s guide that explains step-by-step how contributors can send their contributions to Open3D. These steps include:
    • How to find the current road map of Open3D and which features may be interesting
    • How to prepare a pull request to Open3D
    • How to interact with reviewers
    • What are the standards used by Open3D
  • Related material:
    • The current version of the contributor’s guide can be found here.
  • Project 2 name: Complete Python API documentation
  • Description: Open3D is a multi-language library, that has support for both C++ and Python. There is an initial version of the Python documentation done using Sphinx, which contains basic docstrings for each method and class of Open3D. In this project, we would like to extend the docstrings in a way that makes the functionality clearer, by rephrasing and extending the existing.
  • Related material:
    • The current version of the Python API documentation can be found here.
  • Project 3 name: Complete C++ API documentation
  • Description: Open3D is a multi-language library, that has support for both C++ and Python. There is an initial version of the Python documentation done using Sphinx http://www.open3d.org/docs/index.html#python-api-index. However, C++ documentation has not been completed for the most part of the library. C++ documentation is generated by Doxygen with docstring in the source code. In this project, we would like to fill in missing C++ docstrings, with the reference of Python docstrings.
  • Related material:
  • Project 4 name: High-level docs of core data structures
  • Description: Open3D implements several core data structures, such as PointCloud, TriangleMesh, VoxelGrid, LineSet, Octree, camera parameters and etc. Making sure that users understand the main concepts and usage of the data structures is central to the Open3D library. The current documentation we have is organized as tutorials, which is not well-structured for reference. The scope of this project includes:
    • Review current tutorial documentation at http://www.open3d.org/docs/tutorial/Basic/index.html and get familiar with the basic data structures.
    • Refactor the tutorials structured around each data structure.
    • For each data structure, provide a high-level overview, data structure API and example usage details.
  • Related material:
    • The current version of the contributor’s guide can be found here.
  • Project 5 name: Documentation of high-level users’ use cases, aka How-to do X using Open3D
  • Description: We would like to provide a section in our official documentation explaining how to perform basic functionalities using the Open3D library. Most of this information already exists within the current documentation, but it needs to be refactored and adapted to follow a common format. Examples of these functionalities would be:
    • How to load and visualize a point cloud using Open3D?
    • How to create an Octree from a Point cloud?
    • How to use your own camera poses for point cloud visualization?
    • Etc.
  • Related material:
    • Examples of these use cases can be seen here.

5. Mentors and point of contact

  • Technical writers will be mentored by two key contributors to Open3D, Yixing Lao, and Qian-Yi Zhou. Both Yixing and Qian-Yi have a deep understanding of the Open3D project as well as the process of automatic generation of documentation. They are both recognized industry leaders and their mentoring will definitely be a positive influence for technical writers.
  • If you would like to help us to improve our documentation, please contact German Ros [mail]

Open3D 0.6.0 release

Open3D Version 0.6.0 (released 2019-04-01)

The Open3D team and the Open Source Vision Foundation are excited to announce the 0.6.0 release of the Open3D library.

In this release, we focused our efforts on improving the quality of Open3D documentation and paving the way for upcoming GPU support.

Documentation is a critical aspect of any software project, but it becomes especially critical in open-source projects. This is one of the main ways we engage with the community. For this reason, we improved the internal infrastructure to automatically generate documentation in a new format, which makes the Python API more readable and easy to understand. Please take a look at Open3D docs.

The team has also been working on bringing multi-GPU support to Open3D. We will start rolling this out in upcoming releases. In the meantime, feedback and suggestions are welcomed. Please check our GPU integration branch hereThis release also includes new data types that serve as the foundation for new meshing algorithms that will be rolled out in our next release.

The full list of changes can be seen below. Please send us feedback at info@open3d.org and join our Discord network to participate in the discussions.

 

Enjoy!

The Open3D team


 

Legend:

  • [Added]: Used to indicate addition of new features
  • [Changed]: Updates of existing functionalities
  • [Deprecated]: Functionalities / features removed in future releases
  • [Removed]: Functionalities / features removed in this release
  • [Fixed]: For any bug fixes
  • [Breaking] This functionality breaks the previous API and you need to check your code

Installation and project structure

  • [Changed] simplified cmake include directory structure  #839
  • [Changed] new installation default behavior: don’t install 3rd party header except Eigen and GL #840
  • [Braking] new project directory structure #842 #850 #855

CORE features and applications

  • [Added] Travis build docs, use 16.04, and other fixes #885
  • [Added] update adjacency list after mesh operations #843
  • [Added] HalfEdgeTriangleMesh data type support #851 #868
  • [Added] STL file support #786
  • [Added] Compute vertex adjacency map #830
  • [Changed] Standardize API of SolveLinearSystemPSD #821
  • [Changed] upgraded pybind11 #837
  • [Changed] Upgrade OpenGL GLSL version #854
  • [Fixed] path in the comments of python_binding.py #878
  • [Fixed] clang format discrepancy and links #793 #795 #816
  • [Fixed] autocomplete for python modules #799
  • [Fixed] intrinsic parameters for Kinect2 #801
  • [Fixed] initializers for FastGlobalRegistration class #807 #808
  • [Fixed] Travis fails when unit tests fail in a docker container #810
  • [Fixed] ColorMap divergency and other issues #819 #860
  • [Fixed] add minus sign in SolveJacobianSystemAndObtainExtrinsicMatrixArray #822
  • [Fixed] STL mesh write vertex index #829

Documentation, tutorials, and examples

  • [Added] pybind docs parser and Google-style docstring generator #864
  • [Added] Expandible docs sidebar #832
  • [Changed] extended Python documentation #859 #861 #862 #869 #877 #881
  • [Changed] C++ project documentation for Windows #809
  • [Fixed] issues in the documentation building process #845
  • [Fixed] namespace typo in examples #856
  • [Fixed] small issues in RegistrationRansac example #800 #802

 

Testing and benchmarking

  • [Removed] visualization unit tests #857

 

Introducing Open3D 0.5.0

Open3D Version 0.5.0 (released 2019-01-17)

The Open3D team and the Open Source Vision Foundation (http://www.osvf.org) are excited to announce the 0.5.0 release of the Open3D library.

In this release we show the power of Open3D as a core tool to create machine learning solutions for 3D data. We introduce a re-implementation of the PointNet++ architecture to perform point cloud semantic segmentation using Open3D and TensorFlow. Our Open3D-PointNet++ is able to produce highly accurate results in the Semantic3D benchmark, surpassing the results of the original PointNet++ implementation. Even more exciting is the fact that our re-designed Open3D-PointNet++ is able to perform real-time inference (+10 FPS) on the KITTI dataset. We show how to perform training and inference of Open3D-PointNet++ in both Semantic3D and KITTI. Check out this blog post for more information!

We have also added a new VoxelGrid representation and tooling to convert from point clouds to a VoxelGrid structure. This functionality is extremely useful to produce representations that are easier to digest by neural networks.

We have also done significant improvements to our internal infrastructure, including a simplified CI testing mechanism via docker images, enhanced testing coverage, and easier installation of the library.

Full list of changes below. Please send us feedback at info@open3d.org and join our Discord network [link] to participate in the discussions.

Enjoy!

The Open3D team


 

Legend:

  • [Added]: Used to indicate addition of new features
  • [Changed]: Updates of existing functionalities
  • [Deprecated]: Functionalities / features removed in future releases
  • [Removed]: Functionalities / features removed in this release
  • [Fixed]: For any bug fixes
  • [Breaking] This functionality breaks the previous API and you need to check your code

Installation and project structure

  • [Added] docker images for Open3D in dockerhub
  • [Added] option to disable jupyter build
  • [Added] new way of detecting conda active environment
  • [Added] option to link to static Windows runtime
  • [Changed] 3rdparty folder moved to Open3D-3rdparty repository
  • [Changed] bug_report.md to improve communication with users when issues are reported
  • [Fixed] Conda and Pip packaging issues to build platform-specific targets
  • [Fixed] conda dependency conflicts resulting in forced downgrade
  • [Fixed] python 2.7 import JVisulizer
  • [Fixed] Disabled conda executable check (conda command could be a bash function instead of an executable, CMake may complain)
  • [Fixed] Windows compilation warning with py::ssize_t
  • [Removed] mac flag -Wno-expansion-to-defined in CI

CORE features and applications

  • [Added] New Open3D Point cloud semantic segmentation architecture based on PointNet++
  • [Added] New training code for Point cloud semantic segmentation
  • [Added] New real-time inference code for Point cloud semantic segmentation
  • [Added] New compatibility with TensorFlow operators
  • [Added] New function for building Jacobian matrices that follows RGBDOdometry structure
  • [Added] Non-rigid optimization for more than 6 variables (6D camera pose + anchor points)
  • [Added] A new general purpose image processing function: CreateDepthBoundaryMask
  • [Added] "shift + +/-" key event that can change width of LineSet for the visualization
  • [Added] line_width in RenderOption and corresponding Python binding (Applies to C++/Python API)
  • [Added] I/O functions for LineSet
  • [Added] "lineset" option into ViewGeometry application
  • [Added] New box primitive
  • [Added] VoxelGrid structure
  • [Added] I/O functions for VoxelGrid
  • [Added] Utility function to transform point clouds to voxels
  • [Added] new shader to render voxel clouds
  • [Added] warning output for "degenerated" TriangleMeshes
  • [Added] Promote compiled extension for pycharm autocomplete
  • [Changed] Image class to use namespace directive in order to reduce code line length
    [Changed] Image class to remove global variables
    [Changed] Image class to shorten local variables names
    [Changed] Image class to simplify comparisons using unit_test::ExpectEQ(...)
  • [Changed] KDTreeFlann class to use namespace directive in order to reduce code line length
  • [Changed] KDTreeFlann class to simplify comparisons using unit_test::ExpectEQ(...)
  • [Changed] TriangleMesh class to use namespace directive in order to reduce code line length
  • [Changed] TriangleMesh class to simplify comparisons using unit_test::ExpectEQ(...)
  • [Changed] Relative paths in CMake package config
  • [Changed] Factorization of internal functions in ColormapOptimization module as public functions
  • [Changed] RGBDImage class to use namespace directive in order to reduce code line length
  • [Changed] RGBDImage class to simplified comparisons using unit_test::ExpectEQ(...)
  • [Changed] RGBDImage class to fixed Rand float/double to return unscaled values between 0.0 and 1.0
  • [Changed] PointCloud class to use namespace directive in order to reduce code line length
  • [Changed] PointCloud class to simplify comparisons using unit_test::ExpectEQ(...)
  • [Changed] LineSet class to use namespace directive in order to reduce code line length
  • [Changed] LineSet class to simplify comparisons using unit_test::ExpectEQ(...)
  • [Changed] Vector3dvector and other vector Eigen bindings to improve performance (speedup of 40-200x)
  • [Fixed] Bug due to PinholeCameraIntrinsic constructor not initializing member data
  • [Fixed]  Bug in PinholeCameraTrajectory
  • [Fixed] Bug in ConvertToJsonValue
  • [Fixed] Bug in ConvertFromJsonValue
  • [Fixed] Bug in TransformationEstimationPointToPlane::ComputeRMSE
  • [Fixed] typos in FilePLY.cpp: from ply_poincloud_reader to ply_pointcloud_reader
  • [Fixed] parameter name of create_window
  • [Removed] Unneeded std::move calls

Documentation and tutorials

  • [Added] New tutorial on how to perform real-time PointCloud semantic segmentation using Open3D
  • [Added] Documentation on supported point cloud formats

 

Testing and benchmarking

  • [Added] Test case for IJsonConvertible
  • [Added] Test case for Core/Utility/Eigen
  • [Added] Test case for Core/Utility/FileSystem
  • [Added] Test case for PinholeCameraTrajectory
  • [Added] Test case for PinholeCameraIntrinsic
  • [Added] Test case for RGBDOdometryJacobianFromHybridTerm
  • [Added] Test case for RGBDOdometryJacobianFromColorTerm
  • [Added] New reference data for RGBDImage based on fixes to Rand float/double
  • [Added] New utilities for generating input data for the unit tests
  • [Changed] UnitTest/Utility moved to its own folder
  • [Changed] unit_test:ExpectEQ to removed unused code

On point clouds Semantic Segmentation

In this post, we will walk you through how Open3D can be used to perform real-time semantic segmentation of point clouds for Autonomous Driving purposes. We demonstrate our results in the KITTI benchmark and the Semantic3D benchmark. Please, use the following link to access our demo project. See Figure 1 for an example of semantic segmentation of PointClouds in the Semantic3D dataset.

OutputFigure 1. Example of PointCloud semantic segmentation. Left, input dense point cloud with RGB information. Right, semantic segmentation prediction map using Open3D-PointNet++.

The main purpose of this project is to showcase how to build a state-of-the-art machine learning pipeline for 3D inference by leveraging the building blogs available in Open3D. For this purpose we have to deal with several stages, such as: 1) pre-processing, 2) custom TensorFlow op integration, 3) post-processing and 4) visualization. Furthermore, we want to demonstrate how critical is the correct design of these modules in order to achieve maximum accuracy and run-time performance, and how Open3D can help to simplify this process.

Segmenting PointClouds

We based our development on the well-known PointNet++ architecture, following Mathieu Orhan and Guillaume Dekeyser's repo and the original PointNet++ implementations as a reference. We thank authors for sharing their methods. Our implementation was re-built using Open3D and we deviated from the reference design when needed in order to improve performance, as described in the following section.

PointNetFigure 2. Diagram depicting PointNet++ architecture.

For our experiments we made use of the state-of-the-art Semantic3D and KITTI datasets. In Semantic3D, there is ground truth labels for 8 semantic classes: 1) man-made terrain, 2) natural terrain, 3) high vegetation, 4) low vegetation, 5) buildings, 6) remaining hardscape, 7) scanning artifacts, 8) cars and trucks. The goal for the point cloud classification task is to output per-point class labels given the point cloud.

Semantic3DFigure 3. Semantic 3D snapshot

KITTIFigure 4. KITTI snapshot

Since Semantic3D dataset contains a huge number of points per point cloud (up to 5e8, see dataset stats), we first run voxel-downsampling with Open3D to reduce the dataset size. During both training and inference, PointNet++ is fed with fix-sized cropped point clouds within boxes, we set the box size to be 60m x 20m x Inf, with the Z-axis allowing all values. During inference with KITTI, we set the region of interest to be 30m in front and behind the car, 10m to the left and right of the car center to fit the box size. This allows the PointNet++ model to only predict one sample per frame.

Our semantic segmentation model is trained on the Semantic3D dataset, and it is used to perform inference on both Semantic3D and KITTI datasets. In this document, we focus on the techniques which enable real-time inference on KITTI.

Accelerating PointNet++ with Open3D-enabled TensorFlow op

In PointNet++’s set abstraction layer, the original points are subsampled, and features of the subsampled points must be propagated to all of the original points by interpolation (see Section 3.4 of PointNet++). This is achieved by 3-nearest neighbors search, of which the authors provided a simple C++ implementation via custom TensorFlow op called ThreeNN. However, this turns out to be the bottleneck of the PointNet++ prediction model.

The following benchmark is obtained by the benchmark script running inference on a batch of 64 samples on colored Semantic3D dataset. As we can see, the ThreeNN op accounts for 87% of the graph execution time.

// Batch time
Batch size: 64, batch_time: 1.8208365440368652

// Per-op time
node name |           total execution time |  accelerator execution time |        cpu execution time |
ThreeNN            1.73sec (100.00%, 87.61%),        0us (100.00%, 0.00%),   1.73sec (100.00%, 95.87%)
ThreeInterpolate     60.68ms (12.39%, 3.07%),        0us (100.00%, 0.00%),      60.68ms (4.13%, 3.36%)
GroupPoint            27.31ms (9.32%, 1.38%),   27.03ms (100.00%, 15.85%),        275us (0.77%, 0.02%)
Conv2D                26.91ms (7.94%, 1.36%),    23.99ms (84.15%, 14.07%),       2.91ms (0.76%, 0.16%)

Open3D uses FLANN to build KDTrees for fast retrieval of nearest neighbors, which can be used to accelerate the ThreeNN op. This custom TensorFlow op implementation must be linked with Open3D and the TensorFlow library. To conveniently link the various dependencies, we provide a CMake file that automatically downloads, builds and links Open3D. When Open3D is properly installed (in this case automatically), one can simply use Open3D's CMake finder to include headers and link Open3D like the following.

target_include_directories(tf_interpolate PUBLIC ${Open3D_INCLUDE_DIRS})
target_link_libraries(tf_interpolate tensorflow_framework ${Open3D_LIBRARIES})

For more details on how to link a C++ project to Open3D, please see this documentation.

Next, we refactor the ThreeNN to make use of Open3D. In summary, first, a KDTree with the reference point is created by

open3d::KDTreeFlann reference_kd_tree(reference_pcd);

Then for each target point, we search the 3 nearest neighbors in the KDTree

// for each j:
reference_kd_tree.SearchKNN(target_pcd.points_[j], 3, three_indices, three_dists);

After refactoring ThreeNN with Open3D, we see a ~2X speed up in both the ThreeNN and the full model run time with batch size 64.

// Batch time
Batch size: 64, batch_time: 0.7777869701385498

// Per-op time
node name |             total execution time |  accelerator execution time |         cpu execution time |
ThreeNN            694.14ms (100.00%, 73.72%),         0us (100.00%, 0.00%),   694.14ms (100.00%, 90.20%)
ThreeInterpolate      62.94ms (26.28%, 6.68%),         0us (100.00%, 0.00%),       62.94ms (9.80%, 8.18%)
GroupPoint            27.18ms (19.60%, 2.89%),    26.90ms (100.00%, 15.63%),         287us (1.62%, 0.04%)
Conv2D                26.39ms (16.71%, 2.80%),     23.83ms (84.37%, 13.85%),        2.56ms (1.58%, 0.33%)

Post processing: accelerating label interpolation

Since we subsampled the original dataset before feeding points to PointNet++, the network outputs only correspond to a sparse subset of the original point cloud.

SparseFigure 5. Inference on sparse pointcloud (KITTI).

DenseFigure 6. Inference results after interpolation.

The sparse labels need to be interpolated to generate labels for all input points. This interpolation can be achieved with nearest neighbor search using open3d.KDTreeFlann and majority voting, similar to what we did above in the ThreeNN op.

def interpolate_dense_labels(sparse_points, sparse_labels, dense_points, k=3):
    sparse_pcd = open3d.PointCloud()
    sparse_pcd.points = open3d.Vector3dVector(sparse_points)
    sparse_pcd_tree = open3d.KDTreeFlann(sparse_pcd)

    dense_labels = []
    for dense_point in dense_points:
        _, sparse_indexes, _ = sparse_pcd_tree.search_knn_vector_3d(
            dense_point, k
        )
        knn_sparse_labels = sparse_labels[sparse_indexes]
        dense_label = np.bincount(knn_sparse_labels).argmax()
        dense_labels.append(dense_label)
    return dense_labels

However, doing so in Python could be a major performance hit. We run the full kitti_predict.py inference on KITTI dataset for a benchmark. The interpolation step takes about 90% of the total run time and slows down the full pipeline to about 1 FPS.

$ python kitti_predict.py --ckpt path/to/checkpoint.ckpt
...
[ 1.05 FPS] load_data: 0.0028, predict: 0.0375, interpolate: 0.9076, visualize: 0.0031, total: 0.9545
[ 1.06 FPS] load_data: 0.0028, predict: 0.0355, interpolate: 0.8952, visualize: 0.0025, total: 0.9396
[ 1.04 FPS] load_data: 0.0028, predict: 0.0348, interpolate: 0.9214, visualize: 0.0024, total: 0.9653
...

To address the performance issue, another custom TensorFlow C++ op InterploateLabel is added. The op takes sparse_pointssparse_labelsdense_points and outputs dense_labels. OpenMP is used to parallelize KNN tree search. dense_colors output is also added to the op to directly output label-colored dense points. Please refer to the source code for details.

Another benefit of using such approach is that now the full pipeline of prediction and interpolation is implemented with one TensorFlow op graph. That is, TensorFlow session takes in the original dense points, and directly returns dense labels and label-colored dense points. This approach is more modular and efficient than doing the interpolation outside of the TensorFlow graph. After optimization, the end-to-end pipeline achieved an average of 10+ FPS on KITTI dataset, which faster than KITTI's capture rate at 10 FPS.

$ python kitti_predict.py --ckpt path/to/checkpoint.ckpt
...                      
[10.73 FPS] load_data: 0.0046, predict_interpolate: 0.0840, visualize: 0.0046, total: 0.0932                         
[12.89 FPS] load_data: 0.0047, predict_interpolate: 0.0693, visualize: 0.0035, total: 0.0776                         
[11.44 FPS] load_data: 0.0047, predict_interpolate: 0.0791, visualize: 0.0035, total: 0.0874                         
...

Video example on KITTI dataset

 

How-to train Open3D-PointNet++ on Semantic3D dataset

1. Data download

Download the  Semantic3D dataset and extract

cd dataset/semantic_raw
bash download_semantic3d.sh

The outcome of these commands should look like this:

Open3D-PointNet2-Semantic3D/dataset/semantic_raw
├── bildstein_station1_xyz_intensity_rgb.labels
├── bildstein_station1_xyz_intensity_rgb.txt
├── bildstein_station3_xyz_intensity_rgb.labels
├── bildstein_station3_xyz_intensity_rgb.txt
├── ...

2. Convert txt to pcd file

Run

python preprocess.py

Open3D is able to read .pcd files much more efficiently.

Open3D-PointNet2-Semantic3D/dataset/semantic_raw
├── bildstein_station1_xyz_intensity_rgb.labels
├── bildstein_station1_xyz_intensity_rgb.pcd (new)
├── bildstein_station1_xyz_intensity_rgb.txt
├── bildstein_station3_xyz_intensity_rgb.labels
├── bildstein_station3_xyz_intensity_rgb.pcd (new)
├── bildstein_station3_xyz_intensity_rgb.txt
├── ...

3. Downsample

Run

python downsample.py

The downsampled dataset will be written to dataset/semantic_downsampled. Points with label 0 (unlabled) are excluded during downsampling.

Open3D-PointNet2-Semantic3D/dataset/semantic_downsampled
├── bildstein_station1_xyz_intensity_rgb.labels
├── bildstein_station1_xyz_intensity_rgb.pcd
├── bildstein_station3_xyz_intensity_rgb.labels
├── bildstein_station3_xyz_intensity_rgb.pcd
├── ...

4. Compile TF Ops

We need to build TF kernels in tf_ops. First, activate the virtualenv and make sure TF can be found with current python. The following line shall run without error.

python -c "import tensorflow as tf"

Then build TF ops. You'll need CUDA and CMake 3.8+.

cd tf_ops
mkdir build
cd build
cmake ..
make

After compilation the following .so files shall be in the build directory.

Open3D-PointNet2-Semantic3D/tf_ops/build
├── libtf_grouping.so
├── libtf_interpolate.so
├── libtf_sampling.so
├── ...

Verify that that the TF kernels are working by running

cd .. # Now we're at Open3D-PointNet2-Semantic3D/tf_ops
python test_tf_ops.py

5. Train

Run

python train.py

By default, the training set will be used for training and the validation set will be used for validation. To train with both training and validation set, use the --train_set=train_full flag. Checkpoints will be output to log/semantic.

6. Predict

Pick a checkpoint and run the predict.py script. The prediction dataset is configured by --set. Since PointNet2 only takes a few thousand points per forward pass, we need to sample from the prediction dataset multiple times to get a good coverage of the points. Each sample contains the few thousand points required by PointNet2. To specify the number of such samples per scene, use the --num_samples flag.

python predict.py --ckpt log/semantic/best_model_epoch_040.ckpt \
                  --set=validation \
                  --num_samples=500

The prediction results will be written to result/sparse.

Open3D-PointNet2-Semantic3D/result/sparse
├── sg27_station4_intensity_rgb.labels
├── sg27_station4_intensity_rgb.pcd
├── sg27_station5_intensity_rgb.labels
├── sg27_station5_intensity_rgb.pcd
├── ...

7. Interpolate

The last step is to interpolate the sparse prediction to the full point cloud. We use Open3D's K-NN hybrid search with specified radius.

python interpolate.py

The prediction results will be written to result/dense.

Open3D-PointNet2-Semantic3D/result/dense
├── sg27_station4_intensity_rgb.labels
├── sg27_station5_intensity_rgb.labels
├── ...

8. Submission

Finally, if you're submitting to Semantic3D benchmark, we've included a handy tools to rename the submission file names.

python renamer.py

9. Summary of directories

  • dataset/semantic_raw: Raw Semantic3D data, .txt and .labels files. Also contains the .pcd file generated by preprocess.py.
  • dataset/semantic_downsampled: Generated from downsample.py. Downsampled data, contains .pcd and .labels files.
  • result/sparse: Generated from predict.py. Sparse predictions, contains .pcd and .labels files.
  • result/dense: Dense predictions, contains .labels files.
  • result/dense_label_colorized: Dense predictions with points colored by label type.

 

How-to train and validate Open3D-PointNet++ on the KITTI dataset

This section provides additional information on how to train a model to work with the KITTI dataset

1. Data download and preparation

  • First, make sure you followed steps 1 to 4 in "How-to train Open3D-PointNet++ on Semantic3D dataset".
  • Download the KITTI dataset using existing KITTI download script
cd Open3D-PointNet2-Semantic3D/dataset/kitti_raw 
./raw_data_downloader.sh

2. Install auxiliary dependencies

pip install pykitti

3. Training with adapted Semantic3D data

python train.py --train_set train --config_file semantic_no_color.json

3. Perform real-time Inference

python kitti_predict.py --ckpt logs/semantic_backup_dec_20_for_kitti/semantic/best_model_epoch_060.ckpt --kitti_root=Open3D-PointNet2-Semantic3D/dataset/kitti_raw