Happy new year to all! The Open3D team and many amazing community members bring to you the latest and best Open3D with many new features and bug fixes. Here are the highlights of this release:

• Support for Python 3.12 and Numpy 2.

• Experimental cross-platform GPU support through sycl. Use your integrated GPU to accelerate Open3D! Try out the preview Python 3.10 sycl wheel on Ubuntu 22.04+. For more details, see the documentation.

• The preview wheel also features GPU accelerated raycasting with Embree (Intel GPUs) for fast geometry computation. docs. Contributed by @lumurillo

• 3D geometry metrics: Chamfer distance, Hausdorff distance and F-score. TriangleMesh docs and PointCloud docs.
  

• New FlyingEdges algorithm for extracting an isosurface (triangle mesh) from a dense 3D volume. Contributed by @intelshashi in #6648. docs.

• Add albedo texture to a mesh from calibrated images. docs and example.

  

• macOS ARM64+x86_64 universal Python wheel and C++ binary. Contributed by @sitic in #6695 .

• Sampling from mesh now includes color. Contributed by @nikste in #6842.

• CUDA 12 support, and support for building with CUDA dynamic libraries. Contributed by @davconde in #6815 .

• Support for PyTorch 2.2 and TensorFlow 2.16

Warning: Due to incompatibilities in the cxx11_abi on Linux between PyTorch and TensorFlow, official Python wheels on Linux only support PyTorch, not TensorFlow. [See https://github.com/isl-org/Open3D/pull/6288] for details. If you'd like to use Open3D with TensorFlow on Linux, you can build Open3D wheel from source in docker with support for Tensorflow (but not PyTorch) as:

cd docker
# Build open3d and open3d-cpu wheels for Python 3.10 with Tensorflow support
export BUILD_PYTORCH_OPS=OFF BUILD_TENSORFLOW_OPS=ON
./docker_build.sh cuda_wheel_py310

• Different color pooling modes for creating VoxelGrid from PointCloud. Contributed by @rxba in #6937
  

• File IO now works with Python pathlib.Path. Contributed by @sitic in #6619

• Improved Python documentation, type checking and auto-complete support with mypy and VS Code. Many PRs (#6869, #6896, #6940, #7095, #7096, #7097, #7103, #7105) contributed by tireless @timohl !

• Improved camera visualization
  

• Add mouse callback to VisualizerWithKeyCallback Contributed by @organic-nailer in @6760.

Known Issues

• macOS viewer does not work with file associations, does not open a blank window on opening without a file. This is a side effect of the GLFW update (PR #6587) . Please continue using v0.18 macOS viewer.
• ARM64 Linux builds are not available due to a runtime dynamic linker error: "cannot allocate memory in static TLS block". e.g. see here.
• [Breaking change] Changed lambda to lambda_penalty in orient_normals_consistent_tangent_plane. (https://github.com/isl-org/Open3D/pull/7096)
• [Breaking change] Fix torch furthest_point_sampling argument typo (https://github.com/isl-org/Open3D/pull/7025)
• [Breaking change] If an image read fails, any old image data held in the Image object is cleared to prevent accidental use.
• BUILD_HEADLESS_RENDERING option (OSMesa) gives a build error in Ubuntu 24.04.

Acknowledgment

We would like to thank all of our community contributors for their true labor of love for this release!
Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Happy new year to all! The Open3D team and many amazing community members bring to you the latest and best Open3D with many new features and bug fixes. Here are the highlights of this release:

• Support for Python 3.11. (Python 3.7 is no longer officially supported.)
• Support for PyTorch 2.0 and TensorFlow 2.14!

Warning: Due to incompatibilities in the cxx11_abi on Linux between PyTorch and TensorFlow, official Python wheels on Linux only support PyTorch, not TensorFlow. [See #6288] for details. If you'd like to use Open3D with TensorFlow on Linux, you can build Open3D wheel from source in docker with support for Tensorflow (but not PyTorch) as:

cd docker
# Build open3d and open3d-cpu wheels for Python 3.10 with Tensorflow support
export BUILD_PYTORCH_OPS=OFF BUILD_TENSORFLOW_OPS=ON
./docker_build.sh cuda_wheel_py310

• New Doppler ICP registration for FMCW Lidars (contributed by @heethesh)
  
  Comparison of tunnel reconstructions using point-to-plane ICP (left) and Doppler ICP (right) with measurements collected by an FMCW LiDAR (Image from original Doppler-ICP repo)

• Adding in memory support for xyz files (#5866) (contributed by @samypr100)

• Modern and more user friendly furo theme for Open3D documentation (contributed by @saurabheights)

• The master branch is now renamed to main

Here is the full set of updates:

Visualization and GUI

• [New] Display RGBD frames as a point cloud video (#6342) (contributed by @ssheorey)
• [New] Allow setting view status from json string (#6241) (contributed by @yxlao)
• [Update] Add env var to disable automatic WebVisualizer use in Jupyter environment (#6541) (contributed by @gutnar)
• [Update] Expose Near Clip and Far Clip parameters to the Python Bindings (#6520) (contributed by @zalo)
• [Update] Update jupyter web_visualizer for ipywidgets 8 (#6239) (contributed by @ssheorey)
• [Fix] Fix the triggering in PickPointsInteractor when there are linesets present (#6499) (contributed by @EwingKang)
• [Fix] fixed ProgressBar scrolling behaviour (#6175) (contributed by @cansik)
• [Fix] removed was-open check for combobox (#6178) (contributed by @cansik)
• [Fix] Fix the pybind reference bug of get_view_control() (#6116) (contributed by @eamonn-zh)
• [Fix] Add new Mitsuba Python example + update CPU rendering docs (#5991) (contributed by @errissa)
• [Fix] Enable picking points from TriangleMeshModels in O3D Visualizer (#5978) (contributed by @errissa)

Geometry

• [New] Add GetSurfaceArea() for t::TriangleMesh (#6248) (contributed by @shanagr)
• [New] Implement open3d::t::geometry::TriangleMesh::SelectByIndex (#6415) (contributed by @nsaiapova)
• [New] add add_voxel and remove_voxel in pybind for VoxelGrid::AddVoxel, VoxelGrid::RemoveVoxel (#6023) (contributed by @ohkwon718)
• [New] Voxel Downsample for Tensor interface (#6249) (contributed by @theNded)
• [Update] List intersections (#6442) (contributed by @dmitrishastin)
• [Update] Added optional invert parameter in Pointcloud cropping (similar to tensor pointcloud) (#6377) (contributed by @wakkoyankee)
• [Update] :sparkles: Support multi-threading in the RayCastingScene function to commit scene (#6051) (contributed by @sebastienmascha)
• [Update] Save triangle material IDs and uvs during TriangleMesh::SelectByIndex. (#6099) (contributed by @hanseuljun
• [Update] Update nanoflann to version 1.5.0 (#6216) (contributed by @roehling)
• [Fix] Bugfixes and docs improvement of AABB tensor class (#6486) (contributed by @saurabheights)
• [Fix] Fix seg fault in t::geometry::TriangleMesh::SelectByIndex for negative index (#6489) (contributed by @nsaiapova)
• [Fix] Correct bound check for axis aligned bounding box and use same formatting for printing cuda or cpu tensor (#6444) (contributed by @saurabheights)
• [Fix] Fix for Issue #5924: (#5938) (contributed by @dbs4261)
• [Fix] Throw error on non-contiguous Python buffer provided to Image ctor. (#6406) (contributed by @ssheorey)
• [Fix] Normal orientation issues in complex geometries (#6198) (contributed by @eugeniovaretti)
• [Fix] Fix incomplete VoxelGrid creation from triangles intersecting multiple voxels (#6325) (contributed by @sharon-br)
• [Fix] SimplifyQuadricDecimation fixes (#6163) (contributed by @bjudeworley)
• [Fix] fix segment plane non deterministic sampler issue and add unit test (#6308) (contributed by @yuecideng)
• [Fix] Fix and improve ISS keypoint (#6079) (contributed by @DNKpp)
• [Fix] Updated typo in pointcloud.cpp for tensor (#6228) (contributed by @bkhanal-11)
• [Fix] fix oriented boundingbox computation (#6069) (contributed by @ruaqy)

Pipelines

• [New] Add Doppler ICP in tensor registration pipeline (#5237) (contributed by @heethesh)
• [New] Feaure matching API (#5964) (contributed by @theNded)
• [Fix] Fix unused variable error when OpenMP is disabled (#6529) (contributed by @sitic)
• [Fix] Fix IntegrateRGBD example compilation (#6453) (contributed by @sitic)
• [Fix] Fixes #6291 and #6236 (#6305) (contributed by @theNded)
• [Fix] improve 6120 (OpenCV pose estimation bug fix) (#6306) (contributed by @theNded)
• [Fix] Fix tensor reconstruction example (#6207) (contributed by @saurabheights)
• [Fix] Fix typo in reconstruction cpp app (#6214) (contributed by @yuecideng)

Open3D-ML

• [New] Add TUM Facade dataset (isl-org/open3d-ml#613) (contributed by @ThomasFroech)

• [New] Add Pandaset autonomous driving dataset (isl-org/open3d-ml#611) (contributed by @SulekBartek)

• [New] Support Python 3.11 (isl-org/open3d-ml#619) ) (contributed by @ssheorey)
• [Fix] Changed usage of TensorFlow Status API (#6091) (contributed by @NobuoTsukamoto)
• [Fix] Update boundingbox.py - switch np.int_ to np.int32 (isl-org/open3d-ml#624) (contributed by @YishGene)
• [Fix] Replace colon with dash in timestamp variables (https://github.com/isl-org/Open3D-ML/pull/620) (contributed by @Cassius0924)
• [Fix] Corrected pairing in SSD head of pointpillars (https://github.com/isl-org/Open3D-ML/pull/602) (contributed by @zhongyidu)
• [Fix] Avoid renaming unwanted part of the path in dataset loading scripts (isl-org/Open3D-ML#597) (contributed by @mariusud and @ManonCortial)
• [Fix] Fix pytorch 1.13 error due to 2D weights (isl-org/open3d-ml#601) (contributed by @ssheorey)

I/O

• [New] Adding in memory support for xyz files (#5866) (contributed by @samypr100)
• [New] Write directly to GLB from TGeometries (#6404) (contributed by @errissa)
• [Update] Bump certifi lib from 2022.12.7 to 2023.7.22 (#6282) (contributed by dependabot[bot])
• [Update] support msgpack 6.0 (#6269) (contributed by @cielartisan)
• [Fix] Add IO for NPZ files for Tensor-based TriangleMeshes (#6019) (contributed by @errissa)

Core

• [Update] Use standard filesystem instead of experimental for Clang on macOS (#6320) (contributed by @errissa)
• [Update] Output OSError details to make import error more useful (#5997) (contributed by @errissa)
• [Fix] Check for support of CUDA Memory Pools at runtime (#4679) (#6440) (contributed by @ao2)
• [Fix] Fix python UnicodeDecodeError on windows (#6277) (contributed by @MisakaQwQ)
• [Fix] Fix two crash bugs on Apple Silicon (#6008) (contributed by @errissa)

Build System

• [Update] master -> main rename for branches (#6530) (contributed by @ssheorey)
• [Update] Remove extra documentation build for python examples (#6487) (contributed by @saurabheights)
• [Fix] Google Bucket hold and docs building fix (#6460) (contributed by @ssheorey)
• [Update] Add unicode to check style (#6443) (contributed by @dmitrishastin)
• [New] Add support for python3.11 (#6288) (contributed by @OlivierLDff Olivier Le Doeuff and @ssheorey)
• [New] Debian Open3D package in CI (#6044) (contributed by @ssheorey)
• [Fix] Fix path to executable in viewer deb (#5996) (contributed by @simonschmeisser)
• [Fix] Build fixes MSVC and gcc (#6039) (contributed by @ssheorey)
• [Fix] Windows CI out of disk space workaround (#6281) (contributed by @ssheorey)
• [Fix] Increase Ubuntu wheel CI build space. (#6255) (contributed by @ssheorey)
• [Update] relax nbformat (#6135) (contributed by @Borda)
• [Fix] Fix build with gcc-13 (#6215) (contributed by @roehling)
• [Update] Build system improvements backported from Debian (#6180) (contributed by @roehling)
• [Update] Use GLIBCXX_USE_CXX11_ABI by default (#6206) (contributed by @QuentinTorg)
• [Update] Test Ubuntu wheel on 20.04 (#6205) (contributed by @ssheorey)
• [Fix] Enable multi-core link time optimization (#6181) (contributed by @roehling)

Documentation

• [Update] Add missing keyword arg names to visualization bindings (#6407) (contributed by @djkeyes)
• [Fix] Fix regression - Keep output of cell blocks in jupyter in scrollable area. (#6504) (contributed by @saurabheights)
• [Update] Upgrade documentation techstack (#6470) (contributed by @saurabheights)
• [Update] Add Open3D architecture diagram (#6488) (contributed by @ssheorey)
• [Fix] fixed arguments in example notebooks (#6410) (contributed by @tomole444)
• [Update] Update visualization.ipynb (#6263) (contributed by @Sendeky)
• [Fix] Improve Documentation - Fix code links and improve tutorial layout (#6321) (contributed by @saurabheights)
• [Fix] Fix rendering of Python versions table (#6162) (contributed by @johnthagen)
• [Fix] Improve documentation (#6307) (contributed by @saurabheights)
• [Fix] Bring tutorial code snippets in-sync with latest source #6077 (#6128) (contributed by @dancergraham)

Known Issues

• Build errors in 3rd party libraries with gcc-13 on Ubuntu 23.10 (#6544). Please use prebuilt binaries or gcc-12 for building.

Acknowledgment

We would like to thank all of our community contributors for their true labor of love for this release!
Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Open3D 0.17 Release Notes

We are happy to bring you the best Open3D yet! This is a "tick" release focused on resolving existing issues and eliminating bugs. We resolved over 150 issues for Open3D and Open3D-ML since the last release.

Here are the main highlights of this release:

• Open3D now has support for differentiable rendering with Mitsuba.

Here we use Mitsuba to recover the albedo map from an input image for an object in Open3D. Example.

• New lighter wheel (~105MB) for x86_64 Linux, which is suitable for systems without CUDA. Try it out with:
  pip install open3d-cpu.
• The Open3D viewer app is now available for Windows. This version is unsigned, so expect Windows warnings.
• Apple Silicon wheels now come with 3DML support.
• Preview of distributed training with PyTorch for PointPillars on the Waymo dataset.
• PointCloud partitioning with PCA.

Visualization and GUI

[New] Integration with Mitsuba.

• Open3D geometries and materials seamlessly convert to Mitsuba's Principled BSDF:

  
  dataset = o3d.data.MonkeyModel()
  mesh = o3d.t.io.read_triangle_mesh(dataset.path)
  mesh_center = mesh.get_axis_aligned_bounding_box().get_center()
  mesh.material.set_default_properties()
  mesh.material.material_name = 'defaultLit'
  mesh.material.scalar_properties['metallic'] = 1.0
  mesh.material.texture_maps['albedo'] = o3d.t.io.read_image(dataset.path_map['albedo'])
  mesh.material.texture_maps['roughness'] = o3d.t.io.read_image(dataset.path_map['roughness'])
  mesh.material.texture_maps['metallic'] = o3d.t.io.read_image(dataset.path_map['metallic'])

mi_mesh = mesh.to_mitsuba('monkey')
img = render_mesh(mi_mesh, mesh_center.numpy())
mi.Bitmap(img).write('test.exr')

![test2](https://user-images.githubusercontent.com/3722407/224226235-b60acfa8-dc89-44ae-b373-060bafed54ca.png)

* Also supports applying other Mitsuba BSDFs to Open3D geometries:
```py
bsdf_rough_plastic = mi.load_dict({
    'type': 'roughplastic',
    'diffuse_reflectance': {
        'type': 'rgb',
        'value': [0.1, 0.1, 0.1]
    },
    'alpha': 0.1,
    'int_ior': 'bk7',
    'ext_ior': 'air',
})
mi_mesh = mesh.to_mitsuba('monkey', bsdf=bsdf_rough_plastic)
img = render_mesh(mi_mesh, mesh_center.numpy())
mi.Bitmap(img).write('test3.exr')

See examples/python/visualization/to_mitsuba.py for more details.

Visualization and GUI (other updates)

• [New] The Open3D viewer app is now available for Windows. This version is unsigned, so expect Windows warnings.

• [New] Add draw_web CLI for remote visualization. Try it out with
  (remotehost) $ open3d draw_web --bind_all /path/to/3D/file and then open the browser to http://remotehost:8888
• [New] Unprojection function in the ViewControl (contributed by @PieroV).
• [New] Added interaction methods for Python ImageWidget (contributed by @MartinEekGerhardsen).
• [Update] Updated customized_visualization tutorial (contributed by @ClarytyLLC).
• [Update] Convert material properties when converting from legacy to tensor TriangleMesh.
• [Fix] O3DVisualizer::RemoveGeometry() function vector subscript out of range (contributed by @UnadXiao).
• [Fix] Fast check for 3D tensorboard data. Open3D does not delay Tensorboard startup for large run folders now.
• [Fix] Fix scrolling direction bug for TextEdit (contributed by @cansik).
• [Fix] VisualizerWithVertexSelection: Call selection_changed callback, when deselecting all points (Ctrl + R) (contributed by @micsc12).

Geometry

• [New] robust plane detection (contributed by @plusk01).

• [New] Minimal bounding box computation for point clouds (contributed by @ligerlac).
• [New] Added RemoveStatisticalOutliers to tensor PointCloud. (contributed by @yuecideng).
• [New] Added ComputeTriangleNormals ComputeVertexNormals NormalizeNormals to tensor Trianglemesh. (contributed by @yuecideng).
• [New] Added normals oriented functions to tensor PointCloud. (contributed by @yuecideng).
• [New] Added tensor based OrientedboundingBox. (contributed by @yuecideng).
• [New] Added tutorial for tensor based PointCloud.
• [New] Enable python copy & deepcopy for tensor geometry. (contributed by @yuecideng).
• [New] Added partitioning of point clouds and triangle meshes using PCA.

https://user-images.githubusercontent.com/41028320/223511665-9e4f75f0-5da5-42c4-97ff-4d0a6df432a9.mp4

• [New] Added a function to extract faces in a triangle mesh with a binary mask.

  

• [New] UV MAPS Tutorial (contributed by @samliozge).

• [Update] Added voting and ray jitter to RaycastingScene to improve robustness of signed distance queries.

• [Update] Improved speed of CreateFromTriangleMeshWithinBounds() by > 100x (contributed by @Hodong-Hwang).

• [Update] Added parallelization to UV atlas computation by partitioning the mesh.

• [Update] Function CreateFromVoxelGrid is made static (contributed by @mjaein).

• [Fix] Fixed wrong voxel center calculation in CreateFromTriangleMeshWithinBounds() (contributed by @plusk01).

• [Fix] Replace Vectors from internal embree headers with Eigen.

• [Fix] Use same beta value in SamplePointsPoissonDisk() as in the paper.

• [Fix] Fix Python Image calculations (pybind Image::To argument order).

• [Fix] Consistent face orientation for generated alpha shape.

Pipelines

• [New] Tensor reconstruction system upgrade: make fragments (contributed by @theNded).
• [Update] Added additional python bindings for RGBD odometry functionality (contributed by @jdavidberger).
• [Update] release python GIL for generalized icp (contributed by @yuecideng).
• [Fix] fix call to AdvancedMatching in FGR (contributed by @plusk01).
• [Fix] Update ICP fitness and rmse value at the end of the loop (contributed by @bialasjaroslaw).

I/O

• [Update] Add loading colors and uvs in tgeometry TriangleMesh IO.

Core

• [New] Support for arbitrary imports with Open3D submodules in Python. Now you can use any of the following and more:

  from open3d.t.geometry import TriangleMesh
  from open3d.ml.torch.ops import fixed_radius_search # pybind symbol
  from open3d.ml.tf.models import KPFCNN             # Python code
  from open3d.visualization import gui  # pybind symbol
  from open3d.visualization import draw # Python code
  from open3d.visualization.gui import Color
  from open3d.visualization.rendering import Camera

  No need for awkward shortcuts such as:

  import open3d as o3d
  Tensor = o3d.core.Tensor
  Image = o3d.t.geometry.Image

• [New] unary minus operator (contributed by @yxlao).
• [Update] Replace license text with SPDX identifier.
• [Fix] Fixed error when use CUDA_CALL without open3d namespace. (contributed by @yuecideng).
• [Fix] Fixed multi-device CUDA switching bug. (contributed by @yxlao).
• [Fix] MiniVec's constructor shall not call Device's constructor/destructor (contributed by @yxlao).
• [Fix] Add neighbor search module to the documentation.

Build System

• [New] The Open3D viewer app is now available for Windows.
• [New] New lighter wheel (~100MB) for x86_64 Linux, which is suitable for systems without CUDA. Try it out with:
  pip install open3d-cpu.
• [Update] Update imgui to latest version (contributed by @roehling).
• [Update] Support CUDA 12.0 build. (contributed by @NobuoTsukamoto).
• [Update] Add options to use system libraries for new dependencies (contributed by @roehling).
• [Update] Hide un-needed DSO symbols in Linux and macOS.
• [Update] Dependency upgrades for security fixes: civetweb, curl, zlib, libjpeg-turbo, wheel, setuptools, werkzeug, nbformat, ipywidgets, oathlib, certifi .
• [Fix] Build with libomp 11.1 and package in macOS wheel for interoperability with PyTorch.
• [Fix] Fix Mac build issue with ASSIMP.
• [Fix] Wstringop-owerflow compilation error for GCC (contributed by @bialasjaroslaw).
• [Fix] Support msgpack versions without cmake (contributed by @cielavenir).
• [Fix] fix Python 3.8 cuda dll loading on Windows (contributed by @yxlao).
• [Fix] More portable ARM system detection.
• [Fix] Fix build compatibility with fmtlib 8 (contributed by @roehling).

Open3D-ML

• [New] PointPillars Waymo distributed training model weights and docs.
• [New] 3DML support with PyTorch and TensorFlow for Apple Silicon.
• [Update] Update PyTorch to v1.13.1.
• [Update] RandLaNet update_probs label smoothing logic (contributed by @mariusud).
• [Fix] Update broken links (contributed by @Birkenpapier and @friendship1).

Known Issues [WIP]

• The Python wheel may crash when run on Apple Silicon systems, especially on M2. (#5951)
• The Windows app is unsigned, so expect warnings from the system.

Acknowledgment

We would like to thank all of our community contributors for their true labor of love for this release!

@bialasjaroslaw @Birkenpapier @cansik @cielavenir @ClarytyLLC @friendship1 @geppi @Hodong-Hwang @jdavidberger @johnthagen @ligerlac @mariusud @MartinEekGerhardsen @micsc12 @mjaein @NobuoTsukamoto @PieroV @plusk01 @roehling @samliozge @theNded @UnadXiao @yuecideng @yxlao

Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Open3D 0.16 Release Notes

The fall brings a new "tock" release of Open3D, packed with new features and updates! Here are the highlights:

• Open3D visualization now works on Google Colab, and CPU rendering is now easy to use on all Linux machines.
• The visualizer has a new wireframe mode.
• Many new geometry processing algorithms were added, including mesh clipping, hole filling, extrusion and generation of text meshes.
• Added support for texture baking to create great looking 3D models.
• The Open3D standalone viewer is now available for Apple Silicon.
• Dependency version updates: Python 3.10, Tensorflow 2.8, PyTorch 1.12.

Google Summer of Code

Open3D had a successful Google Season of Code 2022 with many new features added, and more in the works for the next release. Here are the features that are part of this release:

Geometry processing algorithms for tensor-based geometry

[Contributed by @ZhengyuDiao]

• [new] Geometry creation functions: create_arrow(), create_box(), create_cone(), create_coordinate_frame(), create_cylinder(), create_icosahedron(), create_mobius(), create_octahedron(), create_sphere(), create_tetrahedron(), create_torus().
• [update] Improved support for reading / writing text format point cloud files in XYZ, XYZN, XYZI and XYZRGB formats.
• [fix] Fix the png/jpg textures loading in glb (binary glTF) file.
• [update] Tensor point cloud IO now uses float instead of double by default.

Open3D visualization on Google Colab

[Contributed by @FromBei-JingWithLove]

• [new] draw_plotly method brings interactive 3D visualization for Open3D to Jupyter notebooks and cloud environments (e.g., Colab).
• [new] Tutorial notebook demonstrating the various ways to use draw_plotly
  

Build System

• [new] Open3D is now available for Python 3.10. Python 3.6 is no longer supported.
• [new] C++ users can now link to the Open3D dynamic library through pkg-config files. There are available for Linux and macOS in the Open3D binary packages.
• [new] Alpha support for SYCL devices (AlderLake integrated GPUs, Arc GPUs).
• [fix] Windows and Visual Studio build fixes (contributed by @nigels-com, @jmherzog-de, @yuecideng)
• [fix] Fixed build Error on Fedora 36 with GCC 12 (contributed by @NobuoTsukamoto)
• [fix] Fix link error during build due to invalid LIBDIR in newer Debian systems (contributed by @bchretien)
• [update] Improved support for building on Apple Silicon.
• [update] Separate Python build (requirements_build.txt) and runtime (requirements.txt) dependencies. (with help from @johnthagen).
• [update] Documentation can be built faster with the make_docs.py --parallel option.

Core

• [new] Enable pickling for tensor and tensor based geometry (contributed by @yuecideng).
• [new] Random seed Open3D globally. Utility random number generation functions to ensure the global Open3D random seed is used.

  import open3d as o3d
  o3d.utility.random.seed(42)

  #include "open3d/utility/Random.h"
  int main() {
      using namespace open3d;
      utility::random::Seed(42);  // Globally seed.
      std::cout << utility::random::RandUint32() << std::endl;  // Simply get a random number.
      return 0;
  }

• [update] Faster creation of many small Tensors by using statically allocated memory for SizeVector.
• [new] Add tensor minimum(), maximum() ops (contributed by @yuecideng).
• [new] Add tensor Any(), All(), RemoveNonFinite()
• [update] Support for int64 index dtype in NearestNeighborSearch (contributed bu @chrockey)
• [update] The new datasets API is used throughout Open3D, including pipelines, examples and unit tests. New models with PBR materials (Helmet, Jesse monkey, crate and sword) are available. Add redwood indoor datasets (Augmented ICL-NUIM Dataset).
• [update] Support RadiusSearch for EstimateNormals() for Tensor PointCloud (contributed by @yuecideng).

Geometry

• [update] Tensor geometry attributes can now be accessed by the easier dot notation, in addition to the previous dictionary notation. For example:

  pcd.point["colors"] is pcd.point.colors
  tmesh.triangle["normals"] is tmesh.triangle.normals

Geometry classes of the Tensor API also add new functionality based on VTK:

• Mesh clipping along a plane
• Extraction of slices from triangle meshes as line sets
• Hole filling
• Boolean operations between watertight meshes
• A new algorithm for quadric decimation of triangle meshes
• Linear and rotational extrusion
• Generation of text meshes

Further, we add functionality for parametrising meshes with the UVAtlas library and added functions for baking vertex and triangle attributes to textures.

New operations for Tensor PointCloud:

• ClusterDBScan, ConvexHull

• Radius Outlier Filter, RemoveDuplicatedPoints, PaintUniformColor

• FarthestPointDownSample, HiddenPointRemoval and SegmentPlane

• Uniform and random down sample methods (contributed by @yuecideng).

• FPFH features. (contributed by @yuecideng)

• PointCloud boundary detection (contributed by @yuecideng)

  PointCloud	Boundaries

• [new] RemoveDuplicatedPoints() for PointCloud (Eigen API). (contributed by @scimad)

• [new] Farthest point downsampling (Eigen API) (contributed by @yuecideng).

• [update] Plane segmentation in point clouds is now multi-threaded. (Eigen API) (contributed by @yuecideng).

• [update] Add SelectByIndex and minor improvement to SelectByMask (contributed by @yuecideng).

• [new] Add Tensor AxisAlignedBoundingBox (contributed by @yuecideng).

• [update] SelectPolygonVolume.CropInPolygon returns point indices.

• [fix] Blocky / incomplete raycast output for larger voxel sizes fixed with EstimateRange() update (contributed by @jdavidberger).

• [update] Allow setting of number of threads that are used for building a raycasting scene (contributed by @jjabo)

Visualization

• [new] Wireframe mode added to Open3D visualizer
  
• [new] Support for Filament's bloom post-processing effects
• [fix] Make sure float and uint16 texture formats work correctly
• [fix] Fix bugs with point_width and line_width parameters in Material class

CPU rendering

• [update] Easy to use out of the box CPU rendering for Linux. Pre-built Mesa CPU rendering library is provided in the Python wheel if the system Mesa library is too old or not in use. You can enable CPU rendering with the environment variable OPEN3D_CPU_RENDERING=true before importing Open3D in Python or running a C++ program. See the tutorial for full details.
• [new] Updated CPU rendering works in Jupyter and Colab environments.
  

GUI

• [update] Vertical and horizontal radio button groups. (contributed by @forrestjgq)
• [new] Keypress event callback for windows. (contributed by @forrestjgq)
• [fix] Fix mouse event filter issue in popup menu covered area (contributed by @forrestjgq)
• [new] Added extra VisualizerWithVertexSelection point picking functions to python API (contributed by @d-walsh and @cansik)
• [update] Support double click mouse event in WebRTC. (contributed by @forrestjgq)
• [fix] Fix Point Clouds Getting Culled Prematurely.
• [fix] Set correct default material (lit with flat shading) for triangle mesh with triangle normals.
• [update] Add interface for get cropped geometry from VisualizerWithEditing class (contributed by @yuecideng).

IO

• [new] Open3D now implements a distortion and noise model from the Redwood dataset. This will enable you to simulate the output of realistic depth sensors.

Original	With noise + distortion	Difference

• [fix] Fix gltf PBR model load regression when loading through ASSIMP.
• [new] Read ply, stl, obj, off, gltf, glb, fbx file formats directly into a Tensor TriangleMesh.

Reconstruction

• [update] Update ICP interface to get iteration wise loss information.
• [new] Online SLAM Example with Realsense sensor support, bagfile support, and saved dataset support.
• [new] Legacy offline reconstruction C++ pipeline (contributed by @yuecideng)
• [update] InitializePointCloudForColoredICP efficiency improvement. (contributed by @Xiang-Zeng)

Open3D-ML

• [update] Upgrade PyTorch to v1.12 and TensorFlow to v2.8.2. With the new PyTorch version, Open3D PyTorch ops work directly with standard PyTorch wheels and custom wheels are not required any more.
• [fix] Avoid logging.basicConfig in non-scripts (contributed by @ajprax)
• [fix] filter_valid_label: force scores tensor onto the selected device (contributed by @ntw-au)
• [fix] Ensure that a checkpoint is saved on the final training epoch (contributed by @ntw-au)
• [fix] Handle nested dictionaries when merging configs (contributed by @ntw-au)
• [fix] Fix moving zip files (contributed by @tejaswid)
• [fix] Update RandLANet weights link (contributed by @Krupal09)
• [fix] Getting rid of torch._six (for future PyTorch versions) (contributed by @krshrimali)

Acknowledgments

We would like to thank all of our community contributors for their true labor of love for this release!

@ntw-au, @jdavidberger, @Xiang-Zeng, @jamesdi1993, @brentyi, @jjabo, @jbotsch-fy, @scimad, @cansik, @NobuoTsukamoto, @theNded, @chunibyo-wly, @jmherzog-de, @luzpaz, @code-review-doctor, @d-walsh, @johnthagen, @pmokeev, @erbensley, @hanzheteng, @chrockey, @agrellRepli, @bchretien, @nigels-com, @forrestjgq, @equant, @naruarjun, @ajprax, @INF800, @ntw-au, @tejaswid, @Krupal09, @krshrimali

Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Open3D 0.15 Release Notes

We are excited to bring you the best Open3D yet - version 0.15. Take a look below:

Starting from this release, we adopt a "tick-tock" model for balancing resolving issues vs. adding new features. In a nutshell, the "tick" releases are focused on resolving existing issues and eliminating bugs, while the "tock" releases mainly focus on developing new features. Open3D 0.15 is a "tick" release. We resolved over 500 issues for Open3D and Open3D-ML, as the infographic below illustrates.

Google Summer of Code 2022

Open3D has applied for the Google Summer of Code 2022 to increase community participation. Check out details and our project ideas here. Please help in making Open3D better for all.

Build System

• [New] We now provide Open3D binary packages for C++ users. No need to “Build from Source” - just download a binary package for your Operating System and use it “out of the box”. See GitHub releases for v0.15 and getting started guides for the latest development package.
• [New] Docker build tools to build ARM64 Linux wheels and multiple Python versions. See ARM64 build guide for more details.
• [New] Pre-compiled Open3D wheel for ARM64 Linux and macOS. Improved Apple Silicon support. Install Open3D on ARM64 Linux and macOS with pip install open3d.
• [Update] Open3D now builds with the new CXX11 ABI by default on Linux. Set -DGLIBCXX_USE_CXX11_ABI=OFF in cmake if you need the old ABI, e.g. to work with PyTorch / TensorFlow libraries.
• [Update] Starting with version 0.15, Open3D Conda packages are no longer supported. Install Open3D with pip install open3d inside a Conda virtual environment.

Core

Datasets

• [New] Dataset module for automatically downloading and managing example data. The following example demonstrates how to create a Dataset object, extract its path, and display it in the Open3D Visualizer:
  

  import open3d as o3d
  
  if __name__ == "__main__":
      dataset = o3d.data.EaglePointCloud()
      pcd = o3d.io.read_point_cloud(dataset.path)
      o3d.visualization.draw(pcd)

  #include 
  #include 
  #include "open3d/Open3D.h"
  
  int main() {
      using namespace open3d;
  
      data::EaglePointCloud dataset;
      auto pcd = io::CreatePointCloudFromFile(dataset.GetPath());
      visualization::Draw({pcd});
  
      return 0;
  }

Command Line Interface (CLI)

• [New] Open3D-dedicated Command Line Interface (CLI) for visualization and running Python examples. Below is a code snippet to get started with Open3D and its examples.

  # Install Open3D pip package
  pip install open3d
  
  # Print help
  open3d --help
  
  # List all runnable examples
  open3d example --list
  
  # Print source code of an example
  open3d example --show [category]/[example_name]
  
  # Run an example
  open3d example [category]/[example_name]
  
  # Run Open3D Viewer
  open3d draw
  
  # Open a mesh or point cloud file in Open3D Viewer
  open3d draw [filename]

  

• [Update] Python examples directory has been refactored for better namespace consistency and new examples have been added.

Neighbor search

• [Update] Updated neighbor search module. The neighbor search module is now equipped with highly optimized built-in implementations of all search methods (Knn, Radius, and Hybrid) supporting both CPU and GPU. Faiss build dependency is removed.

Visualization and GUI

• [New] Introducing raw mode visualization. The raw mode automatically simplifies the lighting environment and object materials to make it easier to inspect the underlying geometry of point clouds and triangle meshes.
  

• [New] Open3D new visualizer now features CPU Rendering based on Mesa’s LLVMpipe software OpenGL implementation. Interactive applications, demos, and Python scripts are all supported as well as off-screen rendering. This feature can also be used within Docker.

  • Method 1: LD_PRELOAD from the command line

    LD_PRELOAD=/home/open3d/development/mesa-21.3.4/libGL.so python examples/python/visualization/draw.py

  • Method 2: Preload library in Python

    import ctypes
    ctypes.cdll.LoadLibrary('/home/open3d/development/mesa-21.3.4/libGL.so')
    import open3d as o3d
    
    mesh = o3d.io.read_triangle_model('/home/open3d/development/FlightHelmet/FlightHelmet.gltf')
    o3d.visualization.draw(mesh)

    

• [New] WidgetProxy and WidgetStack widgets allow the creation of user interfaces on the fly (contributed by @forrestjgq).
• [New] Background color option for button widgets (contributed by @forrestjgq).
• [New] Set maximum visible items in a list widget to prevent lists from growing too large (contributed by @forrestjgq).
• [New] Function keys are now bindable (contributed by @forrestjgq).
• [New] Support for specifying intrinsic projection matrix in the new visualizer.
• [New] Add support for scaling 3D labels.
• [Fix] Open3D for TensorBoard plugin does not need Open3D-ML now.
• [Fix] Point picking, text input, and UI layout (contributed by @forrestjgq).

Geometry

• [Fix] Oriented Bounding Box
  • Fixed an issue where the orientation of the OrientedBoundingBox was mirrored.
  • [New] added a new parameter for robust oriented bounding box computation for degenerated point clouds.
• [Fix] Convex hull meshes created from point clouds now have outward-pointing triangles.
• [Update] Added a new parameter for robust convex hull computation.
• [Update] TriangleMesh GetSelfIntersectingTriangles() and related functions like IsWatertight(), GetVolume(), etc. are now more than 4 times faster.
• [Fix] Corrected an issue with io::AddTrianglesByEarClipping() where the algorithm could fail for concave polygons.
• [New] New Python examples for reconstruction and voxelization.
• [Fix] Improved logger performance.

Open3D-ML

• [New] MIT-licensed implementation of RandLANet.
  
• [New] Intel OpenVINO inference backend (contributed by @dkurt).
• [Fix] Fixed an issue with S3DIS where the loss gets NaN after a few epochs.
• [Fix] Fixed an issue with IoU calculation which fails for large point clouds while running inference in patches.
• [Fix] Fixed an issue where the labels are not correctly ordered in the visualizer.
• [New] Support for Images in Dataset Visualizer (contributed by @ajinkyakhoche).
  

Acknowledgment

We would like to thank all of our community contributors for their true labor of love for this release!

@ajinkyakhoche @ceroytres @chunibyo-wly @dkurt @forrestjgq @Fuhrmann-sep @jeertmans @junha-l @mag-sruehl @maxim0815 @Nicholas-Mitchell @nigels-com @NobuoTsukamoto @ntw-au @roehling @theNded

Also thanks to the many others who helped the Open3D community by reporting as well as resolving issues.

Open3D 0.14 Release Notes

We are excited to present the new Open3D version 0.14!

In this release, you will find:

• TensorBoard visualization
• Upgraded GUI API
• New RayCastingScene class
• Improved Tensor performance
• Improved I/O performance
• New 3D learning models in Open3D-ML: Point Transformer and PVCNN
• Improved interoperability with Numpy

Changes in Installation and Build system

• Open3D now works with Python 3.9. We release Open3D pre-compiled Python packages in Python 3.6, 3.7 3.8, and 3.9.
• Open3D 0.14 is the last version that supports conda installation. Starting from version 0.15, users will need to install Open3D with pip install open3d. We recommend installing Open3D with pip inside a conda virtual environment.
• Git submodules are no longer required in Open3D. You can now clone Open3D with git clone https://github.com/isl-org/Open3D.git without the --recursive flag. Also please note the updated Github URL.
• Open3D will now build in Release mode by default if CMAKE_BUILD_TYPE is not specified. Python is no longer required for building Open3D for C++ users.
• Open3D-ML is now recommended to be used along with PyTorch 1.8.2 and/or Tensorflow 2.5.2. Checkout Open3D-ML for more information.

Tensorboard Integration

Now you can use Open3D within Tensorboard for interactive 3D visualization! At a glance, you can:

• Save and visualize geometry sequences and their properties. This enables interactive visualization and debugging of 3D data and 3DML model training.
• Visualize 3D semantic segmentation and object detection with input data, ground truth, and predictions. In addition, any custom properties for a PointCloud, from scalar to vector, can be easily visualized.
• Synchronize time steps and viewpoints during different runs. This helps debug and monitor the effect of parameter tuning.

Rich PBR materials

Object detection

Semantic segmentation

To get started, write some sample geometry data to a TensorBoard summary with this snippet:

from torch.utils.tensorboard import SummaryWriter  # TensorFlow also works, see docs.
import open3d as o3d
from open3d.visualization.tensorboard_plugin import summary
from open3d.visualization.tensorboard_plugin.util import to_dict_batch
writer = SummaryWriter("demo_logs/")
cube = o3d.geometry.TriangleMesh.create_box(1, 2, 4)
cube.compute_vertex_normals()
colors = [(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0)]
for step in range(3):
    cube.paint_uniform_color(colors[step])
    writer.add_3d('cube', to_dict_batch([cube]), step=step)

Now you can visualize this in TensorBoard with tensorboard --logdir demo_logs. For more details on how to use TensorBoard with Open3D, check out this tutorial.

Visualizer

Further enhancements have been added to the GUI viewer. Now you can:

• Directly visualize tensor-based geometry classes including PointCloud, TriangleMesh, and LineSet.
• Use physically based rendering (PBR) materials that deliver appealing appearance.
• New default lighting environment and skybox improves visual appeal

• Use all the functionality in Tensorboard!

  

import open3d as o3d
import open3d.visualization as vis
a_sphere = o3d.geometry.TriangleMesh.create_sphere(2.5, create_uv_map=True)
a_sphere.compute_vertex_normals()
a_sphere = o3d.t.geometry.TriangleMesh.from_legacy(a_sphere)
# Compare this...
vis.draw(a_sphere)
a_sphere.material = vis.Material('defaultLit')
a_sphere.material.texture_maps['albedo'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Color.jpg')
a_sphere.material.texture_maps['roughness'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_Roughness.jpg')
a_sphere.material.texture_maps['normal'] =
    o3d.t.io.read_image('examples/test_data/demo_scene_assets/Tiles074_NormalDX.jpg')
# With this!
vis.draw(a_sphere)

A complete, complex demo scene can be found at examples/python/gui/demo-scene.py

Core

• The Open3D Tensor class received a major performance boost with the help of Intel ISPC compiler and optimization for the contiguous code path.
  
  (See python/benchmarks/core for the benchmark scripts. For each operation, the geometric mean of run times with different data types is reported. The time is measured with an Intel i9-10980XE CPU.)
• A major upgrade of Parallel HashMap is done. Now you can choose from multi-valued HashMap and HashSet depending your value types. A comprehensive tutorial is also available.
• Linear Algebra performance have been optimized for small matrices, especially on 4x4 transformations.
• Semantics for tensor and tensor-based geometry have been improved, especially on device selection.

• Functions expecting a Tensor now accept Numpy arrays and Python lists. For example:

  import open3d as o3d
  import numpy as np
  
  mesh = o3d.t.geometry.TriangleMesh()
  mesh.vertex['positions'] = np.array([[0, 0, 0], [1, 0, 0], [1, 1, 0]], dtype=np.float32)
  mesh.vertex['colors'] = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.float32)
  mesh.triangle['indices'] = [[0, 1, 2]]
  o3d.visualization.draw(mesh)

I/O

• We now support I/O from/to Numpy compatible .npz .npy formats for Open3D tensors and tensor maps. It is now easier to convert between Open3D geometry classes and Numpy properties.
• We have improved I/O performance for tensor-based point cloud and triangle-mesh file-formats, including .ply, .pcd, .pts. Geometry loading time is hence improved for the stand-alone visualizer app.
• We added support for material data to the MessagePack serialization format of the RPC module.

Geometry

• We introduce a new class RaycastingScene with basic ray intersections functions and distance transform for meshes, utilizing the award-winning Intel Embree library.

  Example code for rendering a depth map:

  import open3d as o3d
  import matplotlib.pyplot as plt
  
  # Create scene and add a cube
  cube = o3d.t.geometry.TriangleMesh.from_legacy(o3d.geometry.TriangleMesh.create_box())
  scene = o3d.t.geometry.RaycastingScene()
  scene.add_triangles(cube)
  
  # Use a helper function to create rays for a pinhole camera.
  rays = scene.create_rays_pinhole(fov_deg=60, center=[0.5,0.5,0.5], eye=[-1,-1,-1], up=[0,0,1],
                                 width_px=320, height_px=240)
  
  # Compute the ray intersections and visualize the hit distance (depth)
  ans = scene.cast_rays(rays)
  plt.imshow(ans['t_hit'].numpy())

  Distance transform generated with RaycastingScene:

  

  See the tutorials for more information (Ray casting, Distance queries).

• Normal estimation for tensor PointCloud is supported with the tensor-compatible nearest neighbor search modules.

• Customizable tensor-based TriangleMesh, VoxelBlockGrid, and LineSet are implemented that allow user-defined properties. For example:

  import open3d as o3d
  import open3d.core as o3c
  
  mesh = o3d.t.geometry.TriangleMesh()
  mesh.vertex["positions"] = o3c.Tensor([[0.0, 0.0, 1.0],
                                       [0.0, 1.0, 0.0],
                                       [1.0, 0.0, 0.0],
                                       [1.0, 1.0, 1.0]], dtype=o3c.float32)
  mesh.vertex["my_custom_labels"] = o3c.Tensor([0, 1, 2, 4], dtype=o3c.int32)
  mesh.triangle["indices"] = o3c.Tensor([[0, 1, 2], 
                                       [1, 2, 3]], dtype=o3c.int32)

Pipelines

• We have enhanced point cloud registration (ICP) with a tensor interface:
  • Float64 (double) precision point cloud is supported for a higher numerical stability
  • Robust Kernels, including Huber, Tukey, and GM losses are supported for robust registration.
  • Colored-ICP is now supported in the unified tensor-based API.
  • See this tutorial for more details.
• We also provide an initial tensor-based reconstruction system in Python, including
  • Customizable volumetric RGB-D integration;
  • Dense RGB-D SLAM with a GUI;
  • See this tutorial for more details.

Open3D-ML

The Open3D-ML library welcomes more state-of-the-art models and operators that are ready to use for advanced 3D perception, especially semantic segmentation, including

• New state-of-the-art Point Transformer for Semantic Segmentation.
  
• Highly Efficient Point-Voxel Convolution for Semantic Segmentation.
  
• RaggedTensor integration that enables batch SparseConvolution and SparseConvolutionTranspose along with PyTorch.
• Batched voxelization for fast point-voxel conversions.

Refer to the tutorial for training and inference on new models. (PyTorch TensorFlow).

Expreimental ARM64 builds

• Experimental Ubuntu ARM64 build
  • Python 3.6: open3d-0.14.1-cp36-cp36m-linux_aarch64.whl
  • Python 3.7: open3d-0.14.1-cp37-cp37m-linux_aarch64.whl
  • Python 3.8: open3d-0.14.1-cp38-cp38-linux_aarch64.whl
  • Python 3.9: open3d-0.14.1-cp39-cp39-linux_aarch64.whl
  • Visualizer does not work on Raspberry Pi due to the lack of full OpenGL
  • Tested on Nvidia Jetson boards with Ubuntu 18.04
  • You may use pyenv to setup your virtualenvs
• Experimental Apple Silicon M1 ARM64 build:
  • Python 3.8: open3d-0.14.1-cp38-cp38-macosx_11_0_arm64.whl
  • Python 3.9: open3d-0.14.1-cp39-cp39-macosx_11_0_arm64.whl
  • Tested on M1 macOS Monterey
  • You may use miniconda to setup your virtualenvs
• Test your installation

  pip install open3d-xxx.whl
  # Test the new vidualizer
  python -c "import open3d as o3d; c = o3d.geometry.TriangleMesh.create_box(); o3d.visualization.draw([c])"
  # Test the traditional visualizer
  python -c "import open3d as o3d; c = o3d.geometry.TriangleMesh.create_box(); o3d.visualization.draw_geometries([c])"

---

We hope you find Open3D 0.14.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

• e-mail
• Forum
• Discord network

Acknowledgment

We thank all the community contributors for this release!

(alphabetical order)
@cclauss
@chrockey
@chunibyo-wly
@cosama
@forrestjgq
@gsakkis
@junha-l
@ktsujister
@leomariga
@li6in9muyou
@marcov868
@michaelbeale-IL
@muskie82
@nachovizzo
@NobuoTsukamoto
@plusk01
@reyanshsolis
@ShreyanshDarshan
@ShubhamAgarwal12
@SoftwareApe
@stanleyshly
@stotko
@theNded
@zhengminxu

Open3D 0.13.0 Release Notes

We welcome you to the 0.13.0 release of Open3D. This release is full of exciting new features with a strong emphasis in real-time pipelines, but also full of bug fixes and usability improvements. The big highlights of this release are as follows:

• New real-time 3D reconstruction pipeline, featuring GPU and CPU support based on VoxelHashing.
• New real-time point cloud registration algorithm, featuring a high-performance version of Iterative Closest Point (ICP).
• New Neighbor Search module, introducing your favorite search algorithms such as KNN and RadiusSearch, with support for GPU and CPU devices through a common interface.
• New web visualizer, which enables users to access the advanced rendering and visualization features of Open3D in your favourite web environments (remote and locally!), including Jupyter notebooks, Jupyter lab, and standalone web applications.
• New 3D machine learning models and datasets, featuring PointRCNN for 3D object detection, SparseConvNets for point cloud semantic segmentation, and support for ScanNet and SunRGBD.
• Upgraded GUI module, providing improved and more versatile versions of existing widgets, and new ones: ImageWidget and ToggleSwitch.
• Upgraded build system, adding support for CUDA 11.

Real-time 3D reconstruction

We introduce a new CUDA accelerated pipeline including RGBD odometry, frame-to-model tracking, and volumetric integration.

Figure 1. Example of 3D reconstruction from an RGB-D sensor.

Odometry

We introduce the tensor based real-time RGBD Odometry pipeline. In addition to the legacy Hybrid and Intensity based methods, we support the popular point-to-plane method.

TSDFVoxelGrid

We further accelerate volumetric integration and introduce fast ray casting for rendering.

VoxelHashing

Based on the accelerated RGBD odometry and raycasting, we present the fully functional VoxelHashing system. It performs dense volumetric reconstruction with fast frame-to-model tracking. We present an easy-to-use GUI that also shows real-time interactable surface reconstruction.

SLAC

We have further enhanced our legacy offline reconstruction system by introducing the Simultaneous Localization and Calibration (SLAC) algorithm. This algorithm applies advanced dense multi-way registration along with non-rigid deformation to create highly-accurate reconstructions.

Real-time point cloud registration

We present a high-performance implementation of ICP using Open3D’ Tensor library. This module is one of the first on leveraging the new Neighbor search module and the newly crafted parallel kernels. This implementation brings support for multi-scale ICP, which allows us to do iterations on different resolutions in order to accelerate convergence while keeping computation low.

Figure 2. ICP registration of multiple point clouds from a driving dataset.

New Neighbor Search module

Neighbor search is at the core of many 3D algorithms. Therefore, it is critical to have access to a fast implementation able to execute a large number of queries in a fraction of a second. After months of development, the Open3D team is proud to present the new Neighbor Search module!

This module brings support for core search algorithms, such as KNN, Radius search, and Hybrid search. All these algorithms are provided with support for both CPU and GPU, through a common and easy-to-use interface. Write your code once and support multiple devices! Moreover, we have not sacrificed a single flop of computation, making this module one of the fastest neighbor search libraries ever created.

Web visualizer

The need for visualizing complex 3D data in web environments has surged considerably in the past few years, in part thanks to the proliferation of sensors like LIDAR and RGBD cameras. New use cases, such as online dataset inspection and remote visualization are now an integral part of many tasks, requiring the crafting of ad-hoc tools, which often are cumbersome to use.

Figure 3. Standalone visualization of a semantic segmentation model in a browser.

In order to improve this situation, we introduce our new web-based visualization module, which enables 3D visualization from any browsers and any location. This module lets users run advanced rendering and visualization pipelines, both remote and locally through your web browser. All the power of Open3D’ rendering engine --including support for PBR materials, multiple lighting systems, 3D ML visualization, and many other features--, are now supported in your browser. This module also includes a Jupyter extension for interactive web-based visualization! This new feature allows you to run compute-intensive 3D processing in a dedicated server while visualizing the results remotely on any device through your browser.

Figure 4. Visualization of a 3D model on a Jupyter notebook.

3D Machine Learning

Architectures

In this release, we introduce a new point cloud semantic segmentation architecture based on a Sparse Convolution-based UNet model. This architecture leverages the new sparse convolution operators provided by Open3D, and achieves state of art performance for Semantic Segmentation on the ScanNet dataset. We have also added support for PointRCNN for the task of 3D object detection. To enable PointRCNN we have added new operators like furthest_point_sampling, three_interpolate, ball_query, which are available through Open3D for TensorFlow and Pytorch.

Figure 5. Example of 3D semantic segmentation using a SparseConvUNet model on ScanNet.

Figure 6. Example of 3D object detection using a PointRCNN on KITTI.

All these models are provided with their respective training and inference pipelines with support for TensorFlow and Pytorch. Pre-trained models are also provided (check out the following link).

Datasets

This release brings the following datasets: Scannet and SunRGBD.

Frameworks

We now support all models on newer versions of TensorFlow (2.4.1) and PyTorch (1.7.1), on CUDA 11.0.

GUI and rendering modules

Open3D 0.13.0 brings a cascade of improvements and fixes to the renderer and GUI modules.

• The camera can now be controlled with respect to a target object, and the pan-in/pan-out actions are smoother.
• The render supports render targets.
• The black screen issue in MacOS systems is now solved.
• Support for a video widget
• The background of the 3D scene widget can be updated with an image.
• Camera objects can be visualized in the 3D scene.
• Camera objects support intrinsic and extrinsic parameters.
• Support for multiple fonts.
• Support for a new ToggleSwitch widget.
• New GUI/rendering examples for common tasks, such as data acquisition from RGBD sensors, 3D reconstruction, and registration, among others.

Build System

Our pip packages now include support for CUDA 11.0, PyTorch 1.7.1, and TensorFlow 2.4.1 to enable RTX 3000 series devices. Please, notice that we provide custom PyTorch wheels for Linux to work around an incompatibility between CUDA 11, PyTorch, and extension modules such as Open3D-ML.

This release also brings new improved support for CUDA on Windows. Users can now build CUDA accelerated Python wheels for Windows. Open3D is now built with security options enabled by default.

---

We hope you find Open3D 0.13.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

• e-mail
• Forum
• Discord network

The Open3D team

---

Changelog:

Deprecating

• All visualization tools, such as draw_geometries will be deprecated in Open3D 0.14 in favor of the new O3DViewer.

Changes to Open3D

• add ml deps for conda build (#2826)
• tensorkey refactoring: pimpl and optional arguments (#2813)
• Add file and line for error message (#2829)
• color map optimization refactoring (#2832)
• remove openmp macro guards for color map optimization (#2835)
• easy way of initializing c++ tensor (#2796)
• Add numerical operations to Tensor (#2848)
• Implement np style arange for Tensor (#2849)
• Refactor TSDF kernels (#2854)
• core::Scalar class and refactor arange (#2857)
• octree updates (#2833)
• numpy-compatible tensor IO (#2859)
• Upgrade zlib and libpng, use ExteralProject (#2878)
• update mkl download links (#2882)
• Python RSBagReader.open() crash on Ubuntu (#2876)
• refactoring: add pipelines/kernel folder (#2875)
• Standarize To() and Copy() (#2877)
• compile 3rd-party library in a separate step for arm64 CI (#2891)
• Hardened security flags for building Open3D (#2883)
• fix incompatible CUB include dir with CUDA 11 and later (#2913)
• Jsoncpp with ExternalProject (#2907)
• Python example of using a texture on a Material (#2935)
• Add rendering for VoxelGrid and Octree to new renderer (#2932)
• Enable RSUSB backend only on Linux. (#2923)
• Added C++ example OffscreenRendering (#2914)
• Lazy load renderer when creating window in case Python crashes before window becomes visible (#2819)
• Fixed trackpad scrolling to be buttery smooth at slow speeds on macOS (instead of being very discrete-seeming) (#2964)
• add tunable parameter to filter outliers in surface extraction (#2931)
• Added close callback to window (#2958)
• Mode For Light To Follow Camera (#2944)
• Image processing Ops sample with IPP and NPP (#2898)
• Added ability to set camera in open3d.visualization.draw() (#2936)
• Mouse movement by spherical coordinates (#2962)
• Refactored RPC interface to be more amenable to variation in visualizers (fixes add not updating) (#2970)
• Optimised kernel for ICP (#2917)
• Add support for FPS-style mouselook to visualizer (#2551)
• maximize build space for ubuntu (#2986)
• Minor typo in repr for OBB (#2987)
• function for casting the tensor data pointer to a datatype (#2934)
• Undo lazy creation of renderer on Windows; it fixed the hang on exit when a python exception is generated before the run() happens, but broke normal windows (#2996)
• Add Image IO (#2955)
• Implement ray casting for TSDF voxel grids (#2941)
• Procedural Ground Plane (#2980)
• Mousewheel dollies based on distance from center of rotation rather than size of model (#3011)
• Update Mac build to use latest 1.9.11 Filament to fix illegal instruction error (#3013)
• Fix arm64 CI out-of-time (#3020)
• Add Ground Plane support to O3DViewer (#3014)
• Recenter rotation to point on double-click (#2905)
• Implement depth buffer read back (#3028)
• Remove unnecessary include causing unwanted compile dependencies (#3030)
• Update faiss_build.cmake (#2979)
• Added python bindings for rendering::Camera (#3033)
• PointRCNN ops (#3021)
• add int16 dtype and tests (#3035)
• enable ccache for cuda with CMAKE_CUDA_COMPILER_LAUNCHER (#3043)
• Support updating tpointcloud with the same or fewer vertices as original (#3048)
• Use RenderTargets For Scene Caching (#3031)
• Add init pybind for Label3D (#3045)
• Fix linking issue of Torch/TF Ops (#2957)
• GCE CUDA Python tests (#3063)
• Rearrange constructor so input missing "points" attribute reaches LogError (#3062
• Make sure bounding box is non-empty (#3061)
• Replace CMake GLOB and GLOB_RECURSE with file listing (#3059)
• Add pybind param names to docstring for 'TSDFVoxelGrid.integrate' (#3058)
• Support USE_SYSTEM_LIBREALSENSE flag. (#3057)
• Faster FixedRadiusSearch and HybridSearch (#2821)
• IPP/NPP image operators (#2972)
• Add function to randomly downsample from pointcloud (#3050)
• maximize build space (#3071)
• Add 'd' suffix to non-system libpng compiled library discovery for non-MSVC Debug builds (#3067)
• disable incompatible clang flags for now (#3080)
• [FileSystem] Fix GetPathComponents and unit tests on Windows. (#3070)
• Minor spell fix (#3081)
• Add disconnection function to AzureKinectSensor (#3044) (#3082)
• PointRCNN ops tf (#3075)
• Set post-processing coniguration when copying View settings (#3083)
• Fix crash if posted Window callback executes after Window is destroyed (#3084)
• style-guide link update (#3086)
• Update contribution_recipes.rst (#3088)
• changed Registration unit-test to use RegistrationPermuteDevices (#3095)
• LU, LU_with_ipiv, Det, Triu, Tril, Triul Ops (#2879)
• Set position in constructor (#3096)
• memory efficient hybrid search (#3073)
• Implemented ability to render to a bitmap window and interact with UI (#3072)
• Better error message if you don't initialize gui::Application before creating a window (#3103)
• Added pybind for Camera::CopyFrom() (#3104)
• Enable builds on Fedora. (#3102)
• [CUDA] Fix memory leaks. (#3099)
• Use UnownedPointer so Python doesn't retain a valid shared_ptr (#3106)
• Tensor nested initializer list with recursion (#3055)
• Fix window resize when Scene caching is enabled. (#3107)
• delete some ssh keys to keep the profiles small (#3110)
• Fixed crash when referencing submenus (#3114)
• Enable builds on Fedora (FAISS, MKL, TBB) and fix output result of CMAKE. (#3118)
• Added the double-click code from Window.cpp in #2905 that got clobbered by the WindowSystem refactor (#3119)
• TransformConverter supports Float32 and Float64 (#3098)
• Fix one frame blinking in Open3DViewer when layout changed or dialog box was closed (#3121)
• benchmark links 3rd-party library (#3124)
• Set byproducts for external projects. (#3138)
• Open3D C++ example projects as separete repos (#3128)
• gce disk size set to 48GB (#3169)
• download test datasets from intel-isl/open3d_downloads (#3105)
• Add ability to set center of rotation on SceneWidget, which is necessary for copying from a camera. (#3147)
• Minor(?) fix (#3175)
• Clone from specific commit instead of release branch (#3179)
• fix docstrings for ML modules (#3160)
• Add multi-window C++ and Python examples (#3167)
• Expose rendering to a depth image to Python, makes resulting depth image range from 0 (near plane) to 1 (far plane) instead of simply returning the raw values from Filament's buffer (which are reversed), fixes widgets that aren't SceneWidget or a Layout not working if they are a top-level child. (#3178)
• Fix incorrect coordinates for mouse events on Linux (#3177)
• Added gui::ToggleSwitch (#3182)
• Give Material::shader default value of "defaultUnlit" (#3184)
• Add support for textures in TriangleMesh::operator+= (#3170)
• TensorToEigenMatrix Ops. (#3163)
• assigning scalar to r-value tensor of arbitrary shape (#3137)
• t::PointCloud from RGBD image (#3113)
• openblas x86_64 docker (#3195)
• Updated copyrights to 2021 for gui (#3189)
• Added an example for line width (#3194)
• disable CreateFromRGBDImage for arm64 test (#3206)
• Don't allow empty geometry to be added to scene (#3203)
• upgrade scipy and cmake version (#3199)
• Added pybinds for color grading enums, also changed property open3d.visualization.rendering.Open3DScene.get_view to view (it is a property, not a function) (#3193)
• Fix for CreateFromRGBDImage test (#3207)
• speed up arm64 build with docker and ccache (#3205)
• [Windows] Fix linker errors and warnings. (#3151)
• Fix ModuleNotFoundError when build with GCC10. (#3142)
• Tensor-based RGBD Odometry (#3085)
• t-ICP ComputeTransform major performance improvements. (#3019)
• UV-Map for Box, Tetrahedron, Octahedron, Icosahedron (#3097)
• Init voxeldownsample (#3144)
• Fix compilation on Windows when building with CUDA. (#3036)
• Added mouse events + convert point to world coordinates example (#3192)
• Optimising Tensor RGBD Odometry. (#3213)
• handle the invalid tetra warning (#3201)
• SplitString returns tokens (#3223)
• clean up ci_utils for arm64 (#3224)
• less verbose tar print for arm docker (#3226)
• optimised hybrid search, clear tree from master (#3218)
• Added 3D labels to O3DVisualizer, added example (#3233)
• Add ability to set layout minor axis size, moved render target drawing from Open3DScene to SceneWidget which is a little more straightforward and also puts the image in the proper z-level so that SceneWidget can be used in layouts (#3212)
• Improves progress bar in Open3DViewer app, also fixes unlit material instead of lit (#3238)
• refactor logging (#3241)
• Adds ImageWidget and ability to efficiently change Scene background image (#3002)
• Update to filament v1.9.19 for mac for multi-window fixes (#3251)
• Fix spurious warning if a triangle mesh with no normals uses "default Unlit". Happens if you pick points (#3256)
• Fix camera matrix getting pybinds not returning something that can be converted to a Python object (#3257)
• Hashmap refactor (#3211)
• Added camera visualization to LineSet (#3255)
• Added header files for TriangleMeshModel to Open3D.h.in (#3261)
• Update stdgpu dependency to latest upstream (#3265)
• Use more sensible default color for point clouds with no per-vertex color (#3278)
• T-Registration pybinds and python-tests. (#3112)
• Fix memory leak in stdgpu hashmap backend (#3281)
• Upgrade Tensor-based RGBD odometry (#3229)
• UV Maps for Cone, Cylinder, Sphere added. (#3275)
• get_is_open() for CollapsableVert widget (#3286)
• Remove use of CMAKE_BUILD_TYPE. (#3164)
• Support TGeometry Image At Low Level and In Image Widget (#3277)
• simplify MANIFEST.in with graft (#3288)
• Optimised CopyCPU and CopyCUDA for case when dst is contiguous and src has single element. [Fill] (#3136)
• Added polygon picking (#3252)
• Add post_redraw() to O3DVisualizer so that it can be used non-blocking (#3262)
• Fixes crash/hang on exit if rendered to an image (#3273)
• voxel_down_sample pybind for t::pointcloud (#3293)
• Added italic, bold, and bold italic styles (#3298)
• Tensor IsInf, IsNan, IsFinite (#3302)
• Add remaining fixed-size integer dtypes (#3320)
• Port to CMAKE_CUDA_ARCHITECTURES (#3316)
• Tensor Multi Scale ICP (#3285)
• Fix many minor C++ documentation issues (#3326)
• update docs-build dependencies (#3335)
• Added tooltips (#3334)
• Defer layout of dialog to ensure ImGUI context is valid for CalcPreferredSize (#3339)
• Optimize TSDF-related operations (#3314)
• Unify dispatch macros from CoreUtil.h and Dispatch.h (#3295)
• Fix error message when no correct arguments are provided (#3292)
• Add material property for supplying vertex colors in sRGB (#3340)
• CUDA, TF, Pytorch version upgrade (#3049)
• fix header include (#3344)
• pass cuda inc dir to stdgpu to find thrust if cuda is not in a standard (#3343)
• fix conditions for showing warning. drop pytorch version test. (#3355)
• Remove attributes from pointcloud and trianglemesh (#3299)
• fixed int64_t issue on mac (#3291)
• Adjusted odd characters which were giving unicode charmap error (#3336)
• Improve performance of stdgpu hashmap backend (#3346)
• Add pybind for T PointCloud and fix associated bugs (#3345)
• Shubhamagarwal12/tensor mesh io (#3221)
• Update Linux and Windows platforms to Filament v1.9.19 (#3271)
• Fix multiple VectorEdits in a UI not working (#3360)
• Properly scale UInt8 colors when converting to Legacy Point Cloud (#3350)
• Make tooltips prettier (#3348)
• fix scalar boolean handling in pybind (#3362)
• Fixed identically named Checkboxes and ToogleSwitches not working, removed use of stringstream in the rest (#3369)
• Unqueue dead window events (#3370)
• Make sure ResizeRenderer gets called on BitmapWindow (#3374)
• Further improve performance of stdgpu hashmap backend (#3377)
• simplify jsoncpp cmake (#3174)
• [WebRTC] WebRTC library build scripts (#3349)
• Move vertexmap/normalmap related functions to t::Image (#3318)
• Revert "simplify jsoncpp cmake (#3174)" (#3386)
• Don't add menus in Jupyter that are not meaningful (#3385)
• Added ability to setup camera with extrinsic and intrinsic in new visualizer (#3338)
• Increase warning level (#3357)
• Clarify C++ standard in documentation (#3384)
• Optimize RGBD odometry (#3323)
• Python docs fix (#3373)
• Add Docker minimal example to getting_started docs. Fix #3388 (#3396)
• Introducing SLAC (#3108)
• Voxel hashing GUI (#3363)
• Fix screen turning black on resize in standalone viewer (#3394)
• Tensor Registration Update (#3352)
• SLAC Pybinds, Unit Tests, Documentation. (#3393)
• add template parameters for input and output tensors in the cconv op (#3353)
• RGBD Odometry Pybinds (#3359)
• Tensor::Clip() simple implementation (#3266)
• Fix filament byproducts and build from source. (#3375)
• Fix CUDA builds on Windows. (#3380)
• Replace templates with Scalar class (#3391)