Open3D 0.12.0 Release Notes
Open3D 0.12.0 is out, and it comes with new 3D object detection pipelines and datasets, newest versions of some of your preferred classic tools, and many bug fixes.
Keep reading for a summary of the most relevant features introduced in this release:
Extensions to the Open3D-ML module
The previous release of Open3D introduced an exciting new module dedicated to 3D Machine Learning Open3D-ML, featuring support for 3D semantic segmentation workflows. In this release, we have extended Open3D-ML with the task of 3D object detection. This extension introduces support for new datasets, such as the Waymo Open dataset, Lyft level 5 open data, Argoverse, nuScenes, and KITTI. As always, all these datasets can be visualized out-of-the-box using our visualization tool, from Python or C++. The visualization tool is now equipped with the capability to render 3D bounding boxes along with all the previously existing modalities, e.g. semantic labels, XYZRGB, depth, normals, etc.
Open3D-ML features
PointPillars, the first of the many object detection models to come in the near future. To enable the implementation of PointPillars, we have added a set of new ML operators in Open3D, such as: grid_sampling, NMS, and IOU. These operators are available to the community and can be used to build new models, using our Python and C++ APIs.
import os
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d
cfg_file = "ml3d/configs/pointpillars_kitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)
model = ml3d.models.PointPillars(**cfg.model)
cfg.dataset['dataset_path'] = "/path/to/your/dataset"
dataset = ml3d.datasets.KITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.ObjectDetection(model, dataset=dataset, device="gpu", **cfg.pipeline)
...
# run inference on a single example.
result = pipeline.run_inference(data)
We have also updated our model zoo, providing new pretrained models on KITTI for the task of 3D object detection, and new semantic segmentation models on Paris-Lille3D and Semantic3D.
Remember that all the tools provided in Open3D-ML are compatible with PyTorch and TensorFlow!
Support for RealSense SDK v2
RealSense sensors’ support has been upgraded to leverage the RealSense SDK v2. Users can now capture crisp 3D data from L515 devices. As part of this upgrade, we include support for Bag files format (RSBagReader), and direct streaming from sensors. These operations can now be done through a new sensor class: RealSenseSensor, offering a simple and intuitive way to control your sensors.
import open3d as o3d
bag_reader = o3d.t.io.RSBagReader()
bag_reader.open(bag_filename)
while not bag_reader.is_eof():
im_rgbd = bag_reader.next_frame()
# process im_rgbd.depth and im_rgbd.color
bag_reader.close()import json
import open3d as o3d
with open(config_filename) as cf:
rs_cfg = o3d.t.io.RealSenseSensorConfig(json.load(cf))
rs = o3d.t.io.RealSenseSensor()
rs.init_sensor(rs_cfg, 0, bag_filename)
rs.start_capture(True) # true: start recording with capture
for fid in range(150):
im_rgbd = rs.capture_frame(True, True) # wait for frames and align them
# process im_rgbd.depth and im_rgbd.color
rs.stop_capture()
For further information, check this tutorial.
CORE and 3D reconstruction
Open3D 0.12 brings exciting CORE upgrades, including a new Neighbor Search module. This module supports typical neighbor search methods, such as KNN, radius search, and hybrid search, on both CPUs and GPUs, under a common interface!
Furthermore, we have created a new version of the TSDF integration algorithm accelerated on GPU. This version is able to achieve an outstanding computational performance, requiring between 2 and 4 ms to integrate a pair of frames.
New rendering functionalities
We have done an important effort over the last months to put out a modern, real-time, rendering API. This effort is still ongoing, and we are committed to bring top-tier rendering capabilities with a strong emphasis in performance, versatility, ease of use, and beauty. As part of our commitment, in this release we have added relevant extensions to this API:
- Support for Screen-space reflections
- Full programmatic support for headless rendering in Filament (for real)
box = o3d.geometry.TriangleMesh.create_box(2, 2, 1)
render = rendering.OffscreenRenderer(640, 480)
render.scene.add_geometry("box", box, grey)
render.scene.camera.look_at([0, 0, 0], [0, 10, 0], [0, 0, 1])
img = render.render_to_image()- Support for arbitrary camera intrinsic matrices: A small step for the Camera class; a very anticipated step by the Open3D community
Camera::SetProjection(const Eigen::Matrix3d& intrinsics,
double near,
double far,
double width,
double height)- Support for text rendering: Render text in 3D
Label3D::Label3D(const Eigen::Vector3f& pos, const char* text)- Full control over the color grading pipeline
class ColorGradingParams {
public:
ColorGradingParams(Quality q, ToneMapping algorithm);
void SetTemperature(float temperature);
void SetTint(float tint);
void SetContrast(float contrast);
void SetVibrance(float vibrance);
void SetSaturation(float saturation);
void SetChannelMixer(const Eigen::Vector3f& red,
const Eigen::Vector3f& green,
const Eigen::Vector3f& blue);
void SetShadowMidtoneHighlights(const Eigen::Vector4f& shadows,
const Eigen::Vector4f& midtones,
const Eigen::Vector4f& highlights,
const Eigen::Vector4f& ranges);
void SetSlopeOffsetPower(const Eigen::Vector3f& slope,
const Eigen::Vector3f& offset,
const Eigen::Vector3f& power);
void SetCurves(const Eigen::Vector3f& shadow_gamma,
const Eigen::Vector3f& midpoint,
const Eigen::Vector3f& highlight_scale);
}
Control shadow behaviors and post-processing effects:
class View
{
void SetPostProcessing(bool enabled);
void SetAmbientOcclusion(bool enabled, bool ssct_enabled);
void SetAntiAliasing(bool enabled, bool temporal);
void SetShadowing(bool enabled, ShadowType type);
}
Visualization and GUI: O3DViewer (beta)
The visualization module has been extended, using the new rendering capabilities and the GUI API, to create a unified visualizer displaying all the features contained in previous Open3D visualizers, e.g., camera animation, data selection, support for callbacks, and multiple shading modes.
This new visualizer, codename O3DViewer, will be the official visualization tool in Open3D starting in Open3D 0.14. At that time, previous visualizers will be deprecated.
We hope you find Open3D 0.12.0 exciting and useful. Happy coding!
Remember that you can reach out with questions, requests, or feedback through the following channels:
Find the full change log keep reading.
The Open3D team
Deprecating
- All visualization tools, such as draw_geometries will be deprecated in Open3D 0.14 in favor of the new O3DViewer.
- The Open3D 0.12 packages will be the last to support TensorFlow 2.3 and PyTorch 1.6 with CUDA 10.1. We will update to newer versions of these toolkits in the Open3D 0.13 packages. Note that this does not affect binary compatibility if Open3D is built from source.
Changes to Open3D-ML
- Fixed infinte dataset iteration (#184)
- Update readme and config files for parislille3d; align points for parislille3d (#180)
- Disable data augmentation while testing. (#181)
- Fixed absolute path bug (#182)
- Add Data Augmentation (#178)
- PointPillars bug fixes (#179)
- Filter KITTI point cloud (#177)
- Point pillars metrics (#172)
- Add wide lines (#176)
- Fix for changes in t::geometry (#173)
- Added ShapeNet dataset (#157)
- Added weight initialization (#174)
- Update model zoo (#175)
- New validation for torch (#169)
- Point pillars train tf (#171)
- Point pillars train (#170)
- Readme randlanet semantic3d (#167)
- Fixes for changes from TensorList to Tesor for t.geometry objects (#161)
- Add Tensorflow model and inference pipeline. (#159)
- Fix broken link to torch RandLA-Net Toronto 3d model (#163)
- Add comments for visualize predictions (#151)
- PointPillars inference pipeline (#153)
- Link to kpconv parislille3d models in readme (#160)
- Added Agroverse 3D Dataset (#155)
- Change Bounding box class. (#149)
- Add bounding boxes to visualizer (#140)
- Add Download scripts (#145)
- Add Lyft Dataset (#138)
- Create sampler class for sampling point cloud idx and points idx (#135)
- Add NuScenes dataset (#137)
- Add Waymo Dataset (#136)
- Add Kitti object detection dataset (#128)
Changes to Open3D
- Added python bindings, improved descriptiveness of repr for bounding boxes, and added BG options to draw() (#2785)
- Fix aspect ratios of background images in certain situations (#2783)
- Add 3D Labels to SceneWidget (#2781)
- ICP Point to Plane and Point to Point Registration and Example (#2743)
- Make sure projection flags are always properly initialized (#2782)
- RealSense sensor configuration, live capture and recording (#2748)
- CUDA/CPU compatible TSDF integration (#2710)
- Added animation callbacks (also some pybinds for rendering::Scene) (#2769)
- Add ability to add mouse and key callbacks to SceneWidget in Python (#2776)
- Expose Additional Filament features via View class (#2755)
- Add support for SSR (via refraction material) (#2768)
- Added compatible draw() function for external visualizer (#2772)
- Add light entities to filament scene (#2771)
- Fix memcpy issue for nms cuda (#2770)
- Fix background not being drawn in some camera orientations (#2764)
- Add shader for unlit with transparency (#2760)
- RealSense SDK v2 (#2646)
- Better picking, fixes crash removing last selection set (#2746)
- Support multiple non-sun directional lights (#2759)
- Add rpc interface to the new draw function (#2734)
- Fix out of bounds memory access in ragged_to_dense op (#2751)
- Enable HybridSearch on GPU (#2752)
- Fix undefined reference in Debug build with gcc (#2744)
- Expose Filament's Color Grading Controls (#2737)
- O3DVisualizer can accept background images (#2735)
- Fix build for BUILD_GUI=OFF SHARED_LIBRARIES=ON (#2745)
- IoU op for BEV and 3D (#2742)
- Add replacement for draw_geometries() (#2585)
- Included missing header files (#2137)
- Tensor transform pointcloud (#2704)
- Added ability to set background image (#2701)
- Modernize Hashmap (#2676)
- Moved Line3D and Segment3D Transform implementation to cpp file (#2690)
- Docs: Minor tweaks (README, bib ref, notebook instructions, some directives) (#2534)
- Alt-Enter to toggle Fullscreen mode (#1939)
- Faster mesh and pcd conversion (#2719)
- Remove 16.04 kinect workaround (#2718)
- Use assimp to load STL, enable ASCII read (#2506)
- Fixes a second button with same name not working (#2696)
- Add shader and supporting code for variable line width line sets (#2678)
- Remove filament-only arm64 build (#2708)
- Connect FixedRadiusSearch to NearestNeighborSearch (#2680)
- Geometry stores Tensor instead of TensorList (#2692)
- Update pybind11 and tinyobjloader version (#2703)
- FixedRadiusSearch in core::nns (#2582)
- Simplify macos dependencies (#2666)
- Fix windows warning (as error) on master (#2691)
- Fix warnings and link problems on windows for rpc interface (#2661)
- Integration of Custom Filament Headless Backend (#2572)
- RPC receiver for visualizer (#2182)
- Added support for international characters in the GUI (#2655)
- Cmake-3.17 required (#2665)
- Build wheel with Kinect support on CI (#2648)
- Write point cloud with custom attributes (#2594)
- Convert half_edge_mesh.py to jupyter notebook (#2387)
- Integrate Faiss to NearestNeighborSearch (#2400)
- Update RANSAC (#2636)
- Fix for implicit capture of this in cpp20 (#2638)
- Nms op for pytorch and tensorflow (#2615)
- Fixes for LGTM warnings (#2629)
- Substitute set-env with environment files (#2631)
- Modify shader to work with Filament's new inverse z buffer (#2626)
- Voxelize op for torch and tensorflow (#2607)
- Fix crash on exit if Python variable is assigned a Open3DScene (#2618)
- Fixed shape check for faces and lines in SetMeshData (#2623)
- Add function to Open3DScene to allow modifying one geometry's material, and to see if the scene has a geometry (#2612)
- Create axis geometry on-demand when shown rather than with each geometry change (#2617)
- Allow arbitrary pinhole camera (#2564)
- Automatically cast int extents to the right type (#2590)
- Log non-fatal errors in Visualizer as warnings instead of errors (#2579)
- Update to match material handling of old OBJ file loader (#2576)
- Add support for 4 channel (RGBA) color images to CreateFromRGBDImage (#2577)
- Add pybind for triangle_material_ids (#2573)
- Fix reading float from PCD, instead of float_t (#2563)
- Fix string to boolean (#2574)
- Fix reconstruction system (#2567)
- Fixed incomplete function call. (#2562)
- Change variable TriangleMeshSimplification.cpp (#2542)
- Minor documentation fix (#2554)
- Faiss Build Test (#2382)
- Tensor ply read (#2377)
- Give sheen a default value (i.e., don't leave it unitialized) (#2532)
- Replace the non utf-8 char “” with "" (#2535)
- Simplified install instructions for Open3D-ML (#2513)
- Fix Open3DScene with no background color set not clearing on draw on macOS (#2530)
- Added example to add geometry to a scene using the new GUI. (#2515)
- Fixes bug for preloading libc++ and libc++abi in Python (#2527)
- Allow Scene::AddGeometry to accept bounding boxes (#2520)
- Add Filament 1.9.5 (#2517)
- Remove accidental O(n^2) adding of geometries in draw_geometries resulting from a bad merge conflict resolution in d5b95454b (#2523)
- Add robust kernel colorized icp (#2497) (Nacho)
- Add miniconda build (#2477)
- Add dummy texture coordinates for triangle meshes with no texture coordinates to avoid error messages. (#2514)
- Fix z-buffering problem with labels/gradients in ml3d visualizer (#2512)
- Better error msg for cuda mismatch (#2503)
- Fix ambiguous transform (#2491)
- Workflow dispatch fixes (#2500)
- Test import of torch and tensorflow (#2496)
- Fix deadlock with ml3d visualizer on macOS (#2502)
- Removed shared_ptr as the pybind holder of GUI objects to prevent crashes caused by Python keeping the object alive after Filament resources are destroyed on the Python side. (#2501)
- Change pybind's run() to not cleanup at the end. (#2499)
- Added code to fix 'from' import statements in the ml namespaces (#2492)
- Add MK docs pointer in readme and rst (#2486)
- Rework Python API for creating windows to avoid crash on exit in some simple situations. (#2485)
- Documentation build fix (Jupyter + Python visualization.{gui,rendering}) (#2484)