Point cloud and Lidar data - challenges and solutions

Lidar (Light Detection and Ranging) is a fairly new technology, much less common than cameras or radar. Yet the technology comes with multiple advantages that are unseen in other types of sensors.

The first Lidar-like system was developed in 1961, shortly after the Laser had been invented. The device was based on the same principle as radar – sending a signal and then measuring the time it takes to return. The technology has come much further, but the key idea behind the Lidar system has stayed the same.

When talking about Lidar, there are a few main questions to answer first:

According to the Markets and Markets report, the Lidar market is predicted to reach $2.8 billion by 2025 from $1.1 billion in 2020. The autonomous car market is considered one of the key drivers of this growth, with Lidar being one of the key sensors for autonomous cars.

What is Lidar

In the simplest definition, Lidar is a radar-like device that uses a laser pulse instead of radio waves to measure the distance between an object and the sender. A key feature of Lidar is that the sensor delivers highly accurate data, even from a larger distance. While the key information delivered by a radar device is that there is an object in a particular position and it is moving at a particular velocity, Lidar delivers much more precise information.

Lidar provides an exact position of a particular point in space. Having enough points for analysis, it is possible to deliver precise information about the shape of an entity – a thing that the radar is incapable of.

This idea of getting information about a particular entity from the shape of points taken from Lidar leads straight into the idea of point cloud data collection.

What is point cloud data?

Point cloud data can be seen as a 3D version of a “connect the dots” puzzle, though without numbers to guide the user to the answer. With the light speed of the laser delivering a single point in 3D space, a Laser scanner is able to deliver a swarm of dots in little to no time at all. Modern Lidar sensors use several beams at once, further boosting the density of dots gathered.

Nevertheless, data gathered this way can be compelling to the human eye, enriched with a whole life of experiences. Through their experience, humans are able to have a fairly easy time in generalizing an image of a dot-composed car into a car, or a tree composed of dots into an actual understanding of a tree.

But what comes with little to no trouble to humans poses a significant challenge for computers, effectively limiting usage of point cloud data. Or, at least, leaving engineers with a significant challenge to solve.

Challenges with Lidar Point Cloud processing (and how we solve them at Tooploox)

With point cloud technology, there are multiple benefits, yet they come with challenges that are hard to overcome.

Massive Data

Raw lidar data is a massive file with point sets that contain information about objects in a 3D space. Thus, storing this data and transferring it, whether to the data warehouse or a processing point, is troublesome.

How We’ve Solved It:

By using neural networks that encode the 3D point cloud into a multidimensional vector, Tooploox specialists managed to significantly reduce the size of the file required to be transferred. In practice, using compact binary descriptors for 3D point clouds means that we are able to code the entire point cloud into an astonishing 128 bits. The research is available on Arxiv:

Adversarial Autoencoders for Compact Representations of 3D Point Clouds

Processability

Massive data is also hard to process – immense computing power is required to analyze or process the file or a group of files. When combined with the large storage required to work with this type of data, the process gets costly. And rising costs reduce the profitability of point cloud data usage.

How We’ve Solved It:

The multidimensional vector produced by the encoder neural network designed by Tooploox specialists is completely incomprehensible for humans, but it is entirely processable for machines. It can be compared to a situation where a computer is working on a compressed file.

So when using the Tooploox-designed technology, it is possible to process point cloud data on-site with significantly reduced computing power and storage. We are not only capable of encoding 3D shapes into compact representations, but also making further use of them in solving classification and retrieval tasks.

Multimodal Data Processing

The direct effect of the problems mentioned above. Multimodal data processing, or processing data of various sources and types, is at the foundation of all synergies and advantages of modern data processing. Combining multiple types of data is a challenge in itself. When it comes to working with vastly different types of data, the challenge grows even greater.

How We’ve Solved It:

The Tooploox team reduced the heavy point cloud data into a multidimensional vector which represents the object, as delivered by an encoder neural network. Although incomprehensible for humans, this type of data is fully actionable for machines. Thus, it can be combined with other types of data in an easier, more digestible way.

Lidar Point Cloud To 3D Model

This challenge leads us to the situation where one has point cloud data representing a particular object, but there is little to no usage for it – despite the accuracy and precision of the data, the format can be too exotic to make the information useful. For example, it is extremely challenging to deliver a 3D model of a chair (or any other object) using point cloud data alone. It usually comes down to the heavy manual work of a 3D artist.

How We’ve Solved It:

The Tooploox team delivered a neural network that transforms point clouds into a mesh using a 3D ball model and the point cloud itself. Thus, the amount of manual work required in building the 3D visualization of Lidar-scanned models was significantly reduced. The effect is in fact ready for use in 3D printing, for example. The team has tackled this problem in two research papers, available on Arxiv:

Also, the research team has delivered a neural network that transforms the point cloud data into primitives, which can be processed separately. This can come of use if there is some heavy or biomechanical engineering involved, for example in life sciences, where work on a particular element in a 3D model can be of great significance. The research is also available on Arxiv:

UCSG-NET – Unsupervised Discovering of Constructive Solid Geometry Tree

Noisy And Incomplete Data

Another challenge is an immanent element of point cloud data. In a perfect world, there would be no noise and only a swarm of dots perfectly representing the object. But in real-life use cases, there is noise in the data. When it comes to Lidar-generated point clouds, the noise can come from unexpected objects or unusual patterns and materials.

Also, depending on the use case, the data gathered by the Lidar sensor can be incomplete. This can be particularly seen in autonomous cars, where gathering the full shape of a car is nearly impossible and rarely necessary. The car sensor gathers points from a single side of a car, its back or front, regarding the relative position of the car on the road.

How We’ve Solved It:

The Tooploox team delivered a neural network able to deduce the full shape of an object from only a part of its shape. The neural network is highly accurate in determining if the swarm of dots is representing the rear of a vehicle, a tree trunk, or a child on a bicycle. The research paper can be found on Arxiv:

HyperPocket: Generative Point Cloud Completion

This feature can also be used when there is a need for a reconstruction of a complete scene from only partial data. For example, it can represent all furniture and devices in a room reconstructed from data gathered by a scanner that has remained stationary and which has gathered its data without changing the scan angle.

Summary

Lidar system is a powerful scanning tool that delivers data of immense precision. But working with this type of data is a challenging task that requires immense computing power and large-scale computer engineering.

Or with a sophisticated neural network, as it is done in Tooploox. If you wish to get more information on the topic, don’t hesitate to contact us now.