Perception: Cameras, Lidar and IMUs

The sensors compared

Sensor	Measures	Strengths	Limits
Wheel encoders	Wheel rotation	Cheap, precise in the short term (see Mobile Robot Kinematics and Odometry)	Drift with slip and wheel errors
IMU (gyroscope, accelerometer, magnetometer)	Rotation rate, acceleration, field direction	Fast (100–1000 Hz), works in the dark	Bias drift; the magnetometer is disturbed by motors and steel
Camera (mono)	Light: color and texture	Rich, cheap, high resolution	No direct depth; sensitive to lighting
Stereo / depth camera	Depth for each pixel	Dense depth and color	Short range; error grows with distance squared; fails in sunlight (active IR)
2D lidar	Range in a plane	Accurate, long range, works in the dark	One plane only; glass and black surfaces confuse it
3D lidar	A 3D point cloud	Long range, accurate 3D structure	Cost, power, data volume
Ultrasonic and ToF	Range along a beam	Cheap obstacle detection	Wide beams, soft surfaces, crosstalk
Radar	Range and speed	All weather, long range	Coarse angular resolution
GNSS (GPS)	Global position	Absolute position outdoors	Meters of error (centimeters with RTK); no use indoors
Force, torque and tactile	Contact forces	Safe interaction, assembly	Needs calibration and drift compensation (see Sensors and Field Devices)

The pinhole camera model

A camera maps a 3D point (X, Y, Z) in the camera frame to a pixel (u, v) by perspective projection:

u = f_x X ÷ Z + c_x    v = f_y Y ÷ Z + c_y    (f in pixels, (c_x, c_y) the image center)

• Exposure and motion blur: a robot moving at 1.0 m/s past an object 2.0 m away sweeps 667 pixels per second across the image: a 10 ms exposure blurs it by 6.7 pixels, a 1 ms exposure by 0.7. Use a short exposure (and enough light) on a moving robot; a global shutter avoids the skew that a rolling shutter gives to fast motion.
• Frame rate and latency: at 30 fps a frame takes 33 ms to capture before it is processed (see Robot Software Architecture for the latency budget).
• Lighting matters more than resolution: even, bright, diffuse light, and no direct sun in the lens.

Camera calibration and distortion

A real lens bends straight lines (distortion), and its focal length and center are not exactly as specified. Calibration finds the intrinsic matrix K (f_x, f_y, c_x, c_y) and the distortion coefficients from several photographs of a checkerboard or circle grid, with OpenCV or the ROS camera_calibration package. The result is saved with the camera and used to undistort every image.

import cv2, numpy as np, glob
pattern = (9, 6)                                              # inner corners of the checkerboard
objp = np.zeros((pattern[0] * pattern[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern[0], 0:pattern[1]].T.reshape(-1, 2) * 0.025     # square size 25 mm
obj, img = [], []
for name in glob.glob("calib/*.png"):
    gray = cv2.imread(name, cv2.IMREAD_GRAYSCALE)
    ok, corners = cv2.findChessboardCorners(gray, pattern)
    if ok:
        obj.append(objp); img.append(cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1),
                   (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)))
rms, K, dist, _, _ = cv2.calibrateCamera(obj, img, gray.shape[::-1], None, None)
print(rms)                                                    # reprojection error in pixels: under 0.5 is good

• Take 20 or more images with the board at many positions, angles and distances, filling the frame and the corners.
• The reprojection error (how far the projected board corners land from the detected ones) should be well under a pixel; 0.3 px is good.
• Recalibrate after any change of focus, zoom or lens; fix the focus mechanically.
• Extrinsic calibration finds where the camera sits on the robot (a transform for the URDF); see below.

Stereo and depth cameras

Two cameras a baseline B apart see a point at slightly different positions; the difference is the disparity d (pixels), and the depth follows from similar triangles: Z = f B ÷ d.

Depth sensor	How it works	Range	Notes
Passive stereo	Matches texture between two cameras	0.3–20 m, depending on baseline	Needs texture; works outdoors; computing for matching (often on the camera)
Active stereo (IR projector)	Adds a projected pattern to give texture	0.2–10 m	RealSense D400 series: works on blank walls; sunlight washes out the pattern outdoors
Time of flight (ToF)	Measures the delay of modulated infrared light	0.2–8 m	Small, dense depth; multipath and sunlight errors
Structured light	Projects a known pattern, triangulates	0.3–3 m	High precision at close range: scanning and picking

Lidar

A lidar measures range by the time of flight of a laser pulse, while a mirror or an array sweeps the beam. A 2D lidar (RPLidar, Hokuyo, Sick) scans a plane and is the standard sensor for indoor mobile robots; a 3D lidar (Ouster, Velodyne, Livox) gives a point cloud.

• Range and reflectivity: the maximum range is given for a white target; a dark one (10 % reflective) is seen at a fraction of it.
• Problem surfaces: glass and mirrors (the beam passes or reflects away), black matt, rain, fog and dust; sunlight at some wavelengths.
• Motion distortion: a scan takes 100 ms, during which the robot moves; at 1 m/s that is 10 cm: de-skew the scan using odometry.
• Mounting height and tilt: a 2D lidar sees only one plane; mount it where obstacles are, and add a second sensor for tables and overhangs.
• Eye safety: look for class 1 laser products; never look into an unknown beam.

IMUs: gyroscopes and accelerometers

• Calibrate at rest: average the gyroscope while the robot is still to find its bias; recalibrate with temperature.
• The magnetometer gives an absolute heading outdoors but is distorted by motors, batteries and steel: calibrate it in place and distrust it indoors.
• Mount rigidly and away from vibration, and note its axes in the URDF (REP 145 gives the IMU conventions).
• Sample fast and time-stamp precisely: an IMU at 200 Hz or more, with the stamp from the sample time, not the arrival time.

Fiducial markers

AprilTag and ArUco markers are printed black-and-white squares that a camera detects and identifies with a single, reliable pose estimate: ideal for docking stations, calibration targets, localization landmarks and ground-truth in experiments (apriltag_ros, OpenCV aruco).

Vision and point-cloud pipelines

import cv2, numpy as np
frame = cv2.imread("scene.png")
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, (35, 80, 60), (85, 255, 255))               # a colour threshold: green objects
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, np.ones((5, 5), np.uint8))     # remove noise
contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
best = max(contours, key=cv2.contourArea)
x, y, w, h = cv2.boundingRect(best)                                 # the target in pixels

Task	Typical method	Notes
Find a colored or high-contrast object	Color thresholds, edges, contours (OpenCV)	Fast and simple; depends on lighting
Find known objects	Neural detectors (YOLO family), run on a GPU or accelerator	Needs training data; see Machine Learning
Track motion	Optical flow, Kalman tracking	Needs a steady frame rate and time stamps
Estimate camera motion	Visual odometry and visual-inertial odometry (ORB-SLAM, VINS)	Features and texture needed; scale from stereo or IMU
Clean a point cloud	Voxel downsampling, ground-plane removal, outlier filters (PCL, Open3D)	Reduces the data by an order of magnitude
Match two scans	ICP (iterative closest point), NDT	Needs a decent starting guess
Segment and cluster objects	Euclidean clustering, RANSAC planes	Fast and robust for tabletop scenes

• Keep the pipeline in the sensor’s frame, with time stamps, and convert to the robot’s frame with tf2 (see ROS 2 in Practice).
• Test on recorded data: replay a bag so that a change is judged on the same images every time.
• Budget the compute: a detector that runs at 5 fps on a CPU is useless at 1 m/s (see Robot Software Architecture).
• Plan for failure: glare, an empty scene, a covered lens: report “no detection” and let the behavior layer decide.

Time synchronization and extrinsic calibration

• Time-stamp at the source, when the data were captured (the camera’s hardware trigger, the lidar’s own clock), and use one clock across the robot (see ROS 2 in Practice).
• Synchronize multiple sensors by a shared trigger or PTP/NTP; a 30 ms offset between a camera and a lidar on a robot turning at 1 rad/s is a 1.7° misalignment (a point 5 m away lands 15 cm out).
• Extrinsic calibration finds each sensor’s pose relative to the robot (or to each other): camera to lidar by a target seen by both, camera to gripper by hand-eye calibration, IMU to camera by motion (Kalibr). Put the result in the URDF.
• Recalibrate after any knock or remount: a few millimeters or a degree is enough to ruin a fused result.
• Use a known target (a checkerboard, a tag, a pole at a measured position) to verify the calibration at the end.

Choosing sensors

Robot task	Good starting sensors	Why
Indoor navigation	2D lidar + wheel encoders + IMU, plus a depth camera for low obstacles	Reliable mapping and localization; see Localization, Mapping and Navigation
Outdoor navigation	GNSS (RTK) + IMU + 3D lidar or stereo + wheel odometry	Absolute position and structure in sunlight
Picking objects	A depth camera or structured light + force-torque at the wrist	Object pose and contact
Docking	A camera with an AprilTag, plus an IR beacon or bump sensors	A precise, reliable target
Following a line or person	A camera and a short-range depth sensor	Simple tracking
Safety around people	Certified safety lidar or area scanner	A safety function must use a rated sensor (see Industrial Robotics)

Related

• Robotics Tools: the overview of this guide.
• Localization, Mapping and Navigation: using these sensors for SLAM, localization and planning.
• Mobile Robot Kinematics and Odometry: odometry and gyroscope fusion.
• Sensors and Field Devices: encoders, proximity, force and wiring.
• Machine Learning: learned detection and depth.

FAQ

Lidar or camera for a mobile robot?

A 2D lidar is the most reliable for mapping and obstacle avoidance indoors; a camera adds color, objects and low obstacles. Many robots use both, with the lidar for navigation and the camera for recognition.

How accurate is stereo depth?

About 1 % of the distance at close range, getting worse with the square of the distance: centimeters at 2 m, decimeters at 10 m for a small baseline. A wider baseline improves it.

Can I get position from an accelerometer?

Not for more than a few seconds: double integration turns a tiny bias into meters of error. Use it for orientation (gravity) and short-term motion, and fuse it with other sensors.

Why does my depth camera fail outdoors?

Sunlight overwhelms the infrared pattern or time-of-flight signal. Use passive stereo, a lidar or a camera with a daylight filter, and shade the sensor.

References

• Hartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision; Szeliski, R., Computer Vision: Algorithms and Applications (free online).
• OpenCV documentation (camera calibration, ArUco); the AprilTag papers (Olson 2011; Wang and Olson 2016).
• Woodman, O. J., “An introduction to inertial navigation”, University of Cambridge Technical Report, 2007; sensor makers’ datasheets (Intel RealSense, Ouster, Hokuyo, Bosch, InvenSense).

Last reviewed: October 2026. Sensor numbers are typical values for the parts described: check the datasheet.