3D Weld Seam Detection

Built a dual-pipeline computer vision system for automatic weld seam detection using Intel RealSense depth cameras — a 2D pipeline (OpenCV with adaptive Canny + depth gradients + Hough transforms) and a 3D pipeline (Open3D with RANSAC plane removal + DBSCAN clustering + SVD/PCA line fitting). The novel groove detection algorithm is an original approach to finding weld seams in industrial point clouds. This is the same class of perception work done at Path Robotics, Novarc Technologies, and Fanuc iRVision.

Built the OpenCV 2D+depth pipeline (WeldSeamDetector.process_frame()): adaptive Canny edge detection with thresholds computed from median image intensity (lower = 0.66 × median, upper = 1.33 × median — eliminates manual threshold tuning for different workpiece materials and lighting), Sobel operator depth gradient computation with adaptive threshold at 2.5 × median_gradient to detect step changes in depth where two metal plates meet, workspace depth masking filtering to the 10cm–150cm RealSense working range, and multi-directional morphological edge connection using horizontal, vertical, and 45° diagonal closing kernels to bridge gaps in detected edges. Hough line detection runs on the combined color + depth edge map, then rs2_deproject_pixel_to_point() back-projects each detected line from 2D pixels into 3D robot coordinates using the camera's intrinsic matrix.

Developed the Open3D 3D point cloud pipeline (Open3DWeldSeamDetector.process_point_cloud()): RGBD-to-point-cloud conversion using pinhole camera intrinsics, voxel downsampling at 2mm resolution, then RANSAC dominant plane removal (1cm distance tolerance, 1000 iterations) to segment away the table/fixture surface and isolate the workpiece geometry. A novel groove detection algorithm identifies the deepest 10% of points (most likely located in weld grooves), clusters them using DBSCAN (eps=0.01m, min_samples=10) to group spatially connected groove points, and validates each cluster's linearity using SVD decomposition — only clusters with linearity > 0.65 pass through as candidate weld seams.

PCA-based 3D line fitting extracts the weld seam path from validated clusters: SVD decomposition on mean-centered cluster points yields a straightness metric S[0]/sum(S) measuring how well the points fit a line. The first right singular vector gives the 3D line direction, and the centroid gives a point on the line. A second DBSCAN pass (eps=0.03m, min_points=2) groups edge points into distinct line segments for workpieces with multiple seams. The dual-pipeline approach provides speed vs. accuracy tradeoffs: 2D is fast and works well for simple butt and lap joints; 3D handles complex groove geometries, V-joints, and parts with significant depth variation.

Integrated with ROS as a full perception node: publishes annotated images, raw edge detection results, visualization_msgs/MarkerArray for 3D seam visualization in RViz, and sensor_msgs/PointCloud2 for downstream processing by the trajectory planner. Supports both snapshot mode (triggered via /seam_detection/trigger topic) and live mode with configurable frame rate. Applied Intel RealSense post-processing filters (decimation, spatial smoothing, temporal averaging, and hole-filling) to reduce depth noise on reflective and curved metal surfaces — a key challenge in industrial welding environments where shiny metal creates noisy depth maps.