Estimate 3D Line from Image projections with known Camera Pose and Calibration

Question

I know the principle of triangulation for 3D Point estimation from images. However, how would you solve the following problem, I have images from a Line in 3D space with known Camera position and also known calibration. But since I don't know how much/which segment of the Line is seen in eag image, I am not sure how to form an equation for the Line estimation. See image (I have more than 2 images available, in all images most part of the Line visible should be the same, but not exact the same):

I am thinking of spanning a plane from the Camera through the line in the image and intersecting all Planes spanned from each perspective to get an estimate of the Line? However, I don't really know if that's possible or how I could do this.

Thanks for any help. Cheers

Answer 1

Afraid other answers and comments have it backwards, pun intended.

Backprojecting ("triangulating") image lines into 3D space and then trying to fit them together with some ad-hoc heuristics may be good for an initial approximation.

However, to refine this approximation, you should then assume that a 3D line exists with unknown parameters (a point and a unit vector), plus additional scalar parameters identifying the initial and final points along the line of the segments you observe. Using the projection equations, you then set up an optimization problem whose goal is to find the set of parameters that minimize the projection errors of the 3D line with those parameters onto the images. This is essentially bundle adjustment, but expressed in the language of your problem, and in fact you can use any good software package for bundle adjustment (hint: Ceres) to solve it. The initial approximation computed with some ad-hoc heuristic will be used as the starting point of the bundle adjustment.

Answer 2

Plane can be defined with one point and direction. If bot cameras are calibrated and have known position then ... Make two planes, first from first camera, using eye and projected line second plane at the same way. Line from screen you can easily convert to line in space because you have camera position. What is left to intersect those two planes.