Camera calibration for Structure from Motion with OpenCV (Python)
<p>I want to calibrate a car video recorder and use it for 3D reconstruction with Structure from Motion (SfM). The original size of the pictures I have took with this camera is 1920x1080. Basically, I have been using the source code from the <a href="http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html" rel="nofollow">OpenCV tutorial</a> for the calibration. </p>

<p>But there are some problems and I would really appreciate any help. </p>

<p>So, as usual (at least in the above source code), here is the pipeline:</p>

<ol>
<li>Find the chessboard corner with <code>findChessboardCorners</code></li>
<li>Get its subpixel value with <code>cornerSubPix</code></li>
<li>Draw it for visualisation with <code>drawhessboardCorners</code></li>
<li>Then, we calibrate the camera with a call to <code>calibrateCamera</code> </li>
<li>Call the <code>getOptimalNewCameraMatrix</code> and the <code>undistort</code> function to undistort the image</li>
</ol>

<p>In my case, since the image is too big (1920x1080), I have resized it to 640x320 (during SfM, I will also use this size of image, so, I don't think it would be any problem). And also, I have used a 9x6 chessboard corners for the calibration.</p>

<p>Here, the problem arose. After a call to the <code>getOptimalNewCameraMatrix</code>, the distortion come out totally wrong. Even the returned ROI is <code>[0,0,0,0]</code>. Below is the original image and its undistorted version: </p>

<p><a href="http://i.stack.imgur.com/elJx1.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/elJx1.jpg" alt="Original image"></a>
<a href="http://i.stack.imgur.com/nEbeN.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/nEbeN.jpg" alt="Undistorted image"></a>
You can see the image in the undistorted image is at the bottom left.</p>

<p>But, if I didn't call the <code>getOptimalNewCameraMatrix</code> and just straight <code>undistort</code> it, I got a quite good image.
<a href="http://i.stack.imgur.com/L08QS.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/L08QS.jpg" alt="Undistorted image"></a></p>

<p>So, I have three questions. </p>

<ol>
<li><p>Why is this? I have tried with another dataset taken with the same camera, and also with my iPhone 6 Plus, but the results are same as above. </p></li>
<li><p>Another question is, what is the <code>getOptimalNewCameraMatrix</code> does? I have read the documentations several times but still cannot understand it. From what I have observed, if I didn't call the <code>getOptimalNewCameraMatrix</code>, my image will retain its size but it would be zoomed and blurred. Can anybody explain this function in more detail for me?</p></li>
<li><p>For SfM, I guess the call to <code>getOptimalNewCameraMatrix</code> is important? Because if not, the undistorted image would be zoomed and blurred, making the keypoint detection harder (in my case, I will be using the optical flow)?</p></li>
</ol>

<p>I have tested the code with the opencv sample pictures and the results are just fine.</p>

<p>Below is my source code:</p>

<pre><code>from sys import argv
import numpy as np
import imutils  # To use the imutils.resize function. 
                       # Resizing while preserving the image's ratio.
                       # In this case, resizing 1920x1080 into 640x360.
import cv2
import glob

# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((9*6,3), np.float32)
objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

images = glob.glob(argv[1] + '*.jpg')
width = 640

for fname in images:
    img = cv2.imread(fname)
    if width:
        img = imutils.resize(img, width=width)

    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    # Find the chess board corners
    ret, corners = cv2.findChessboardCorners(gray, (9,6),None)

    # If found, add object points, image points (after refining them)
    if ret == True:
        objpoints.append(objp)

        corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
        imgpoints.append(corners2)

        # Draw and display the corners
        img = cv2.drawChessboardCorners(img, (9,6), corners2,ret)
        cv2.imshow('img',img)
        cv2.waitKey(500)

cv2.destroyAllWindows()
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)

for fname in images:
    img = cv2.imread(fname)
    if width:
        img = imutils.resize(img, width=width)

    h,  w = img.shape[:2]
    newcameramtx, roi=cv2.getOptimalNewCameraMatrix(mtx,dist,(w,h),1,(w,h))

    # undistort
    dst = cv2.undistort(img, mtx, dist, None, newcameramtx)

    # crop the image
    x,y,w,h = roi
    dst = dst[y:y+h, x:x+w]
    cv2.imshow("undistorted", dst)
    cv2.waitKey(500)

mean_error = 0
for i in xrange(len(objpoints)):
    imgpoints2, _ = cv2.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
    error = cv2.norm(imgpoints[i],imgpoints2, cv2.NORM_L2)/len(imgpoints2)
    mean_error += error

print "total error: ", mean_error/len(objpoints)
</code></pre>

<p>Already ask someone in answers.opencv.org and he tried my code and my dataset with success. I wonder what is actually wrong.</p>

<p><strong>Question #2:</strong></p>

<p>With <code>cv::getOptimalNewCameraMatrix(...)</code> you can compute a new camera matrix according to the free scaling parameter <code>alpha</code>.</p>

<p>If <code>alpha</code> is set to <code>1</code> then all the source image pixels are retained in the undistorted image that is you'll see black and curved border along the undistorted image (like a pincushion). This scenario is unlucky for several computer vision algorithms, because new edges are appeared on the undistorted image for example. </p>

<p>By default <code>cv::undistort(...)</code> regulates the subset of the source image that will be visible in the corrected image and that's why only the sensible pixels are shown on that - no pincushion around the corrected image but data loss.</p>

<p>Anyway, you are allowed to control the subset of the source image that will be visible in the corrected image:</p>

<pre><code>cv::Mat image, cameraMatrix, distCoeffs;
// ...

cv::Mat newCameraMatrix = cv::getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, image.size(), 1.0);

cv::Mat correctedImage;
cv::undistort(image, correctedImage, cameraMatrix, distCoeffs, newCameraMatrix);
</code></pre>

<p><strong>Question #1:</strong></p>

<p>It is just my feeling, but you should also take care, if you resize your image after the calibration then the camera matrix must be also "scaled" as well, for example:</p>

<pre><code>cv::Mat cameraMatrix;
cv::Size calibSize; // Image during the calibration, e.g. 1920x1080
cv::Size imageSize; // Your current image size, e.g. 640x320
// ...

cv::Matx31d t(0.0, 0.0, 1.0);
t(0) = (double)imageSize.width / (double)calibSize.width;
t(1) = (double)imageSize.height / (double)calibSize.height;

cameraMatrixScaled = cv::Mat::diag(cv::Mat(t)) * cameraMatrix;
</code></pre>

<p>This must be done only for the camera matrix, because the distortion coefficients do not depend on the resolution.</p>

<p><strong>Question #3:</strong></p>

<p>Whatever I think <code>cv::getOptimalNewCameraMatrix(...)</code> is not important in your case, the undistorted image can be zoomed and blurred because you remove the effect of a non-linear transformation. If I were you then I would try the optical flow without calling <code>cv::undistort(...)</code>. I think that even a distorted image can contain a lot of good features for tracking.</p>

python opencv camera-calibration 3d-reconstruction structure-from-motion