Camera Cailbration

 I have been looking for a Java camera calibration code for a long time.  There is a C++ example of camera calibration at the official OpenCV documentation site.  Now that Java OpenCV is available, I decided that I will make a Java version.

The C++ example is quite complicated, in that it can use multiple calibration patterns, and can using a list of image as well as live video. I decided that this example only uses a folder of several chessboard photos as input.

import java.io.File;
import java.util.ArrayList;
import org.opencv.calib3d.Calib3d;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.MatOfPoint3f;
import org.opencv.core.Point3;
import org.opencv.core.Size;
import org.opencv.core.TermCriteria;
import org.opencv.highgui.Highgui;
import org.opencv.imgproc.Imgproc;

public class CalibChessBoard {
    int flagsCorner = Calib3d.CALIB_CB_ADAPTIVE_THRESH
            | Calib3d.CALIB_CB_FAST_CHECK 
            | Calib3d.CALIB_CB_NORMALIZE_IMAGE;
    int flagsCalib = Calib3d.CALIB_ZERO_TANGENT_DIST
            | Calib3d.CALIB_FIX_PRINCIPAL_POINT 
            | Calib3d.CALIB_FIX_K4
            | Calib3d.CALIB_FIX_K5;
    TermCriteria criteria = new TermCriteria(TermCriteria.EPS
            + TermCriteria.MAX_ITER, 40, 0.001);
    Size winSize = new Size(5, 5), zoneSize = new Size(-1, -1);
    Size patternSize;
    ArrayList objectPoints, imagePoints = new ArrayList();
    ArrayList vCorners;
    ArrayList vImg;
    Mat cameraMatrix = Mat.eye(3, 3, CvType.CV_64F);
    Mat distCoeffs = Mat.zeros(8, 1, CvType.CV_64F);
    ArrayList rvecs = new ArrayList();
    ArrayList tvecs = new ArrayList();

    CalibChessBoard() {
    }

    CalibChessBoard(Size patternSize) {
        this.patternSize = patternSize;
    }

    boolean getCorners(Mat gray, MatOfPoint2f corners) {
        if (!Calib3d.findChessboardCorners(gray, patternSize,
                corners, flagsCorner))
            return false;
        Imgproc.cornerSubPix(gray, corners, winSize, zoneSize,
                                   criteria);
        return true;
    }

    MatOfPoint3f getCorner3f() {
        MatOfPoint3f corners3f = new MatOfPoint3f();
        double squareSize = 50;
        Point3[] vp = new Point3[(int) (patternSize.height * 
                                            patternSize.width)];
        int cnt = 0;
        for (int i = 0; i < patternSize.height; ++i)
            for (int j = 0; j < patternSize.width; ++j, cnt++)
                vp[cnt] = new Point3(j * squareSize, 
                                     i * squareSize, 0.0d);
        corners3f.fromArray(vp);
        return corners3f;
    }

    public static void main(String[] args) {
        test0();
    }

    static void test0() {
        CalibChessBoard cb = new CalibChessBoard(new Size(8, 6));
        cb.getAllCornors("/the/photo/folder");
        cb.calibrate();
    }

    void calibrate() {
        double errReproj = Calib3d.calibrateCamera(objectPoints, 
                imagePoints,vImg.get(0).size(), cameraMatrix, 
                distCoeffs, rvecs, tvecs,flagsCalib);
        System.out.println("done, \nerrReproj = " + errReproj);
        System.out.println("cameraMatrix = \n" + cameraMatrix.dump());
        System.out.println("distCoeffs = \n" + distCoeffs.dump());
    }

    void getAllCornors(String path) {
        vImg = new ArrayList();
        objectPoints = new ArrayList();
        imagePoints = new ArrayList();
        MatOfPoint3f corners3f = getCorner3f();
        for (File f : new File(path).listFiles()) {
            Mat mat = Highgui.imread(f.getPath(), 
                           Highgui.CV_LOAD_IMAGE_COLOR);
            if (mat == null || mat.channels() != 3)
                continue;
            System.out.println("fn = " + f.getPath());
            System.out.println("mat.channels() = " + mat.channels() 
                    + ", " + mat.cols() + ", " + mat.rows());
            Mat gray = new Mat();
            Imgproc.cvtColor(mat, gray, Imgproc.COLOR_BGR2GRAY);
            MatOfPoint2f corners = new MatOfPoint2f();
            if (!getCorners(gray, corners))
                continue;
            objectPoints.add(corners3f);
            imagePoints.add(corners);
            vImg.add(mat);
        }
    }

    static {
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }
}

Here are the list of images that give the following results:

errReproj = 0.25108
cameraMatrix =
[614.561, 0, 319.5;
  0, 615.445, 239.5;
  0, 0, 1]
distCoeffs = [0.155040; 0.075046; 0; 0; -2.318117]

Java OpenCV Webcam

Camera is a very important piece of equipment in computer vision. My first tutorial will be a webcam program based on OpenCV. 1. We need an efficient piece of code that convert OpenCV image, a Mat, to a BufferedImage.

import java.awt.image.BufferedImage;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.highgui.Highgui;
import org.opencv.imgproc.Imgproc;

public class Mat2Image {
    Mat mat = new Mat();
    BufferedImage img;
    byte[] dat;
    public Mat2Image() {
    }
    public Mat2Image(Mat mat) {
        getSpace(mat);
    }
    public void getSpace(Mat mat) {
        this.mat = mat;
        int w = mat.cols(), h = mat.rows();
        if (dat == null || dat.length != w * h * 3)
            dat = new byte[w * h * 3];
        if (img == null || img.getWidth() != w || img.getHeight() != h
            || img.getType() != BufferedImage.TYPE_3BYTE_BGR)
                img = new BufferedImage(w, h, 
                            BufferedImage.TYPE_3BYTE_BGR);
        }
        BufferedImage getImage(Mat mat){
            getSpace(mat);
            mat.get(0, 0, dat);
            img.getRaster().setDataElements(0, 0, 
                               mat.cols(), mat.rows(), dat);
        return img;
    }
    static{
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }
}

2. Here is a simple class that captures video:

import java.awt.image.BufferedImage;
import org.opencv.core.Core;
import org.opencv.highgui.VideoCapture;

public class VideoCap {
    static{
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }

    VideoCapture cap;
    Mat2Image mat2Img = new Mat2Image();

    VideoCap(){
        cap = new VideoCapture();
        cap.open(0);
    } 
 
    BufferedImage getOneFrame() {
        cap.read(mat2Img.mat);
        return mat2Img.getImage(mat2Img.mat);
    }
}

3. Now we need a JFrame to display the video:

package org.ski;
import java.awt.EventQueue;
import java.awt.Graphics;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.border.EmptyBorder;

public class MyFrame extends JFrame {
    private JPanel contentPane;

  /**
  * Launch the application.
  */
    public static void main(String[] args) {
        EventQueue.invokeLater(new Runnable() {
            public void run() {
                try {
                    MyFrame frame = new MyFrame();
                    frame.setVisible(true);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });
    }

  /**
  * Create the frame.
  */
    public MyFrame() {
        setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        setBounds(100, 100, 650, 490);
        contentPane = new JPanel();
        contentPane.setBorder(new EmptyBorder(5, 5, 5, 5));
        setContentPane(contentPane);
        contentPane.setLayout(null);
  
        new MyThread().start();
    }
 
    VideoCap videoCap = new VideoCap();
 
    public void paint(Graphics g){
        g = contentPane.getGraphics();
        g.drawImage(videoCap.getOneFrame(), 0, 0, this);
    }
 
    class MyThread extends Thread{
        @Override
        public void run() {
            for (;;){
                repaint();
                try { Thread.sleep(30);
                } catch (InterruptedException e) {    }
            }  
        } 
    }
}

Computer Vision and Java

In the field of computer vision, there are several choices in programming languages.  Matlab is the first choice in terms of ease of use.  Python is chosen for similar reasons.  C++ is used for it's pure speed.  Java just falls somewhere in the middle, and mostly neglected.  But Java has been my first choice for computer vision research/development for several years.

I choose Java for several reasons.  First, it's very easy to debug a java program.  Every time a program crashes, it comes right to the line of the source code, speed up debugging greatly.  It comes with full featured graphics tools, unlike C++, where you need another tool for image loading/writing.  Netbeans makes my quick and dirty prototyping much easier.

I had developed some tools for image processing, computer vision algorithm developing in Java for several year, meanwhile OpenCV are getting more and more versatile.  Finally, I decided that I couldn't limit my programming language in Java any longer.  A little over a year ago, I moved my main algorithm development to C++, based on OpenCV.  It has been rewarding.  Still, I miss my days when I can mainly code in Java.

Then desktop Java interface for OpenCV becomes available.  Now I can combine ease of coding in Java and the capabilities and speed of OpenCV.  Life just couldn't get any better.

It's still in the early stage of OpenCV Java interface.  There are not a ton of examples/tutorial available.  Since I have been working with JNI for several year, I'd like to translate some OpenCV C++ examples in to running Java example for some time to come.