Android NDK C++ Code Error Checking Trouble with Eclipse

Eclipse for Android NDK is mostly a user friendly environment.  But sometimes I encounter error checkng problem with c++ code.  While compile/build goes smoothly, every time I open a c++ file, many syntax error show up, and I got 100+ error messages under problem tab.  Under this circumstance, I cannot build the project before the error-flagging file is closed, and the error messages are manually deleted.  The build process goes smoothly: there is no error to begin with.

This brings a lot of inconvenience to coding.  On the other hand, some of my projects do not have this problem.  After some experimenting, I finally figure out why:  the include path list for syntax is not updated after first build.  One way to solve this problem is get Android.mk and Application.mk ready before the first build.  Here are my example, OpenCV is used in this project:


Android.mk:

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
OPENCV_SDK_JNI = C:\\OpenCV-2.4.10-android-sdk\\sdk\\native\\jni
include $(OPENCV_SDK_JNI)/OpenCV.mk
LOCAL_MODULE    := hello
LOCAL_SRC_FILES := hello.cpp
LOCAL_C_INCLUDES += $(OPENCV_SDK_JNI)/include
include $(BUILD_SHARED_LIBRARY)

Application.mk:

APP_ABI := armeabi-v7a
NDK_TOOLCHAIN_VERSION := 4.9
APP_CPPFLAGS += -std=c++11 -fexceptions  -frtti
APP_STL := gnustl_static

After this, I built the project for first time, this list of include folders will keep Eclipse from falsely flag error is c++ source code.

Trouble with Versioned Shared Libraries and Android

In one of my projects I am currently working on, I need to build libwebsocket  for Android platform.  libwebsockets needs OpenSSL or an alternative.  Luckily, some nice person has built OpenSSL for Android.  It's looked that this was going to be a straight forward task.

Actually, it was not.

In Linux, share libraries have versions, and they usually haven file names such as "libssl.so.1.0.0", and "libwebsockets.so.5".   Since building tools in Linux will generate symbolic "libssl.so" and "libwebsockets.so" for them respectively, everything works out fine.

One the other hand, Android does not use versioned shared libs, and valid shared lib file names must end with ".so".  So, we have to change file names accordingly, and everything should be with Android, right?

Not really.

After I change "libssl.so.1.0.0" to "libssl.so" (actually, it was changed at the above mentioned site already), it still have property called SONAME, "libssl.so.1.0.0".  libwebsockets.so I built, re-named from "libwebsockets.so.5", will ask for "libssl.so.1.0.0", when it is loaded into Android.  Just change the file names is not enough.

Here comes the hack as the last resort.  We can edit ".so" files such that SONAME's will end with ".so", and they will only ask for shared libs ended with ".so".  Two powerful tools needed are "objdump" and "rpl".  ".so" files are for machine consumption, and therefore not meant to be edited manually. You should use "rpl" with extra care.  Here is the example of "libssl.so", before and after the delicate operations.

Before:
huiying@huiying-PORTEGE-R835:~$ objdump -p libssl.so | grep so
libssl.so:     file format elf32-little
  NEEDED               libcrypto.so.1.0.0
  NEEDED               libdl.so
  NEEDED               libc.so
  SONAME               libssl.so.1.0.0

After: 

huiying@huiying-PORTEGE-R835:~$ rpl -R -e .so.1.0.0 "_1_0_0.so" libssl.so 

Replacing ".so.1.0.0" with "_1_0_0.so" (case sensitive) (partial words matched)
.
A Total of 2 matches replaced in 1 file searched.
huiying@huiying-PORTEGE-R835:~$ objdump -p libssl.so | grep so
libssl.so:     file format elf32-little
  NEEDED               libcrypto_1_0_0.so
  NEEDED               libdl.so
  NEEDED               libc.so
  SONAME               libssl_1_0_0.so

And don't forget to change file name "libssl.so" to "libssl_1_0_0.so".

You may ask, why I change from "libcrypto.so.1.0.0" to "libssl_1_0_0.so".  This is because they are of the same length.  If the replacement string is not of the same length, the library will be corrupted.

There are several websites describing the problem of SONAME, but none mentioned NEEDED field in .so files.  This is hack, but it works.  I have a running Android app to prove it.  I hope my experience will be helpful for others who come across similar situations.

Camera Cailbration

 I have been looking for a Java camera calibration code for a long time.  There is a C++ example of camera calibration at the official OpenCV documentation site.  Now that Java OpenCV is available, I decided that I will make a Java version.

The C++ example is quite complicated, in that it can use multiple calibration patterns, and can using a list of image as well as live video. I decided that this example only uses a folder of several chessboard photos as input.

import java.io.File;
import java.util.ArrayList;
import org.opencv.calib3d.Calib3d;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.MatOfPoint3f;
import org.opencv.core.Point3;
import org.opencv.core.Size;
import org.opencv.core.TermCriteria;
import org.opencv.highgui.Highgui;
import org.opencv.imgproc.Imgproc;

public class CalibChessBoard {
    int flagsCorner = Calib3d.CALIB_CB_ADAPTIVE_THRESH
            | Calib3d.CALIB_CB_FAST_CHECK 
            | Calib3d.CALIB_CB_NORMALIZE_IMAGE;
    int flagsCalib = Calib3d.CALIB_ZERO_TANGENT_DIST
            | Calib3d.CALIB_FIX_PRINCIPAL_POINT 
            | Calib3d.CALIB_FIX_K4
            | Calib3d.CALIB_FIX_K5;
    TermCriteria criteria = new TermCriteria(TermCriteria.EPS
            + TermCriteria.MAX_ITER, 40, 0.001);
    Size winSize = new Size(5, 5), zoneSize = new Size(-1, -1);
    Size patternSize;
    ArrayList objectPoints, imagePoints = new ArrayList();
    ArrayList vCorners;
    ArrayList vImg;
    Mat cameraMatrix = Mat.eye(3, 3, CvType.CV_64F);
    Mat distCoeffs = Mat.zeros(8, 1, CvType.CV_64F);
    ArrayList rvecs = new ArrayList();
    ArrayList tvecs = new ArrayList();

    CalibChessBoard() {
    }

    CalibChessBoard(Size patternSize) {
        this.patternSize = patternSize;
    }

    boolean getCorners(Mat gray, MatOfPoint2f corners) {
        if (!Calib3d.findChessboardCorners(gray, patternSize,
                corners, flagsCorner))
            return false;
        Imgproc.cornerSubPix(gray, corners, winSize, zoneSize,
                                   criteria);
        return true;
    }

    MatOfPoint3f getCorner3f() {
        MatOfPoint3f corners3f = new MatOfPoint3f();
        double squareSize = 50;
        Point3[] vp = new Point3[(int) (patternSize.height * 
                                            patternSize.width)];
        int cnt = 0;
        for (int i = 0; i < patternSize.height; ++i)
            for (int j = 0; j < patternSize.width; ++j, cnt++)
                vp[cnt] = new Point3(j * squareSize, 
                                     i * squareSize, 0.0d);
        corners3f.fromArray(vp);
        return corners3f;
    }

    public static void main(String[] args) {
        test0();
    }

    static void test0() {
        CalibChessBoard cb = new CalibChessBoard(new Size(8, 6));
        cb.getAllCornors("/the/photo/folder");
        cb.calibrate();
    }

    void calibrate() {
        double errReproj = Calib3d.calibrateCamera(objectPoints, 
                imagePoints,vImg.get(0).size(), cameraMatrix, 
                distCoeffs, rvecs, tvecs,flagsCalib);
        System.out.println("done, \nerrReproj = " + errReproj);
        System.out.println("cameraMatrix = \n" + cameraMatrix.dump());
        System.out.println("distCoeffs = \n" + distCoeffs.dump());
    }

    void getAllCornors(String path) {
        vImg = new ArrayList();
        objectPoints = new ArrayList();
        imagePoints = new ArrayList();
        MatOfPoint3f corners3f = getCorner3f();
        for (File f : new File(path).listFiles()) {
            Mat mat = Highgui.imread(f.getPath(), 
                           Highgui.CV_LOAD_IMAGE_COLOR);
            if (mat == null || mat.channels() != 3)
                continue;
            System.out.println("fn = " + f.getPath());
            System.out.println("mat.channels() = " + mat.channels() 
                    + ", " + mat.cols() + ", " + mat.rows());
            Mat gray = new Mat();
            Imgproc.cvtColor(mat, gray, Imgproc.COLOR_BGR2GRAY);
            MatOfPoint2f corners = new MatOfPoint2f();
            if (!getCorners(gray, corners))
                continue;
            objectPoints.add(corners3f);
            imagePoints.add(corners);
            vImg.add(mat);
        }
    }

    static {
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }
}

Here are the list of images that give the following results:

errReproj = 0.25108
cameraMatrix =
[614.561, 0, 319.5;
  0, 615.445, 239.5;
  0, 0, 1]
distCoeffs = [0.155040; 0.075046; 0; 0; -2.318117]

Java OpenCV Webcam

Camera is a very important piece of equipment in computer vision. My first tutorial will be a webcam program based on OpenCV. 1. We need an efficient piece of code that convert OpenCV image, a Mat, to a BufferedImage.

import java.awt.image.BufferedImage;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.highgui.Highgui;
import org.opencv.imgproc.Imgproc;

public class Mat2Image {
    Mat mat = new Mat();
    BufferedImage img;
    byte[] dat;
    public Mat2Image() {
    }
    public Mat2Image(Mat mat) {
        getSpace(mat);
    }
    public void getSpace(Mat mat) {
        this.mat = mat;
        int w = mat.cols(), h = mat.rows();
        if (dat == null || dat.length != w * h * 3)
            dat = new byte[w * h * 3];
        if (img == null || img.getWidth() != w || img.getHeight() != h
            || img.getType() != BufferedImage.TYPE_3BYTE_BGR)
                img = new BufferedImage(w, h, 
                            BufferedImage.TYPE_3BYTE_BGR);
        }
        BufferedImage getImage(Mat mat){
            getSpace(mat);
            mat.get(0, 0, dat);
            img.getRaster().setDataElements(0, 0, 
                               mat.cols(), mat.rows(), dat);
        return img;
    }
    static{
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }
}

2. Here is a simple class that captures video:

import java.awt.image.BufferedImage;
import org.opencv.core.Core;
import org.opencv.highgui.VideoCapture;

public class VideoCap {
    static{
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }

    VideoCapture cap;
    Mat2Image mat2Img = new Mat2Image();

    VideoCap(){
        cap = new VideoCapture();
        cap.open(0);
    } 
 
    BufferedImage getOneFrame() {
        cap.read(mat2Img.mat);
        return mat2Img.getImage(mat2Img.mat);
    }
}

3. Now we need a JFrame to display the video:

package org.ski;
import java.awt.EventQueue;
import java.awt.Graphics;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.border.EmptyBorder;

public class MyFrame extends JFrame {
    private JPanel contentPane;

  /**
  * Launch the application.
  */
    public static void main(String[] args) {
        EventQueue.invokeLater(new Runnable() {
            public void run() {
                try {
                    MyFrame frame = new MyFrame();
                    frame.setVisible(true);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });
    }

  /**
  * Create the frame.
  */
    public MyFrame() {
        setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        setBounds(100, 100, 650, 490);
        contentPane = new JPanel();
        contentPane.setBorder(new EmptyBorder(5, 5, 5, 5));
        setContentPane(contentPane);
        contentPane.setLayout(null);
  
        new MyThread().start();
    }
 
    VideoCap videoCap = new VideoCap();
 
    public void paint(Graphics g){
        g = contentPane.getGraphics();
        g.drawImage(videoCap.getOneFrame(), 0, 0, this);
    }
 
    class MyThread extends Thread{
        @Override
        public void run() {
            for (;;){
                repaint();
                try { Thread.sleep(30);
                } catch (InterruptedException e) {    }
            }  
        } 
    }
}

Computer Vision and Java

In the field of computer vision, there are several choices in programming languages.  Matlab is the first choice in terms of ease of use.  Python is chosen for similar reasons.  C++ is used for it's pure speed.  Java just falls somewhere in the middle, and mostly neglected.  But Java has been my first choice for computer vision research/development for several years.

I choose Java for several reasons.  First, it's very easy to debug a java program.  Every time a program crashes, it comes right to the line of the source code, speed up debugging greatly.  It comes with full featured graphics tools, unlike C++, where you need another tool for image loading/writing.  Netbeans makes my quick and dirty prototyping much easier.

I had developed some tools for image processing, computer vision algorithm developing in Java for several year, meanwhile OpenCV are getting more and more versatile.  Finally, I decided that I couldn't limit my programming language in Java any longer.  A little over a year ago, I moved my main algorithm development to C++, based on OpenCV.  It has been rewarding.  Still, I miss my days when I can mainly code in Java.

Then desktop Java interface for OpenCV becomes available.  Now I can combine ease of coding in Java and the capabilities and speed of OpenCV.  Life just couldn't get any better.

It's still in the early stage of OpenCV Java interface.  There are not a ton of examples/tutorial available.  Since I have been working with JNI for several year, I'd like to translate some OpenCV C++ examples in to running Java example for some time to come.