Thursday, November 17, 2016

Background Subtraction

Since OpenCV 3, background subtraction by Java becomes possible.  After searching for one example without success, I decided to put out one myself.

The class I'd like use is BackgroundSubtractorMOG2.  The confusing part is that you cannot use its constructor to create an instance.  Video.createBackgroundSubtractorMOG2() is needed for this task.

1. Create an instance of BackgroundSubtractorMOG2.

  • BackgroundSubtractorMOG2 fgbg =Video.createBackgroundSubtractorMOG2();


2. Apply it to the incoming image.

  • fgbg.apply(mat, mask);

where mat is RGB image from a webcam, and mask is the output gray scale foreground mask

I also updated my ImageConvertor, so it can convert both three channel and one channel OpenCV Mat to BufferedImage for displaying.  Here is a screenshot of example: after waiting for a while so my office scene disappears from the mask, I put my hand in the view.

Top is mat, bottom is mask


The complete code listings is at https://github.com/huiyingShen/Java-Background-Subtraction


Friday, January 8, 2016

Android App with OpenCV and C++, Part 3

1. Add Camera

a. Add permission line  in file "AndroidManifest.xml": 
<uses-permission android:name="android.permission.CAMERA" />
b. Subclass SurfaceView: MyView:
 public class MyView extends SurfaceView{
     Bitmap bmp2Display;
     public MyView(Context context, AttributeSet attrs) {
           super(context, attrs);
     }
     Paint paint = new Paint();
     @Override
     protected void onDraw(Canvas canvas) {
         super.onDraw(canvas);
         canvas.drawText("time = "+System.currentTimeMillis(), 50, 100, paint);
     }
}

c. Add an instance of MyView in "activity_mail.xml":
<org.ski.helloworld.MyView
      android:id="@+id/myView"
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:layout_alignParentLeft="true"
      android:layout_alignParentTop="true" />

d. Add "implements SurfaceHolder.Callback" in MainActivity, and added unimplemented functions as hinted

public class MainActivity extends Activity implements SurfaceHolder.Callback{
    //...
    // ...
}

e. Add MyView, Camera, its callback and image size:

MyView myView; 
int width = 960, height=720;
Camera mCamera;
Camera.PreviewCallback cbWithBuffer;

f. Define callback function:

 Camera.PreviewCallback cabWithBuffer = new Camera.PreviewCallback() {
    public void onPreviewFrame(byte[] _data, Camera _camera) {
                       // do something
        _camera.addCallbackBuffer(_data);
    }
 };
g. Add implementations for SurfaceHolder.Callback
@Override
public void surfaceCreated(SurfaceHolder holder) {
    mCamera = Camera.open();
    mCamera.addCallbackBuffer(new byte[width * height * 2]); // YUV
    mCamera.setPreviewCallbackWithBuffer(cabWithBuffer);  
    try {
        mCamera.setPreviewDisplay(holder);
    } catch (IOException exception) {
        mCamera.release();
        mCamera = null;
    }
}

public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
    Camera.Parameters parameters = mCamera.getParameters();
    parameters.setPreviewSize(width, height);
    mCamera.setParameters(parameters);
    mCamera.setPreviewCallbackWithBuffer(cbWithBuffer);
    mCamera.startPreview();
}

@Override
public void surfaceDestroyed(SurfaceHolder holder) {
    mCamera.stopPreview();
    mCamera.setPreviewCallbackWithBuffer(null);
    mCamera.release();
    mCamera = null;
}

2. Process video: 

a. Declare matrices:
Mat mYuv, mRgba;
b. Allocate space for mYuv, mRgb in createOpencvMatrices():
    mYuv = new Mat(height + height/2, width, CvType.CV_8UC1);
    mRgb = new Mat(height, width, CvType.CV_8UC3);

c. Process image in cbWithBuffer:

    mYuv.put(0, 0, _data); 
    Imgproc.cvtColor(mYuv, mRgb, Imgproc.COLOR_YUV2RGB_NV21, 3);
    processImage(mRgb.getNativeObjAddr());

d. Add function declaration

    public native int processImage(long matAddrRgb);

e. Add image processing implementation in HelloWorld.cpp:

JNIEXPORT void JNICALL Java_org_ski_helloworld_MainActivity_processImage(JNIEnv*, jobject, jlong addrRgb)
{
    Mat &rgb = *(Mat *) addrRgb;
    int w = rgb.cols, h = rgb.rows;
    Rect r(w/4,h/4,w/2,h/2);
    Mat roi = rgb(r).clone();
    Mat gray, edge;
    cvtColor(roi, gray, CV_RGB2GRAY);
    blur( gray, gray, Size(5,5));
    Canny( gray, edge, 50, 150, 3);
    edge.convertTo(gray, CV_8UC1);
    cvtColor(gray,roi, CV_GRAY2RGB);
    roi.copyTo(rgb(r));
}
The zipped project is HelloWorld_02.zip

Thursday, January 7, 2016

Android App with OpenCV and C++, Part 2

1. Add opencv to the workspace

File->Import..->Android->Existing Code into Workspace.  Locate the path and check "Copy projects into workspace"


2. Add opencv dependency to the project:

Right click on the project->properties->Android, click "Add", and add the opencv lib:

3. Add opencv support code:


Add declarations:
Mat mat;
private BaseLoaderCallback  mLoaderCallback;
Overload onResume():
@Override
public void onResume(){
    super.onResume();
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_9, this, mLoaderCallback);
}

Add initilization in onCreate(), and any matrices are to be created after opencv lib are laoded successfully:
@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    mLoaderCallback = new BaseLoaderCallback(this) {
    
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
            case LoaderCallbackInterface.SUCCESS:
                Log.i("HelloWorld: ", "OpenCV loaded successfully");
                System.loadLibrary("HelloWorld");
                createOpencvMatrices();
                break;
            default:
               super.onManagerConnected(status);
            }
        }
    };
}
    
void createOpencvMatrices(){
    mat = new Mat();
    // other matrices to be created, ...
}

Add a test function:
 public native int test0(long matAddrDat); 

Edit file activity_main.xml in res->layout:
<TextView android:id="@+id/tvHello" android:layout_width="wrap_content" android:layout_height="wrap_content" android:text="Hello World" />
Declare TextView tvHello, and added this to onCreate:
tvHello = (TextView) findViewById(R.id.tvHello);

Add following after createOpencvMatrices();
createOpencvMatrices(); // matrix creation comes after loadLibrary
int out = test0(mat.getNativeObjAddr());  // test function
tvHello.setText("output from test0() = "+ out);

Edit HelloWorld.cpp:
......
JNIEXPORT int JNICALL Java_org_ski_helloworld_MainActivity_test0(JNIEnv*, jobject, jlong addrData)
{
    Mat& mat = *(Mat*)addrData;
    mat.create(1,3,CV_32F);
    Mat_<float> &tmp = (Mat_<float> &)mat;
    Mat_<float>::iterator it;
    int i=1;
    for (it =tmp.begin(); it!= tmp.end(); it++,i++)
        *it = i;

    int sum = 0;
    for (it = tmp.begin(); it!= tmp.end(); it++)
        sum += *it;
    return sum;
}
......

You should get "output from test0() = 6" after you launch the app. The zip file of the project: link.

Android App with OpenCV and C++, Part 1

1. Preparation


  • Install Eclipse
  • Install Android SDK
  • Install Android NDK
  • Download and save OpenCV4Android

2. Create a hello world project

a. Create a regular Android application project

File->New->Android Application Project:

    Select Blank Activity, click "Next", and "Finish"

    b. Add NDK support:

    Right click on HelloWorld project->Android Tools->Add Native Support.

    Now you have new folder jni with two files:

    Create a file "Application.mk", with following content:
    APP_ABI := armeabi-v7a
    NDK_TOOLCHAIN_VERSION := 4.9
    APP_CPPFLAGS += -std=c++11 -fexceptions  -frtti
    APP_STL := gnustl_static
    
    

    And put to folder "jni".

    Edit "Android.mk" so it contains:
    LOCAL_PATH := $(call my-dir)
    include $(CLEAR_VARS)
    OPENCV_SDK_JNI = c:\\OpenCV-2.4.11-android-sdk\\sdk\\native\\jni
    include $(OPENCV_SDK_JNI)/OpenCV.mk
    LOCAL_MODULE    := HelloWorld
    LOCAL_SRC_FILES := HelloWorld.cpp
    LOCAL_C_INCLUDES += $(OPENCV_SDK_JNI)/include
    include $(BUILD_SHARED_LIBRARY)
    
    Where OPENCV_SDK_JNI should point to the correct location in your machine.

    Edit HelloWorld.cpp. Save everything.

    #include <jni.h>
    #include <opencv2/highgui/highgui.hpp>
    #include <opencv2/imgproc/imgproc.hpp>
    using namespace std;
    using namespace cv;
    extern "C" {
    JNIEXPORT int JNICALL Java_org_ski_helloworld_MainActivity_test0(JNIEnv*, jobject, jlong addrData){
             Mat& mDat = *(Mat*)addrData;
            //  doing stuff ...
        }
    }
    
    

    Select "Build Project".  Now you should have correct syntax highlighting of c++ files.



    Tuesday, August 25, 2015

    Android NDK C++ Code Error Checking Trouble with Eclipse

    Eclipse for Android NDK is mostly a user friendly environment.  But sometimes I encounter error checkng problem with c++ code.  While compile/build goes smoothly, every time I open a c++ file, many syntax error show up, and I got 100+ error messages under problem tab.  Under this circumstance, I cannot build the project before the error-flagging file is closed, and the error messages are manually deleted.  The build process goes smoothly: there is no error to begin with.

    This brings a lot of inconvenience to coding.  On the other hand, some of my projects do not have this problem.  After some experimenting, I finally figure out why:  the include path list for syntax is not updated after first build.  One way to solve this problem is get Android.mk and Application.mk ready before the first build.  Here are my example, OpenCV is used in this project:


    Android.mk:

    LOCAL_PATH := $(call my-dir)
    include $(CLEAR_VARS)
    OPENCV_SDK_JNI = C:\\OpenCV-2.4.10-android-sdk\\sdk\\native\\jni
    include $(OPENCV_SDK_JNI)/OpenCV.mk
    LOCAL_MODULE    := hello
    LOCAL_SRC_FILES := hello.cpp
    LOCAL_C_INCLUDES += $(OPENCV_SDK_JNI)/include
    include $(BUILD_SHARED_LIBRARY)

    Application.mk:

    APP_ABI := armeabi-v7a
    NDK_TOOLCHAIN_VERSION := 4.9
    APP_CPPFLAGS += -std=c++11 -fexceptions  -frtti
    APP_STL := gnustl_static

    After this, I built the project for first time, this list of include folders will keep Eclipse from falsely flag error is c++ source code.








    Monday, May 18, 2015

    Trouble with Versioned Shared Libraries and Android

    In one of my projects I am currently working on, I need to build libwebsocket  for Android platform.  libwebsockets needs OpenSSL or an alternative.  Luckily, some nice person has built OpenSSL for Android.  It's looked that this was going to be a straight forward task.

    Actually, it was not.

    In Linux, share libraries have versions, and they usually haven file names such as "libssl.so.1.0.0", and "libwebsockets.so.5".   Since building tools in Linux will generate symbolic "libssl.so" and "libwebsockets.so" for them respectively, everything works out fine.

    One the other hand, Android does not use versioned shared libs, and valid shared lib file names must end with ".so".  So, we have to change file names accordingly, and everything should be with Android, right?

    Not really.

    After I change "libssl.so.1.0.0" to "libssl.so" (actually, it was changed at the above mentioned site already), it still have property called SONAME, "libssl.so.1.0.0".  libwebsockets.so I built, re-named from "libwebsockets.so.5", will ask for "libssl.so.1.0.0", when it is loaded into Android.  Just change the file names is not enough.

    Here comes the hack as the last resort.  We can edit ".so" files such that SONAME's will end with ".so", and they will only ask for shared libs ended with ".so".  Two powerful tools needed are "objdump" and "rpl".  ".so" files are for machine consumption, and therefore not meant to be edited manually. You should use "rpl" with extra care.  Here is the example of "libssl.so", before and after the delicate operations.

    Before:
    huiying@huiying-PORTEGE-R835:~$ objdump -p libssl.so | grep so
    libssl.so:     file format elf32-little
      NEEDED               libcrypto.so.1.0.0
      NEEDED               libdl.so
      NEEDED               libc.so
      SONAME               libssl.so.1.0.0

    After: 
    huiying@huiying-PORTEGE-R835:~$ rpl -R -e .so.1.0.0 "_1_0_0.so" libssl.so 
    Replacing ".so.1.0.0" with "_1_0_0.so" (case sensitive) (partial words matched)
    .
    A Total of 2 matches replaced in 1 file searched.
    huiying@huiying-PORTEGE-R835:~$ objdump -p libssl.so | grep so
    libssl.so:     file format elf32-little
      NEEDED               libcrypto_1_0_0.so
      NEEDED               libdl.so
      NEEDED               libc.so
      SONAME               libssl_1_0_0.so

    And don't forget to change file name "libssl.so" to "libssl_1_0_0.so".

    You may ask, why I change from "libcrypto.so.1.0.0" to "libssl_1_0_0.so".  This is because they are of the same length.  If the replacement string is not of the same length, the library will be corrupted.

    There are several websites describing the problem of SONAME, but none mentioned NEEDED field in .so files.  This is hack, but it works.  I have a running Android app to prove it.  I hope my experience will be helpful for others who come across similar situations.



    Wednesday, October 30, 2013

    Camera Cailbration

     I have been looking for a Java camera calibration code for a long time.  There is a C++ example of camera calibration at the official OpenCV documentation site.  Now that Java OpenCV is available, I decided that I will make a Java version.

    The C++ example is quite complicated, in that it can use multiple calibration patterns, and can using a list of image as well as live video. I decided that this example only uses a folder of several chessboard photos as input.




    import java.io.File;
    import java.util.ArrayList;
    import org.opencv.calib3d.Calib3d;
    import org.opencv.core.Core;
    import org.opencv.core.CvType;
    import org.opencv.core.Mat;
    import org.opencv.core.MatOfPoint2f;
    import org.opencv.core.MatOfPoint3f;
    import org.opencv.core.Point3;
    import org.opencv.core.Size;
    import org.opencv.core.TermCriteria;
    import org.opencv.highgui.Highgui;
    import org.opencv.imgproc.Imgproc;
    
    public class CalibChessBoard {
        int flagsCorner = Calib3d.CALIB_CB_ADAPTIVE_THRESH
                | Calib3d.CALIB_CB_FAST_CHECK 
                | Calib3d.CALIB_CB_NORMALIZE_IMAGE;
        int flagsCalib = Calib3d.CALIB_ZERO_TANGENT_DIST
                | Calib3d.CALIB_FIX_PRINCIPAL_POINT 
                | Calib3d.CALIB_FIX_K4
                | Calib3d.CALIB_FIX_K5;
        TermCriteria criteria = new TermCriteria(TermCriteria.EPS
                + TermCriteria.MAX_ITER, 40, 0.001);
        Size winSize = new Size(5, 5), zoneSize = new Size(-1, -1);
        Size patternSize;
        ArrayList objectPoints, imagePoints = new ArrayList();
        ArrayList vCorners;
        ArrayList vImg;
        Mat cameraMatrix = Mat.eye(3, 3, CvType.CV_64F);
        Mat distCoeffs = Mat.zeros(8, 1, CvType.CV_64F);
        ArrayList rvecs = new ArrayList();
        ArrayList tvecs = new ArrayList();
    
        CalibChessBoard() {
        }
    
        CalibChessBoard(Size patternSize) {
            this.patternSize = patternSize;
        }
    
        boolean getCorners(Mat gray, MatOfPoint2f corners) {
            if (!Calib3d.findChessboardCorners(gray, patternSize,
                    corners, flagsCorner))
                return false;
            Imgproc.cornerSubPix(gray, corners, winSize, zoneSize,
                                       criteria);
            return true;
        }
    
        MatOfPoint3f getCorner3f() {
            MatOfPoint3f corners3f = new MatOfPoint3f();
            double squareSize = 50;
            Point3[] vp = new Point3[(int) (patternSize.height * 
                                                patternSize.width)];
            int cnt = 0;
            for (int i = 0; i < patternSize.height; ++i)
                for (int j = 0; j < patternSize.width; ++j, cnt++)
                    vp[cnt] = new Point3(j * squareSize, 
                                         i * squareSize, 0.0d);
            corners3f.fromArray(vp);
            return corners3f;
        }
    
        public static void main(String[] args) {
            test0();
        }
    
        static void test0() {
            CalibChessBoard cb = new CalibChessBoard(new Size(8, 6));
            cb.getAllCornors("/the/photo/folder");
            cb.calibrate();
        }
    
        void calibrate() {
            double errReproj = Calib3d.calibrateCamera(objectPoints, 
                    imagePoints,vImg.get(0).size(), cameraMatrix, 
                    distCoeffs, rvecs, tvecs,flagsCalib);
            System.out.println("done, \nerrReproj = " + errReproj);
            System.out.println("cameraMatrix = \n" + cameraMatrix.dump());
            System.out.println("distCoeffs = \n" + distCoeffs.dump());
        }
    
        void getAllCornors(String path) {
            vImg = new ArrayList();
            objectPoints = new ArrayList();
            imagePoints = new ArrayList();
            MatOfPoint3f corners3f = getCorner3f();
            for (File f : new File(path).listFiles()) {
                Mat mat = Highgui.imread(f.getPath(), 
                               Highgui.CV_LOAD_IMAGE_COLOR);
                if (mat == null || mat.channels() != 3)
                    continue;
                System.out.println("fn = " + f.getPath());
                System.out.println("mat.channels() = " + mat.channels() 
                        + ", " + mat.cols() + ", " + mat.rows());
                Mat gray = new Mat();
                Imgproc.cvtColor(mat, gray, Imgproc.COLOR_BGR2GRAY);
                MatOfPoint2f corners = new MatOfPoint2f();
                if (!getCorners(gray, corners))
                    continue;
                objectPoints.add(corners3f);
                imagePoints.add(corners);
                vImg.add(mat);
            }
        }
    
        static {
            System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        }
    }
    
    
    Here are the list of images that give the following results:

    errReproj = 0.25108
    cameraMatrix =
    [614.561, 0, 319.5;
      0, 615.445, 239.5;
      0, 0, 1]
    distCoeffs = [0.155040; 0.075046; 0; 0; -2.318117]