本文档主要是简单地描述了在安卓环境下如何部署模型，在部署过程中涉及到opencv和百度飞浆的整合以及安卓中JNI的使用。

本文档环境如下

操作系统：win10
IDE ：android studio 4.1.2
opencv：3.4.5

整合过程中，会先在百度飞浆sample项目的基础上整合百度飞浆的功能，然后引入项目中需要使用到的opencv模块，接着使用jni调用自定义模型进行数据处理，最后接收处理的结果并显示在安卓界面上。

一基础环境准备

在android studio中创建一个Native工程，步骤如下：

图1 创建Native项目

图2 配置项目

图3 配置activity

选择如图所示的配置，点击完成按钮，稍等一段时间，会看到如下图4所示的界面，表示项目创建完成。

图4 项目初始化界面

从上图4可以看到，IDE已经构建好了完整的基础JNI项目骨架,尝试运行项目，可以在安卓模拟器中看到如下运行结果

图5 运行结果

从上图5可知，一个基础的JNI开发调试环境已经构建完成。

二引入飞浆

在引入飞浆功能，需要先修改项目的app目录下的 build.gradle 文件，加上飞浆相关的依赖文件。

图6 需要修改的文件

修改后的完整的 build.gradle 文件内容如下：

import java.security.MessageDigest

plugins {
    id 'com.android.application'
}

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.3"

    defaultConfig {
        applicationId "com.example.aisample"
        minSdkVersion 16
        targetSdkVersion 30
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags ""
            }
            ndk {
                //选择要添加的对应cpu类型的.so库。
                abiFilters 'armeabi-v7a'
                // 还可以添加 'x86', 'x86_64', 'mips', 'mips64'
            }
        }
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"
        }
    }
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    sourceSets {
        main {
            jniLibs.srcDirs = ['src/main/jniLibs']
        }
    }
}

dependencies {
    implementation fileTree(include: ['*.jar'], dir: 'libs')
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'com.google.android.material:material:1.1.0'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    testImplementation 'junit:junit:4.+'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    implementation files('libs/PaddlePredictor.jar')
}


def paddleLiteLibs = 'https://paddlelite-demo.bj.bcebos.com/libs/android/paddle_lite_libs_v2_9_0.tar.gz'
task downloadAndExtractPaddleLiteLibs(type: DefaultTask) {
    doFirst {
        println "Downloading and extracting Paddle Lite libs"
    }
    doLast {
        // Prepare cache folder for libs
        if (!file("cache").exists()) {
            mkdir "cache"
        }
        // Generate cache name for libs
        MessageDigest messageDigest = MessageDigest.getInstance('MD5')
        messageDigest.update(paddleLiteLibs.bytes)
        String cacheName = new BigInteger(1, messageDigest.digest()).toString(32)
        // Download libs
        if (!file("cache/${cacheName}.tar.gz").exists()) {
            ant.get(src: paddleLiteLibs, dest: file("cache/${cacheName}.tar.gz"))
        }
        // Unpack libs
        if (!file("cache/${cacheName}").exists()) {
            copy {
                from tarTree("cache/${cacheName}.tar.gz")
                into "cache/${cacheName}"
            }
        }

        // Copy PaddlePredictor.jar
        if (!file("libs/PaddlePredictor.jar").exists()) {
            copy {
                from "cache/${cacheName}/java/PaddlePredictor.jar"
                into "libs"
            }
        }
        // Copy libpaddle_lite_jni.so for armeabi-v7a and arm64-v8a
        if (!file("src/main/jniLibs/armeabi-v7a/libpaddle_lite_jni.so").exists()) {
            copy {
                from "cache/${cacheName}/java/libs/armeabi-v7a/"
                into "src/main/jniLibs/armeabi-v7a"
//                into "src/main/jni/armeabi-v7a"
            }
        }
        if (!file("src/main/jniLibs/arm64-v8a/libpaddle_lite_jni.so").exists()) {
            copy {
                from "cache/${cacheName}/java/libs/arm64-v8a/"
                into "src/main/jniLibs/arm64-v8a"
//                into "src/main/jni/arm64-v8a"
            }
        }
    }
}
preBuild.dependsOn downloadAndExtractPaddleLiteLibs

def paddleLiteModels = [
        [
                'src' : 'https://paddlelite-demo.bj.bcebos.com/models/deeplab_mobilenet_fp32_for_cpu_v2_9_0.tar.gz',
                'dest': 'src/main/assets/image_segmentation/models/deeplab_mobilenet_for_cpu'
        ],
]
task downloadAndExtractPaddleLiteModels(type: DefaultTask) {
    doFirst {
        println "Downloading and extracting Paddle Lite models"
    }
    doLast {
        // Prepare cache folder for models
        String cachePath = "cache"
        if (!file("${cachePath}").exists()) {
            mkdir "${cachePath}"
        }
        paddleLiteModels.eachWithIndex { model, index ->
            MessageDigest messageDigest = MessageDigest.getInstance('MD5')
            messageDigest.update(model.src.bytes)
            String cacheName = new BigInteger(1, messageDigest.digest()).toString(32)
            // Download the target model if not exists
            boolean copyFiles = !file("${model.dest}").exists()
            if (!file("${cachePath}/${cacheName}.tar.gz").exists()) {
                ant.get(src: model.src, dest: file("${cachePath}/${cacheName}.tar.gz"))
                copyFiles = true; // force to copy files from the latest archive files
            }
            // Copy model file
            if (copyFiles) {
                copy {
                    from tarTree("${cachePath}/${cacheName}.tar.gz")
                    into "${model.dest}"
                }
            }
        }
    }
}
preBuild.dependsOn downloadAndExtractPaddleLiteModels

注意，默认的项目没有assets文件夹，需要手动创建，创建步骤如下

图7 创建assets文件夹

创建好文件夹后，复制飞浆所需要的资源到项目对应的路径下面

图8 复制好资源的项目

复制好所需的项目资源后，接下来复制所需的的java代码到对应的项目路径了。然后运行项目，在安卓模拟器里可以看到如下的界面。

图9 飞浆运行结果

三引入opencv

在引入opencv时，需要先到opencv官网下载好对应的资源包。

图10 opencv官网下载地址

选择下载的版本为 3.4.5 ，类型的为安卓平台。注意这里下载的opencv版本需要根据项目进行调整，并不是任意一个版本或者是越新的版本越好。下载完成后，将下载好的压缩包解压，得到如下的结构

图11 opencv解压后的目录

在项目里创建一个JNI目录，创建此目录的方法如下

图12 创建JNI目录

先将图11中对应的libs里面的内容复制到此JNI目录里。然后将图11中对应的jni目录里的include文件夹复制到项目的cpp的目录下。

复制完资源到对应的目录后，然后修改cpp路径下的 CMakeLists.txt 文件，通过cmake引入opencv功能。修改后的完整的 CMakeLists.txt内容如下

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.10.2)

# Declares and names the project.

project("aisample")


# 设置include文件夹的地址
include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)
include_directories("include")


# 设置opencv的动态库
add_library(libopencv_java3 SHARED IMPORTED)
set_target_properties(libopencv_java3 PROPERTIES IMPORTED_LOCATION
        ${CMAKE_SOURCE_DIR}/../jni/${ANDROID_ABI}/libopencv_java3.so)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        native-lib

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        native-lib.cpp)

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        native-lib

        # 引入opencv
        libopencv_java3
        # Links the target library to the log library
        # included in the NDK.
        ${log-lib})

然后修改app目录下的build.gradle文件，在defaultConfig目录下的externalNativeBuild的cmake增加如下配置arguments "-DANDROID_STL=c++_shared" 。除此以外，还需要加上opencv相关的jar引用配置。

修改后的文件的内容如下

import java.security.MessageDigest

plugins {
    id 'com.android.application'
}

android {
    compileSdkVersion 28
    buildToolsVersion "30.0.3"

    defaultConfig {
        applicationId "com.example.aisample"
        minSdkVersion 16
        targetSdkVersion 28
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags ""
                arguments "-DANDROID_STL=c++_shared"
            }
            ndk {
                //选择要添加的对应cpu类型的.so库。
                abiFilters 'armeabi-v7a'
                // 还可以添加 'x86', 'x86_64', 'mips', 'mips64'
            }
        }
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"
        }
    }
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    sourceSets {
        main {
            jniLibs.srcDirs = ['src/main/jniLibs']
        }
    }
}

dependencies {
    implementation fileTree(include: ['*.jar'], dir: 'libs')
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'com.google.android.material:material:1.1.0'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    implementation 'com.android.support:appcompat-v7:28.0.0'
    implementation 'com.android.support:design:28.0.0'
    testImplementation 'junit:junit:4.+'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    implementation files('libs/PaddlePredictor.jar')
    implementation 'org.opencv:openCVLibrary:3.4.0'
}


def paddleLiteLibs = 'https://paddlelite-demo.bj.bcebos.com/libs/android/paddle_lite_libs_v2_9_0.tar.gz'
task downloadAndExtractPaddleLiteLibs(type: DefaultTask) {
    doFirst {
        println "Downloading and extracting Paddle Lite libs"
    }
    doLast {
        // Prepare cache folder for libs
        if (!file("cache").exists()) {
            mkdir "cache"
        }
        // Generate cache name for libs
        MessageDigest messageDigest = MessageDigest.getInstance('MD5')
        messageDigest.update(paddleLiteLibs.bytes)
        String cacheName = new BigInteger(1, messageDigest.digest()).toString(32)
        // Download libs
        if (!file("cache/${cacheName}.tar.gz").exists()) {
            ant.get(src: paddleLiteLibs, dest: file("cache/${cacheName}.tar.gz"))
        }
        // Unpack libs
        if (!file("cache/${cacheName}").exists()) {
            copy {
                from tarTree("cache/${cacheName}.tar.gz")
                into "cache/${cacheName}"
            }
        }

        // Copy PaddlePredictor.jar
        if (!file("libs/PaddlePredictor.jar").exists()) {
            copy {
                from "cache/${cacheName}/java/PaddlePredictor.jar"
                into "libs"
            }
        }
        // Copy libpaddle_lite_jni.so for armeabi-v7a and arm64-v8a
        if (!file("src/main/jniLibs/armeabi-v7a/libpaddle_lite_jni.so").exists()) {
            copy {
                from "cache/${cacheName}/java/libs/armeabi-v7a/"
                into "src/main/jniLibs/armeabi-v7a"
//                into "src/main/jni/armeabi-v7a"
            }
        }
        if (!file("src/main/jniLibs/arm64-v8a/libpaddle_lite_jni.so").exists()) {
            copy {
                from "cache/${cacheName}/java/libs/arm64-v8a/"
                into "src/main/jniLibs/arm64-v8a"
//                into "src/main/jni/arm64-v8a"
            }
        }
    }
}
preBuild.dependsOn downloadAndExtractPaddleLiteLibs

def paddleLiteModels = [
        [
                'src' : 'https://paddlelite-demo.bj.bcebos.com/models/deeplab_mobilenet_fp32_for_cpu_v2_9_0.tar.gz',
                'dest': 'src/main/assets/image_segmentation/models/deeplab_mobilenet_for_cpu'
        ],
]
task downloadAndExtractPaddleLiteModels(type: DefaultTask) {
    doFirst {
        println "Downloading and extracting Paddle Lite models"
    }
    doLast {
        // Prepare cache folder for models
        String cachePath = "cache"
        if (!file("${cachePath}").exists()) {
            mkdir "${cachePath}"
        }
        paddleLiteModels.eachWithIndex { model, index ->
            MessageDigest messageDigest = MessageDigest.getInstance('MD5')
            messageDigest.update(model.src.bytes)
            String cacheName = new BigInteger(1, messageDigest.digest()).toString(32)
            // Download the target model if not exists
            boolean copyFiles = !file("${model.dest}").exists()
            if (!file("${cachePath}/${cacheName}.tar.gz").exists()) {
                ant.get(src: model.src, dest: file("${cachePath}/${cacheName}.tar.gz"))
                copyFiles = true; // force to copy files from the latest archive files
            }
            // Copy model file
            if (copyFiles) {
                copy {
                    from tarTree("${cachePath}/${cacheName}.tar.gz")
                    into "${model.dest}"
                }
            }
        }
    }
}
preBuild.dependsOn downloadAndExtractPaddleLiteModels

四整合项目

将项目自定义文件复制到src\main\cpp\ 目录下，然后在 CMakeLists.txt 文件里增加配置，修改后的配置文件如下

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.10.2)

# Declares and names the project.

project("aisample")


# 设置include文件夹的地址
include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)
include_directories("include")


# 设置opencv的动态库
add_library(libopencv_java3 SHARED IMPORTED)
set_target_properties(libopencv_java3 PROPERTIES IMPORTED_LOCATION
        ${CMAKE_SOURCE_DIR}/../jni/${ANDROID_ABI}/libopencv_java3.so)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        native-lib

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        native-lib.cpp
		
		# 需要使用到的cpp文件这里都要引入，否则会出问题
        demo/demo.cpp
        demo/Scanner.cpp
        demo/document_detection.cpp


        )

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        native-lib

        # 引入opencv
        libopencv_java3
        # Links the target library to the log library
        # included in the NDK.
        ${log-lib})

然后修改MainActivity文件，主要修改的地方如下

在模型运行成功后调用自定义模型对飞浆的输出结果进行处理

public void onRunModelSuccessed() {
    // obtain results and update UI
    tvInferenceTime.setText("Inference time: " + predictor.inferenceTime() + " ms");
    //飞浆训练后的图片
    Bitmap outputImage = predictor.outputImage();
    // 原始的图片
    Bitmap rawImage = predictor.rawImage();
    //裁剪后的图片
    Bitmap scaledImage = predictor.scaledImage();

   //接收飞浆出来的数据，使用此数据进行训练
    new Thread(() -> {
        train(rawImage, outputImage);
    }).start();

    if (outputImage != null) {
        ivInputImage.setImageBitmap(outputImage);
    }
    tvOutputResult.setText(predictor.outputResult());
    tvOutputResult.scrollTo(0, 0);
}

自定义处理过程的核心代码如下


private void train(Bitmap rawImage, Bitmap outputImage) {

    saveImg(rawImage, "original_rawImage_");
    saveImg(outputImage, "original_outputImage_");

    Bitmap whiteImg = ImageUtil.fileBackgroup(outputImage);

    saveImg(whiteImg, "original_fill_outputImage_");


    //mat转灰度
    Mat mat_source_gray = ImageUtil.bitMap2GrayMat(rawImage);
    Mat mat_mark_gray = ImageUtil.bitMap2GrayMat(whiteImg);


    // 回显灰度图检查是否正确
    Bitmap bitmap_mark_dist = ImageUtil.mat2Bitmap(mat_mark_gray);
    saveImg(bitmap_mark_dist, "mark_gray_");


    Mat demo = new Mat();

    test(mat_source_gray.getNativeObjAddr(), mat_mark_gray.getNativeObjAddr(), demo.getNativeObjAddr());

    Bitmap resultImg = ImageUtil.mat2Bitmap(demo);

    saveImg(resultImg, "return_image_");

    ////通过 activity 的 runOnUiThread方法更新界面
    runOnUiThread(() -> ivInputImage.setImageBitmap(resultImg));


}

这里使用到的JNI接口的定义如下


public native long test(long src, long dist, long returnVal);

对应的native code里的接口的定义如下

extern "C" JNIEXPORT jlong JNICALL
Java_com_example_aisample_MainActivity_test(JNIEnv *env, jobject cls, jlong addr_src_pic,
                                            jlong addr_mask_pic, jlong return_val) {
    ////////
    // 这里省略具体的处理代码
    //////
    
}

此时完整的项目结构如下

图13 完整的项目结构

五输出结果

完成相关的处理代码后，运行项目，可以看到在安卓模拟器的界面显示如下

图14 运行结果

从图14可以看到，输出结果显示图像已经处理完成，取得了预期的效果。

此外，打开IDE的文件浏览器功能，在/storage/emulated/0/Pictures目录下可以看到已经生成了下面几张图片，从这几张图片里可以完全了解整个图片处理的全过程。

图15 处理过程中生成的图片

安卓环境下人工智能模型部署

有没有一扇窗,能让你不绝望,看一看花花世界,然来梦一场,有人哭,有人笑,有人输,有人老,到结局还不是一样

一基础环境准备

二引入飞浆

三引入opencv

四整合项目

五输出结果

FEATURED TAGS

FRIENDS

一 基础环境准备

二 引入飞浆

三 引入opencv

四 整合项目

五 输出结果

FEATURED TAGS

FRIENDS

一基础环境准备

二引入飞浆

三引入opencv

四整合项目

五输出结果