C / C++ 部署 Tensorflow 模型

前言

最近有个项目，线上环境需要使用 C++ 来进行识别。为了方便调试，模型使用 python 代码进行训练，模型保存为 Tensorflow pb 格式。

环境

macOS Monterey 12.6
Tensorflow 2.10.0
Bazel 5.1.1 (Tensorflow 依赖)
Python 3.10 (Bazel 依赖)
OpenJDK 11 (Bazel 依赖)
XCode 14.0.1 (14A400)

直接安装

一般的包管理工具的 libtensorflow 仅包含 libtensorflow.so，只能调用 C Api。

# MacOS
brew install libtensorflow
 
# Ubuntu
sudo apt-get install libtensorflow

编译

此方法可以根据需求自行编译 libtensorflow.so 或 libtensorflow_cc.so，自由度更大。

~/Downloads/tensorflow-2.10.0 % ./configure
You have bazel 5.1.1 installed.
Please specify the location of python. [Default is /opt/homebrew/opt/python@3.10/bin/python3.10]:
 
Found possible Python library paths:
  /opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Please input the desired Python library path to use.  Default is [/opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages]
 
# 是否启用 ROCm 计算平台支持
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
 
# 是否启用 CUDA 支持
Do you wish to build TensorFlow with CUDA support? [y/N]: N
No CUDA support will be enabled for TensorFlow.
 
# 是否使用新版本 Clang 编译器
Do you wish to download a fresh release of clang? (Experimental) [y/N]: N
Clang will not be downloaded.
 
# 其他编译选项
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:
 
# 是否启用 Android 支持
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N
Not configuring the WORKSPACE for Android builds.
 
# 是否启用 iOS 支持
Do you wish to build TensorFlow with iOS support? [y/N]: N
No iOS support will be enabled for TensorFlow.
 
# 其余支持编译时传入的参数
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
    --config=mkl            # Build with MKL support.
    --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
    --config=monolithic     # Config for mostly static monolithic build.
    --config=numa           # Build with NUMA support.
    --config=dynamic_kernels    # (Experimental) Build kernels into separate shared objects.
    --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
    --config=nogcp          # Disable GCP support.
    --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
 
# 编译 C Api libtensorflow.so
~/Downloads/tensorflow-2.10.0 % bazel build --config=monolithic -c opt //tensorflow:libtensorflow.so
 
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=209
...
INFO: Analyzed target //tensorflow:libtensorflow.so (269 packages loaded, 21402 targets configured).
INFO: Found 1 target...
Target //tensorflow:libtensorflow.so up-to-date:
  bazel-bin/tensorflow/libtensorflow.so
INFO: Elapsed time: 29.067s, Critical Path: 6.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
 
# 编译 C++ Api libtensorflow_cc.so
~/Downloads/tensorflow-2.10.0 % bazel build --config=monolithic -c opt //tensorflow:libtensorflow_cc.so
 
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=209
...
INFO: Analyzed target //tensorflow:libtensorflow_cc.so (1 packages loaded, 53 targets configured).
INFO: Found 1 target...
Target //tensorflow:libtensorflow_cc.so up-to-date:
  bazel-bin/tensorflow/libtensorflow_cc.so
INFO: Elapsed time: 6.587s, Critical Path: 5.69s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

项目构建

项目构建需要将 Bazel 编译后的头文件和动态链接库加载到项目构建中，相关文件路径可以参考 Bazel 编译完成后输出的 Target 路径。C++ API 编译同时需要拷贝其余第三方依赖库的 Header 文件。包括但不限于 Eigen, absl, protobuf。

查看模型详情

# guesslang DNN 模型
# https://github.com/yoeo/guesslang/tree/master/guesslang/data/model
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model
The given SavedModel contains the following tag-sets:
'serve'
 
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model --tag_set serve
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "classification"
SignatureDef key: "predict"
SignatureDef key: "serving_default"
 
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['inputs'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: Placeholder:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['classes'] tensor_info:
      dtype: DT_STRING
      shape: (-1, 54)
      name: head/Tile:0
  outputs['scores'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 54)
      name: head/predictions/probabilities:0
Method name is: tensorflow/serving/classify
 
 
# ResNet 模型
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain
The given SavedModel contains the following tag-sets:
'serve'
 
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain/ --tag_set serve
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default"
 
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain/ --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['resnet50_input'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 3)
      name: serving_default_resnet50_input:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['dense_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 5)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

C / C++ 模型加载和预测

上述流程中获取到了模型相关的详情信息，下面将以 guesslang 模型为例，看看 C / C++ 中如何对 Tensorflow 模型进行加载和预测。

C 版本

TensorFlow provides a C API that can be used to build bindings for other languages. The API is defined in c_api.h and designed for simplicity and uniformity rather than convenience.

从 Tensorflow C API 的定位来说，它更偏向底层，因此代码看起来会更加复杂。

#include <stdio.h>
#include <tensorflow/c/c_api.h>
 
void NoOpDeallocator(void* data, size_t a, void* b) {}
 
int main() {
    const char* model_dir = "/path/to/model/dir"
    TF_Status* status = TF_NewStatus();
    TF_Graph* tfGraph = TF_NewGraph();
    TF_Buffer* run_opts = NULL;
    TF_Buffer* meta_g = NULL;
    TF_SessionOptions* opts = TF_NewSessionOptions();
    TF_Output *inputs = (TF_Output *) malloc(sizeof(TF_Output) * ninputs);
    TF_Output *outputs = (TF_Output *) malloc(sizeof(TF_Output) * noutputs);
    TF_Tensor **input_values = (TF_Tensor **) malloc(sizeof(TF_Tensor *) * ninputs);
    TF_Tensor **output_values = (TF_Tensor **) malloc(sizeof(TF_Tensor *) * noutputs);
     
    // 此处 tag 对应模型信息中的 tag-sets
    int ntag = 1;
    const char* tags = "serve";
     
    // 加载训练好的模型
    TF_Session* sess = TF_LoadSessionFromSavedModel(
        opts, run_opts,
        model_dir, &tags, ntag,
        tfGraph, meta_g,
        status
    );
     
    if (TF_GetCode(status) != TF_OK) {
        printf("%s\n", TF_Message(status));
        goto out;
    }
    TF_DeleteStatus(status);
     
    int ninputs = 1;
    int noutputs = 2;
 
    // 构建输入输出
    // 输入输出的 Operation Name 对应 SignatureDef 中输入输出 Tensor 描述中的名称
    TF_Output input = {TF_GraphOperationByName(tfGraph, "Placeholder"), 0};
    if(NULL == input.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
    TF_Output classes = {TF_GraphOperationByName(tfGraph, "head/Tile"), 0};
    if(NULL == classes.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
    TF_Output score = {TF_GraphOperationByName(tfGraph, "head/predictions/probabilities"), 0};
    if(NULL == score.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
 
    const char* data = "package main\nimport (\n\"fmt\"\n)\nfunc main() {\nfmt.Println(\"Hello Go!\")}";
    size_t size = strlen(data);
 
    // 模型 Input 为 DT_STRING
    // 实现构建 TF_TSTRING 类型变量作为构建 Tensor 入参
    TF_TString tstr[1];
    TF_TString_Init(&tstr[0]);
    TF_TString_Copy(&tstr[0], data, size);
 
    // 此处为 Tensor 的 Shape 参数
    // 对应 Tensor 的维度信息
    // dim - 0 常量
    // dim - 1 向量
    // dim - 2 矩阵
    // ... (重要)
    int ndims = 1;
    int64_t dims[] = {1};
 
    // 构建输入 Tensor，必须根据模型入参传入对应类型的 Tensor 和参数
    TF_Tensor* int_tensor = TF_NewTensor(TF_STRING, dims, ndims, &tstr[0], sizeof(tstr), &NoOpDeallocator, NULL);
    if (NULL == int_tensor) {
        printf("construction of input tensor failed\n");
        goto out;
    }
 
    // 出入参写入对应指针指向的内存
    inputs[0] = input;
    outputs[0] = classes;
    outputs[1] = score;
    input_values[0] = int_tensor;
 
    TF_Status* status = TF_NewStatus();
     
    // 执行预测
    TF_SessionRun(
        sess,
        NULL,
        inputs, input_values, ninputs,
        outputs, output_values, noutputs,
        NULL, 0,
        NULL,
        status
    );
    if (TF_GetCode(status) != TF_OK) {
        printf("%s\n", TF_Message(status));
        goto out;
    }
    TF_DeleteStatus(status);
 
    // 输出结果数量也是根据模型 output shape 得出
    size_t num_class = 54;
     
    // 结果解析
    void* out_classes = TF_TensorData(output_values[0]);
    TF_TString* res_classes = (TF_TString *)out_classes;
    void* out_scores = TF_TensorData(output_values[1]);
    float* res_scores = (float *)out_scores;
 
    const char * tag;
    float max_score = 0;
    for (int i = 0; i < num_class; i++) {
        if(res_scores[i]>max_score){
            max_score = res_scores[i];
            tag = TF_TString_GetDataPointer(&res_classes[i]);
        }
    }
 
    //释放资源
out:
    if (NULL != int_tensor) {
        TF_DeleteTensor(int_tensor);
    }
     
    if (NULL != tstr[0]) {
        TF_TString_Dealloc(&tstr[0]);
    }
     
    if (NULL != tfGraph) {
        TF_DeleteGraph(tfGraph);
    }
     
    if (NULL != sess) {
        TF_Status* status = TF_NewStatus();
        TF_DeleteSession(sess, status);
        TF_DeleteStatus(status);
    }
 
    free(inputs);
    free(outputs);
    free(input_values);
    free(output_values);
}

C++ 版本

include <iostream>
 
#include "tensorflow/cc/client/client_session.h"
#include "tensorflow/cc/ops/standard_ops.h"
#include "tensorflow/core/framework/tensor.h"
#include "tensorflow/cc/saved_model/loader.h"
#include "tensorflow/cc/saved_model/tag_constants.h"
#include "tensorflow/core/framework/logging.h"
 
using namespace tensorflow;
using namespace tensorflow::ops;
 
int main() {
    const char* model_dir = "/path/to/model/dir"
     
    // 模型加载
    auto* loadModel = new SavedModelBundleLite();
    SessionOptions sessionOptions;
    RunOptions runOptions;
 
    Status loadStatus = LoadSavedModel(
        sessionOptions,
        runOptions,
        model_dir,
        { kSavedModelTagServe },
        loadModel
    );
 
    TF_CHECK_OK(loadStatus);
     
    const char* data = "package main\nimport (\n\"fmt\"\n)\nfunc main() {\nfmt.Println(\"Hello Go!\")}";
     
    // 构建 Tensor
    Tensor input_tensor(data);
 
    // 构建 Input Vector，绑定 Tensor
    std::vector<std::pair<std::string, tensorflow::Tensor>> feedInputs = { {"Placeholder:0", input_tensor} };
     
    // 创建 Output Vector
    std::vector<std::string> fetches = { "head/Tile:0", "head/predictions/probabilities:0" };
    std::vector<tensorflow::Tensor> outputs;
 
    // 执行预测
    auto status = loadModel->GetSession()->Run(feedInputs, fetches, {}, &outputs);
    TF_CHECK_OK(status);
 
    for (const auto& record : outputs) {
        LOG(INFO) << record.DebugString();
    }
     
    delete(loadModel)
}

C / C++ 部署 Tensorflow 模型

前言

环境

直接安装

编译

项目构建

查看模型详情

C / C++ 模型加载和预测

C 版本

C++ 版本

macOS 环境下内存泄露检测

PHP 源码中 similar_text 函数的具体实现

C / C++ 部署 Tensorflow 模型

前言

环境

Tensorflow Share Library

直接安装

编译

项目构建

查看模型详情

C / C++ 模型加载和预测

C 版本

C++ 版本

macOS 环境下内存泄露检测

PHP 源码中 similar_text 函数的具体实现