java - CaffeonSpark构建'src/main/java/caffe/Caffe.java需要caffe.proto错误

Question

我正在尝试让 CaffeOnSpark 在本地运行，并且我按照 CaffeOnSpark wiki 上的此过程进行操作:https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_local

前 4 项对我来说很好。但是，当我在第 5 步中进行 Make 构建时。我在协议(protocol)上收到此错误:

整个错误堆栈:

jiaqi@jiaqi-ThinkPad-L450:~/github/CaffeOnSpark$ make build
cd caffe-public; make proto; make -j4 -e distribute; cd ..
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Nothing to be done for 'proto'.
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/tools/caffe.bin
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<int, int>(int const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `main':
compute_image_mean.cpp:(.text.startup+0x168): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/compute_image_mean.bin' failed
make[1]: *** [.build_release/tools/compute_image_mean.bin] Error 1
make[1]: *** Waiting for unfinished jobs....
.build_release/tools/convert_imageset.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
convert_imageset.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/convert_imageset.o: In function `main':
convert_imageset.cpp:(.text.startup+0x347): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/convert_imageset.bin' failed
make[1]: *** [.build_release/tools/convert_imageset.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/upgrade_net_proto_binary.bin' failed
make[1]: *** [.build_release/tools/upgrade_net_proto_binary.bin] Error 1
.build_release/tools/caffe.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
caffe.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/caffe.o: In function `main':
caffe.cpp:(.text.startup+0x50): undefined reference to `google::SetVersionString(std::string const&)'
caffe.cpp:(.text.startup+0x83): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/caffe.bin' failed
make[1]: *** [.build_release/tools/caffe.bin] Error 1
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
export LD_LIBRARY_PATH="/home/y/lib64:/home/y/lib64/mkl/intel64:/home/jiaqi/github/CaffeOnSpark/caffe-public/distribute/lib:/home/jiaqi/github/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn -B package
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks

protoc:
     [exec] make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-distri'
     [exec] make[1]: Leaving directory '/home/jiaqi/github/CaffeOmake[1]: *** No rule to make target '../caffe-public/distribute/proto/caffe.proto', needed by 'src/main/java/caffe/Caffe.java'. nSpark/caffe-distri'
     [exec]  Stop.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] caffe .............................................. SUCCESS [  0.019 s]
[INFO] caffe-distri ....................................... FAILURE [  2.425 s]
[INFO] caffe-grid ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.144 s
[INFO] Finished at: 2016-05-13T12:33:40+08:00
[INFO] Final Memory: 9M/238M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (proto) on project caffe-distri: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec failonerror="true" dir="/home/jiaqi/github/CaffeOnSpark/caffe-distri" executable="make">... @ 5:97 in /home/jiaqi/github/CaffeOnSpark/caffe-distri/target/antrun/build-protoc.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
Makefile:9: recipe for target 'build' failed
make: *** [build] Error 1

protoc 似乎找不到它的规则。您遇到过这个问题吗？请帮帮我!

我的协议(protocol)版本是2.5.0。

Answer 1

我之前遇到过 CaffeOnSpark 的构建错误。我知道这篇文章很旧，但无论如何它可能会对某人有所帮助。

从日志中我认为您存在一些依赖性问题。 CaffeOnSpark 需要满足 Caffe 的所有依赖项。所以我想问的第一个问题是 Caffe 是否工作(如果已安装)。

1. 如果未安装 Caffe，您可以尝试安装 Caffe 并通过打开 python 并输入 import caffe 来验证其工作情况。如果 Caffe 不工作，请转到步骤 5。

2. 如果 caffe 正在工作，请验证您是否执行了下载 git 的确切命令:

   git 克隆 https://github.com/yahoo/CaffeOnSpark.git --recursive

3. 然后请检查 CaffeOnSpark/caffe-public 中的 Makefile.config。确保正确给出所有标志。检查完后，运行这些:

   cd CaffeOnSpark
使干净
进行构建

4. 如果您遇到此错误:

   make[1]: Leaving directory `/home/das/CaffeOnSpark/caffe-public'
   export LD_LIBRARY_PATH="/usr/local/cuda-7.5/lib64:/home/das/CaffeOnSpark/caffe-public/distribute/lib:/home/das/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn -B package -DskipTests
   /bin/sh: 1: mvn: not found
   make: *** [build] Error 127
   

   运行此命令安装maven:

   sudo apt-get install maven
   

5. 如果仍然出现错误，您可能需要重新安装/再次检查依赖项。以防万一，我会发布说明。请注意，其中许多实际上并不是必需的。我安装这些来设置我的深度学习机器:

   sudo apt-get update
   sudo apt-get install -y build-essential cmake g++ gfortran git pkg-config python-dev software-properties-common
   sudo apt-get install -y python-pip python-dev python-numpy python-scipy python-nose python-h5py python-skimage python-matplotlib python-pandas python-sklearn python-sympy
   sudo apt-get install --no-install-recommends libboost-all-dev
   sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev
   

   并从 Caffe 根文件夹安装要求:

   sudo pip install -r python/requirements.txt
   

6. 如果存在 BLAS 问题，请正确安装:

   git clone https://github.com/xianyi/OpenBLAS.git
   cd OpenBLAS
   make FC=gfortran -j 4
   sudo make PREFIX=/usr/local install
   

7. 请确保 .bashrc 文件的所有路径均正确。

   export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
   export CAFFE_ROOT=/home/das/Caffe
   export PYTHONPATH=$CAFFE_ROOT/python:$PYTHONPATH
   export CAFFE_ON_SPARK=/home/das/CaffeOnSpark
   export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
   export HADOOP_HOME=/usr/local/hadoop
   export SPARK_HOME=/usr/local/spark
   export LD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib
   

..如果仍然存在错误，请发布它以便我们进行调查。

谢谢。