python - Windows中的Simstring(python)安装

Question

我正在尝试通过https://github.com/Georgetown-IR-Lab/simstring在Windows中安装simstring python包装器。对于Linux，它工作正常，但对于Windows，它在安装时给我错误。

    D:\Users\source\repos>python setup.py install
    running install
    running build
    running build_py
    running build_ext
    building '_simstring' extension
    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I. -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /EHsc /Tpexport.cpp /Fobuild\temp.win-amd64-3.6\Release\export.obj
    export.cpp
    export.cpp(7): fatal error C1083: Cannot open include file: 'iconv.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.12.25827\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

之后，我在项目中包含了iconv.h。但是现在它显示了不同的错误。

running install
running build
running build_py
running build_ext
building '_simstring' extension
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I. -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /EHsc /Tpexport.cpp /Fobuild\temp.win-amd64-3.6\Release\export.obj
export.cpp
d:\users\aki\source\repos\simstring\cdbpp.h(101): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data
export.cpp(37): error C2664: 'size_t libiconv(libiconv_t,const char **,size_t *,char **,size_t *)': cannot convert argument 2 from 'char **' to 'const char **'
export.cpp(37): note: Conversion loses qualifiers
export.cpp(140): note: see reference to function template instantiation 'bool iconv_convert<std::string,std::wstring>(libiconv_t,const source_type &,destination_type &)' being compiled
        with
        [
            source_type=std::string,
            destination_type=std::wstring
        ]
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.12.25827\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

任何帮助或指导表示赞赏。

Answer 1

地面笔记:

我设法进行了构建过程，但是我一度陷入困境。我创建了[SO]: Compile error for (char based) STL (stream) containers in Visual Studio(在该问题上花费了很多时间)。我以某种方式工作了，但是在尝试构建simstring时出现了其他(类似？)错误，因此我不得不从

中剥离一些(基于Nix的)代码(未编译)。

simstring用C++编写。生成C++(C)代码时，结果为PE或可移植可执行文件(.exe，.dll)。检查[SO]: LNK2005 Error in CLR Windows Form (@CristiFati's answer)以获取有关如何转换代码的更多详细信息。处理依赖于(加载).dll的.exe时，存在某些限制:

.exe(在本例中为python.exe)的体系结构(32位对64位或(x86对x64(或AMD64)))必须与它加载的任何.dll(以及其他已加载的.dll加载，依此类推)，因此依赖关系树中的所有dll都将加载，否则.dll将不会加载

在某些情况下，平台(调试与发布)应该匹配。如果不这样做，可能会发生以下情况:[SO]: When using fstream in a library I get linker errors in the executable (@CristiFati's answer)，但我不认为我们处于这种情况

在某些(其他)情况下，构建工具也应匹配。例子:

编译器类型([SO]: Python extensions with C: staticforward (@CristiFati's answer))

CRT运行时([SO]: Errors when linking to protobuf 3 on MSVC 2013 (@CristiFati's answer))

在我们的案例中，CRT运行时版本很重要。检查[Python.Wiki]: WindowsCompilers以获得Python和VStudio版本之间的兼容性。请注意，这仅适用于已下载和安装的Python版本(如果您是从源代码构建Python的，则应使用相同的构建工具-但我想情况并非如此)

我看到您正在使用VStudio 2017，因此兼容版本为Python 3.5和Python 3.6 1 。我的机器上安装了约10个Python安装程序(有些安装，有些是由我构建的-使用不同的编译器；大多数是x64，我也有一些VEnv，但这没什么区别)。我还安装了5个VStudio版本，在我的情况下，setup.py自动选择VStudio 2015(但可以，因为VStudio 2017具有编译器 v14.0 )

simstring依赖于libiconv，后者也以.dll的形式出现(实际上还有更多，但我们只关心其中一个)。使用Dependency Walker检查.dll(请参阅下文)，发现它是x86 2 。这意味着:

应该使用Python 32位(x86)。这是我要使用的变体。从 1 和 2 ，我的机器上唯一可用的版本是Python 3.6 x86(Python 3.5是我选择的版本，我也有32bit格式，但是我搞砸了并且没有重新安装)

从源代码构建libiconv，并摆脱限制 2 。但是，这可能会花费一些时间，这超出了当前问题的范围。如果有关于构建它的问题，我将花一些时间尝试一下，因为我喜欢这种任务([SO]: How to build a DLL version of libjpeg 9b? (@CristiFati's answer))

演练:

创建一个目录并cd到它(应该为空)。这将是％ROOT_DIR％，我要使用的所有路径都是相对的(当然，绝对路径除外)，这将是默认目录(未指定时)

下载simstring源([GitHub]: Georgetown-IR-Lab/simstring - simstring-master.zip)

解压缩存档-它将在dir simstring-master中进行(将自动创建)

创建目录libiconv。在其中下载:

[SourceForge]: gnuwin32/GnuWin - libiconv-1.9.2-1-lib.zip

[SourceForge]: gnuwin32/GnuWin - libiconv-1.9.2-1-bin.zip

从这些文件中提取所需的内容:

来自＃1 。:

include dir-在编译阶段使用

lib dir-在链接阶段使用

这两个阶段均由setup.py(下)

执行

来自＃2 。:

bin目录-在运行时使用(使用(导入)模块时)

cd到simstring-master目录。要构建扩展，我使用setup.py的build_ext命令(通过安装递归调用-如您的输出所示):[Python 3]: distutils.command.build_ext - Build any extensions in a package

运行build_ext会产生错误:

export.cpp(7): fatal error C1083: Cannot open include file: 'iconv.h': No such file or directory

那是因为Python构建系统不知道我们做了什么(在libiconv目录中)。让它知道，传递:

-I (--include-dirs)-将被翻译为[MS.Docs]: /I (Additional include directories)

-L (--library-dirs)-将被翻译为[MS.Docs]: /LIBPATH (Additional Libpath)

-l (-库)-将翻译为[MS.Docs]: LINK Input Files

标志( python setup.py build_ext --help将显示所有标志)。现在，不要通过＃2。和＃3。因为我们不会进入链接阶段(需要它们):

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>"e:\Work\Dev\VEnvs\py36x86_test\Scripts\python.exe" setup.py build_ext -I"../libiconv/include"
running build_ext
building '_simstring' extension
C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\BIN\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I. -I../libiconv/include -Ic:\Install\x86\Python\Python\3.6\include -Ic:\Install\x86\Python\Python\3.6\include "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\INCLUDE" "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /EHsc /Tpexport.cpp /Fobuild\temp.win32-3.6\Release\export.obj
export.cpp
export.cpp(112): warning C4297: 'writer::~writer': function assumed not to throw an exception but does
export.cpp(112): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
export.cpp(126): warning C4297: 'writer::~writer': function assumed not to throw an exception but does
export.cpp(126): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
export.cpp(37): error C2664: 'size_t libiconv(libiconv_t,const char **,size_t *,char **,size_t *)': cannot convert argument 2 from 'char **' to 'const char **'
export.cpp(37): note: Conversion loses qualifiers
export.cpp(140): note: see reference to function template instantiation 'bool iconv_convert<std::basic_string<char,std::char_traits<char>,std::allocator<char>>,std::wstring>(libiconv_t,const source_type &,destination_type &)' being compiled
with
[
    source_type=std::basic_string<char,std::char_traits<char>,std::allocator<char>>,
    destination_type=std::wstring
]
error: command 'C:\\Install\\x86\\Microsoft\\Visual Studio Community\\2015\\VC\\BIN\\cl.exe' failed with exit status 2

要做的事情(发现错误是一个接一个地纠正，只需要export.cpp进行更改):

#define ICONV_CONST const(cl.exe不会自动转换常量)

#define __SIZEOF_WCHAR_T__ 2(因为sizeof(wchar_t)是 2 )

删除无法编译的代码(我在开始时谈到):具有4个字节字符的STL容器不在Win上编译，想要修复该代码，当Win将支持此类字符时，代码将编译OOTB，但是我却不能，所以我不得不做OSX所做的任何事情。因此，应将#ifdef __APPLE__替换为#if defined(__APPLE__) || defined(WIN32)(5次)

注意＃1。和＃2。可以(应该)通过cmdline( -D 标志，但是我无法为已定义标志指定值)或在setup.py中完成(因此即使它们需要被定义，它们也只能定义一次)在很多文件中声明)，但是我并没有花太多时间，所以我直接在源代码中替换了它们。

手动应用更改，或者保存:

--- export.cpp.orig 2016-11-30 18:53:32.000000000 +0200
+++ export.cpp  2018-02-14 13:36:31.317953200 +0200
@@ -19,9 +19,18 @@
 #endif/*USE_LIBICONV_GNU*/

 #ifndef ICONV_CONST
+#if defined (WIN32)
+#define ICONV_CONST const
+#else
 #define ICONV_CONST
+#endif
 #endif/*ICONV_CONST*/

+#if defined (WIN32)
+#define __SIZEOF_WCHAR_T__ 2
+#endif
+
+
 template <class source_type, class destination_type>
 bool iconv_convert(iconv_t cd, const source_type& src, destination_type& dst)
 {
@@ -269,7 +278,7 @@
     iconv_close(bwd);
 }

-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(WIN32)
 #include <cassert>
 #endif

@@ -283,7 +292,7 @@
         retrieve_thru(dbr, query, this->measure, this->threshold, std::back_inserter(ret));
         break;
     case 2:
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(WIN32)
 #if __SIZEOF_WCHAR_T__ == 2
         retrieve_iconv<wchar_t>(dbr, query, UTF16, this->measure, this->threshold, std::back_inserter(ret));
 #else
@@ -294,7 +303,7 @@
 #endif
         break;
     case 4:
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(WIN32)
 #if __SIZEOF_WCHAR_T__ == 4
         retrieve_iconv<wchar_t>(dbr, query, UTF32, this->measure, this->threshold, std::back_inserter(ret));
 #else
@@ -317,7 +326,7 @@
         std::string qstr = query;
         return dbr.check(qstr, translate_measure(this->measure), this->threshold);
     } else if (dbr.char_size() == 2) {
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(WIN32)
 #if __SIZEOF_WCHAR_T__ == 2
         std::basic_string<wchar_t> qstr;
 #else
@@ -333,7 +342,7 @@
         iconv_close(fwd);
         return dbr.check(qstr, translate_measure(this->measure), this->threshold);
     } else if (dbr.char_size() == 4) {
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(WIN32)
 #if __SIZEOF_WCHAR_T__ == 4
         std::basic_string<wchar_t> qstr;
 #else

作为simstring_win.diff。这是一个差异。有关如何在Win上应用修补程序的信息，请参见 [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (@CristiFati's answer)( 修补utrunner 部分)(基本上，每行以 开头的一个“+” 符号进入，而每行以 开头的一个行以“-” 符号退出) 。我正在使用Cygwin，顺便说一句。我还将此修补程序提交给 [GitHub]: Georgetown-IR-Lab/simstring - Support for Win，今天又将其提交给 ，即(180222)。

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>"c:\Install\x64\Cygwin\Cygwin\AllVers\bin\patch.exe" -i "../simstring_win.diff"
patching file export.cpp

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>rem Looking at export.cpp content, you'll notice the changes

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>"e:\Work\Dev\VEnvs\py36x86_test\Scripts\python.exe" setup.py build_ext  -I"../libiconv/include" -L"../libiconv/lib" -llibiconv
running build_ext
building '_simstring' extension
C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\BIN\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I. -I../libiconv/include -Ic:\Install\x86\Python\Python\3.6\include -Ic:\Install\x86\Python\Python\3.6\include "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\INCLUDE" "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /EHsc /Tpexport.cpp /Fobuild\temp.win32-3.6\Release\export.obj
export.cpp
export.cpp(121): warning C4297: 'writer::~writer': function assumed not to throw an exception but does
export.cpp(121): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
export.cpp(135): warning C4297: 'writer::~writer': function assumed not to throw an exception but does
export.cpp(135): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\BIN\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I. -I../libiconv/include -Ic:\Install\x86\Python\Python\3.6\include -Ic:\Install\x86\Python\Python\3.6\include "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\INCLUDE" "-IC:\Install\x86\Microsoft\Visual Studio Community\2015\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.16299.0\winrt" /EHsc /Tpexport_wrap.cpp /Fobuild\temp.win32-3.6\Release\export_wrap.obj
export_wrap.cpp
C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\BIN\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\Install\x86\Python\Python\3.6\Libs /LIBPATH:../libiconv/lib /LIBPATH:e:\Work\Dev\VEnvs\py36x86_test\libs /LIBPATH:e:\Work\Dev\VEnvs\py36x86_test\PCbuild\win32 "/LIBPATH:C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\LIB" "/LIBPATH:C:\Install\x86\Microsoft\Visual Studio Community\2015\VC\ATLMFC\LIB" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.16299.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.16299.0\um\x86" libiconv.lib /EXPORT:PyInit__simstring build\temp.win32-3.6\Release\export.obj build\temp.win32-3.6\Release\export_wrap.obj /OUT:build\lib.win32-3.6\_simstring.cp36-win32.pyd /IMPLIB:build\temp.win32-3.6\Release\_simstring.cp36-win32.lib
   Creating library build\temp.win32-3.6\Release\_simstring.cp36-win32.lib and object build\temp.win32-3.6\Release\_simstring.cp36-win32.exp
Generating code
Finished generating code

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>dir /b "build\lib.win32-3.6"
_simstring.cp36-win32.pyd

最后，它建立了。 .pyd只是一个.dll。这是在Dependency Walker中的样子:

让我们尝试看看是否可以使用它:

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>"e:\Work\Dev\VEnvs\py36x86_test\Scripts\python.exe" sample.py
Traceback (most recent call last):
  File "E:\Work\Dev\StackOverflow\q048528041\simstring-master\simstring.py", line 18, in swig_import_helper
    fp, pathname, description = imp.find_module('_simstring', [dirname(__file__)])
  File "e:\Work\Dev\VEnvs\py36x86_test\lib\imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_simstring'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sample.py", line 3, in <module>
    import simstring
  File "E:\Work\Dev\StackOverflow\q048528041\simstring-master\simstring.py", line 28, in <module>
    _simstring = swig_import_helper()
  File "E:\Work\Dev\StackOverflow\q048528041\simstring-master\simstring.py", line 20, in swig_import_helper
    import _simstring
ModuleNotFoundError: No module named '_simstring'

那是因为在导入simstring时，反过来又导入_simstring(.pyd)，Python找不到它。要解决此问题:

将.pyd路径添加到％PYTHONPATH％

如图所示，.pyd依赖libiconv2.dll，因此OS必须知道在哪里寻找它。最简单的方法是将其路径添加到％PATH％([MS.Docs]: Dynamic-Link Library Search Order)

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>set PYTHONPATH=%PYTHONPATH%;build\lib.win32-3.6

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>set PATH=%PATH%;..\libiconv\bin

(py36x86_test) E:\Work\Dev\StackOverflow\q048528041\simstring-master>"e:\Work\Dev\VEnvs\py36x86_test\Scripts\python.exe" sample.py
('Barack Hussein Obama II',)
('James Gordon Brown',)
()
('Barack Hussein Obama II',)

最后说明:

该模块有一些输出，它与Lnx(Ubtu)上的输出相同(我也在其中构建了它-在那里我没有遇到任何问题)，我不确定它在语义上是否正确

我没有运行setup.py的 install 命令(而且我不会)，我能想到的一件事可能出了问题(尽管我不确定会不会出现)，它没有复制/包括libiconv2 .dll放入pkg。如果是这样，您可能需要修改setup.py(更改应该很小)