python - 使用 boost::python 时将 python.io 对象转换为 std::istream

Question

在编写我的第一个 Django 应用程序时，我遇到了 boost::python 的以下问题。从 python 代码，我需要传递 io.BytesIO到采用 std::istream 的 C++ 类。

我有一个遗留的 C++ 库，用于读取特定格式的文件。让我们调用 somelib。这个库的接口(interface)使用 std::istream 作为输入。像这样:

class SomeReader
{
public:
    bool read_from_stream(std::istream&);
};

我想包装它，以便我可以通过以下方式使用 python 中的库:

reader = somelib.SomeReader()
print ">>Pyhton: reading from BytesIO"
buf = io.BytesIO("Hello Stack Overflow")
reader.read(buf)

我找到了如何为实际的 python 文件对象执行此操作。但不清楚如何为任意 file-like 对象执行此操作。这是我目前拥有的 python 绑定(bind)的定义:

using namespace boost::python;
namespace io = boost::iostreams;

struct SomeReaderWrap: SomeReader, wrapper<SomeReader>
{
    bool read(object &py_file)
    {
        if (PyFile_Check(py_file.ptr()))
        {
            FILE* handle = PyFile_AsFile(py_file.ptr());
            io::stream_buffer<io::file_descriptor_source> fpstream (fileno(handle), io::never_close_handle);
            std::istream in(&fpstream);
            return this->read_from_stream(in);
        }
        else
        {
            //
            // How do we implement this???
            //
            throw std::runtime_error("Not a file, have no idea how to read this!");
        }
    }
};


BOOST_PYTHON_MODULE(somelib)
{
    class_<SomeReaderWrap, boost::noncopyable>("SomeReader")
        .def("read", &SomeReaderWrap::read);
}

是否有或多或少的通用方法将 python IO 对象转换为 C++ 流？

提前谢谢你。

---

作为实验的结果，我创建了一个小的 github repo这说明了这个问题。

Answer 1

而不是转换 Python io.BytesIO对象，考虑实现 Boost.IOStreams 的模型 Source能够从 Python io.BytesIO 对象中读取的概念。这将允许构建一个 boost::iostreams::stream并可供 SomeReader::read_from_stream() 使用。

这tutorial演示如何创建和使用自定义 Boost.IOStream 源。总的来说，这个过程应该是相当简单的。只需要根据 io.BufferedIOBase.read() 实现 Source 概念的 read() 函数:

/// Type that implements the Boost.IOStream's Source concept for reading
/// data from a Python object supporting read(size).
class PythonInputDevice
  : public boost::iostreams::source // Use convenience class.
{
public:

  explicit
  PythonInputDevice(boost::python::object object)
    : object_(object)
  {}

  std::streamsize read(char_type* buffer, std::streamsize buffer_size) 
  {
    namespace python = boost::python;
    // Read data through the Python object's API.  The following is
    // is equivalent to:
    //   data = object_.read(buffer_size)
    boost::python::object py_data = object_.attr("read")(buffer_size);
    std::string data = python::extract<std::string>(py_data);

    // If the string is empty, then EOF has been reached.
    if (data.empty())
    {
      return -1; // Indicate end-of-sequence, per Source concept.
    }

    // Otherwise, copy data into the buffer.
    copy(data.begin(), data.end(), buffer);
    return data.size();
  }

private:
  boost::python::object object_;
};

然后使用源设备创建一个boost::iostreams::stream:

boost::iostreams::stream<PythonInputDevice> input(py_object);
SomeReader reader;
reader.read_from_stream(input);

由于 PythonInputDevice 是根据 object.read() 实现的，duck typing允许 PythonInputDevice 与任何支持具有相同前置条件和后置条件的 read() 方法的 Python 对象一起使用。这包括内置的 Python file对象，从而不再需要在 SomeReaderWrap::read() 中根据类型进行条件分支。

---

这是一个基于原始代码的完整的最小示例:

#include <algorithm> // std::copy
#include <iosfwd> // std::streamsize
#include <iostream>
#include <boost/python.hpp>
#include <boost/iostreams/concepts.hpp>  // boost::iostreams::source
#include <boost/iostreams/stream.hpp>

class SomeReader
{
public:
  bool read_from_stream(std::istream& input)
  {
    std::string content(std::istreambuf_iterator<char>(input.rdbuf()),
                        (std::istreambuf_iterator<char>()));
    std::cout << "SomeReader::read_from_stream(): " << content << std::endl;
    return true;      
  }
};

/// Type that implements a model of the Boost.IOStream's Source concept
/// for reading data from a Python object supporting:
///   data = object.read(size).
class PythonInputDevice
  : public boost::iostreams::source // Use convenience class.
{
public:

  explicit
  PythonInputDevice(boost::python::object object)
    : object_(object)
  {}

  std::streamsize read(char_type* buffer, std::streamsize buffer_size) 
  {
    namespace python = boost::python;
    // Read data through the Python object's API.  The following is
    // is equivalent to:
    //   data = object_.read(buffer_size)
    boost::python::object py_data = object_.attr("read")(buffer_size);
    std::string data = python::extract<std::string>(py_data);

    // If the string is empty, then EOF has been reached.
    if (data.empty())
    {
      return -1; // Indicate end-of-sequence, per Source concept.
    }

    // Otherwise, copy data into the buffer.
    copy(data.begin(), data.end(), buffer);
    return data.size();
  }

private:
  boost::python::object object_;
};

struct SomeReaderWrap
  : SomeReader,
    boost::python::wrapper<SomeReader>
{
  bool read(boost::python::object& object)
  {
    boost::iostreams::stream<PythonInputDevice> input(object);
    return this->read_from_stream(input);
  }
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  python::class_<SomeReaderWrap, boost::noncopyable>("SomeReader")
    .def("read", &SomeReaderWrap::read)
    ;
}

交互使用:

$ echo -n "test file" > test_file
$ python
>>> import example
>>> with open('test_file') as f:
...     reader = example.SomeReader()
...     reader.read(f)
... 
SomeReader::read_from_stream(): test file
True
>>> import io
>>> with io.BytesIO("Hello Stack Overflow") as f:
...     reaader = example.SomeReader()
...     reader.read(f)
... 
SomeReader::read_from_stream(): Hello Stack Overflow
True