[Swig-devel] Make std::vector support python buffer protocol

[Jacky Tsao]

tHi,

I'm developing a memory structure for python based deeplearning
framework. I thought it will be so nice if it support Python buffer
protocol, like zero-copy its buffer to NUMPY for calculation.

So I ran a lot of options in SWIG currently, none quite fit my need.
So I came up with a hack way to get it done. But I think if done it
right, it will benefit more that it is currently, and I also want to
know if there is a better way to do it.

For convenient, I start using a class template inherited from
std::vector, named carray (I know, and I'll work on the name later).
Implemented a method called 'getbuffer(PyObject *self, Py_buffer
*view, int flags)', and I'm using '-builtin' compilation flag to make
sure SWIG generate a built-in type. All I need is the built-in type of
this class can dispatch 'getbufferproc' to it, then I can fill in the
protocol with vector's data(). Attached file is a preliminary concept
work.

I just new to SWIG and use it for like weeks, so I need opinions if
this is a feasible way to do this, or can this work be generalized to
help others.

[William S Fulton]

Post by Jacky Tsao
tHi,
I'm developing a memory structure for python based deeplearning
framework. I thought it will be so nice if it support Python buffer
protocol, like zero-copy its buffer to NUMPY for calculation.
So I ran a lot of options in SWIG currently, none quite fit my need.
So I came up with a hack way to get it done. But I think if done it
right, it will benefit more that it is currently, and I also want to
know if there is a better way to do it.
For convenient, I start using a class template inherited from
std::vector, named carray (I know, and I'll work on the name later).
Implemented a method called 'getbuffer(PyObject *self, Py_buffer
*view, int flags)', and I'm using '-builtin' compilation flag to make
sure SWIG generate a built-in type. All I need is the built-in type of
this class can dispatch 'getbufferproc' to it, then I can fill in the
protocol with vector's data(). Attached file is a preliminary concept
work.
I just new to SWIG and use it for like weeks, so I need opinions if
this is a feasible way to do this, or can this work be generalized to
help others.

Hi Jacky

I havn't tried running your implementation, but I can't see anything
obviously wrong with it. The Python support functionality (such as
getbuffer) could simply be global functions or added by %extend so that it
could work more generically (ie other STL containers).

Adding the Buffer protocol to the std::vector and possibly other STL
container wrappers would be a great addition. It would be great if you
could put port your changes to std::vector and submit as a patch. You may
want to use std::array from C++11 instead of crafting your own class. The
'format' field implementation would need some thinking about as this seems
to be specific to the underlying container.

Incidentally, there is a bug report about Numpy, Python 3 and std::vector
wrappers problems in https://github.com/swig/swig/issues/888. This is using
the default proxy wrapping approach (ie not -builtin).

The numpy.i file
https://docs.scipy.org/doc/numpy/reference/swig.interface-file.html is
meant to add Numpy support for C arrays, but this does not seem to have
anything for -builtin or any Buffer Protocol support. Not sure why.

William

[Jacky Tsao]

Thanks for the information and I'll look into it.

On Fri, May 12, 2017 at 2:53 AM, William S Fulton

Post by William S Fulton

Post by Jacky Tsao
tHi,
I'm developing a memory structure for python based deeplearning
framework. I thought it will be so nice if it support Python buffer
protocol, like zero-copy its buffer to NUMPY for calculation.
So I ran a lot of options in SWIG currently, none quite fit my need.
So I came up with a hack way to get it done. But I think if done it
right, it will benefit more that it is currently, and I also want to
know if there is a better way to do it.
For convenient, I start using a class template inherited from
std::vector, named carray (I know, and I'll work on the name later).
Implemented a method called 'getbuffer(PyObject *self, Py_buffer
*view, int flags)', and I'm using '-builtin' compilation flag to make
sure SWIG generate a built-in type. All I need is the built-in type of
this class can dispatch 'getbufferproc' to it, then I can fill in the
protocol with vector's data(). Attached file is a preliminary concept
work.
I just new to SWIG and use it for like weeks, so I need opinions if
this is a feasible way to do this, or can this work be generalized to
help others.

Hi Jacky
I havn't tried running your implementation, but I can't see anything
obviously wrong with it. The Python support functionality (such as
getbuffer) could simply be global functions or added by %extend so that it
could work more generically (ie other STL containers).
Adding the Buffer protocol to the std::vector and possibly other STL
container wrappers would be a great addition. It would be great if you could
put port your changes to std::vector and submit as a patch. You may want to
use std::array from C++11 instead of crafting your own class. The 'format'
field implementation would need some thinking about as this seems to be
specific to the underlying container.
Incidentally, there is a bug report about Numpy, Python 3 and std::vector
wrappers problems in https://github.com/swig/swig/issues/888. This is using
the default proxy wrapping approach (ie not -builtin).
The numpy.i file
https://docs.scipy.org/doc/numpy/reference/swig.interface-file.html is meant
to add Numpy support for C arrays, but this does not seem to have anything
for -builtin or any Buffer Protocol support. Not sure why.
William