[Swig-devel] SWIG Sequences as Lists in Python

[Nicholas Cox]

Hi,

I'm new to this list and signed up because I've been working on
incorporating SWIG into our build system in the robots group at Otherlab.
My background is mostly in control systems and robot software.

Part of what I've been working on is extending the typemaps that come with
SWIG for use in wrapping code for Python to create more general OUTPUT
typemaps (other than the ones for primitive C types defined in
python/typemaps.i. This led me down the road of doing something like this
in an interface file (which I called argout_maps.i):

%define ARGOUT_PTR_MAP(type)
%typemap(in, numinputs=0) type* OUT (type temp) {
$1 = &temp;
}
%typemap(argout) type* OUT {
PyObject* my_out;
my_out = SWIG_NewPointerObj((new $*1_ltype(static_cast< const $*1_ltype&

(*$1))),

$1_descriptor,
SWIG_POINTER_OWN | 0);
$result = SWIG_Python_AppendOutput($result, my_out);
}
%enddef

(I suspect that in the above I should be doing some check on (*$1) to make
sure that it is still argout map expects before sending it to
SWIG_NewPointerObj)

I also wanted to combine this with handling std::vector<type> in a similar
way. The std_vector.i file seems to provide a good amount of
functionality, but I couldn't find anything on creating OUTPUT typemaps for
it, so I sketched a quick typemap together to get something like what I
wanted.

%typemap(in, numinputs=0) std::vector<double> *OUT (std::vector<double>
temp) {
$1 = &temp;
}
%typemap(argout) std::vector<double> *OUT {
PyObject* my_out$argnum;
my_out$argnum = swig::from(std::vector<double>(*$1));
$result = SWIG_Python_AppendOutput($result, my_out$argnum);
}

But, I noticed that there is some behavior that seemed somewhat strange to
me. It appears that by default, swig wrapped functions returns lists,
instead of tuples (for which there is a #define SWIG_PYTHON_OUTPUT_TUPLE
flag available to change.) But when this flag is used in combination with
outputting multiple std::vector<> returns, the behavior can be poor. The
returned tuples merge together strangely because std::vectors get mapped to
sequences, which are mapped to tuples. So an output my look like:

(1, 2, 3, 4 (1, 2, 3, 4))

when two lists ((1, 2, 3, 4), (1, 2, 3, 4)) would be the desired return.
Specifically, the SWIG_PYTHON_OUTPUT_TUPLE flag doesn't play well with the
std::vector<> wrapping. One way I could think of to solve this is to allow
an option for SWIG to wrap std::vectors as Python lists instead of Python
tuples. The place that seemed obvious to me to allow this is in
python/pycontainer.swg, by modifying the template to have a similar
compiler #if/#else statement. ie.:

template <class Seq, class T = typename Seq::value_type >
struct traits_from_stdseq {
typedef Seq sequence;
typedef T value_type;
typedef typename Seq::size_type size_type;
typedef typename sequence::const_iterator const_iterator;

static PyObject *from(const sequence& seq) {
%#ifdef SWIG_PYTHON_EXTRA_NATIVE_CONTAINERS
swig_type_info *desc = swig::type_info<sequence>();
if (desc && desc->clientdata) {
return SWIG_NewPointerObj(new sequence(seq), desc, SWIG_POINTER_OWN);
}
%#endif
size_type size = seq.size();
if (size <= (size_type)INT_MAX) {
%#ifdef SWIG_PYTHON_SEQUENCE_AS_LIST
PyObject *obj = PyList_New((Py_ssize_t)size);
Py_ssize_t i = 0;
for (const_iterator it = seq.begin(); it != seq.end(); ++it, ++i) {
PyList_SetItem(obj,i,swig::from<value_type>(*it));
}
%#else
PyObject *obj = PyTuple_New((Py_ssize_t)size);
Py_ssize_t i = 0;
for (const_iterator it = seq.begin(); it != seq.end(); ++it, ++i) {
PyTuple_SetItem(obj,i,swig::from<value_type>(*it));
}
%#endif
return obj;
} else {
PyErr_SetString(PyExc_OverflowError,"sequence size not valid in python");
return NULL;
}
}
};
}
}


Modifying the code like this appears to work and provide the expected
results. Is there a way in which a modification like this to
pycontainer.swg might break some other behavior of SWIG that I am not
seeing? Would this be a useful extension to pycontainer.swg?

I can submit code to show an example of the usage, but I felt that this
email was already code heavy, and I don't know if this is the appropriate
place for that :-)

Cheers,
Nick

[William S Fulton]

Post by Nicholas Cox
Hi,
I'm new to this list and signed up because I've been working on
incorporating SWIG into our build system in the robots group at Otherlab.
My background is mostly in control systems and robot software.
Part of what I've been working on is extending the typemaps that come with
SWIG for use in wrapping code for Python to create more general OUTPUT
typemaps (other than the ones for primitive C types defined in
python/typemaps.i. This led me down the road of doing something like this
%define ARGOUT_PTR_MAP(type)
%typemap(in, numinputs=0) type* OUT (type temp) {
$1 = &temp;
}
%typemap(argout) type* OUT {
PyObject* my_out;
my_out = SWIG_NewPointerObj((new $*1_ltype(static_cast< const $*1_ltype&

(*$1))),

$1_descriptor,
SWIG_POINTER_OWN | 0);
$result = SWIG_Python_AppendOutput($result, my_out);
}
%enddef
(I suspect that in the above I should be doing some check on (*$1) to make
sure that it is still argout map expects before sending it to
SWIG_NewPointerObj)
I also wanted to combine this with handling std::vector<type> in a similar
way. The std_vector.i file seems to provide a good amount of
functionality, but I couldn't find anything on creating OUTPUT typemaps for
it, so I sketched a quick typemap together to get something like what I
wanted.
%typemap(in, numinputs=0) std::vector<double> *OUT (std::vector<double>
temp) {
$1 = &temp;
}
%typemap(argout) std::vector<double> *OUT {
PyObject* my_out$argnum;
my_out$argnum = swig::from(std::vector<double>(*$1));
$result = SWIG_Python_AppendOutput($result, my_out$argnum);
}
But, I noticed that there is some behavior that seemed somewhat strange to
me. It appears that by default, swig wrapped functions returns lists,
instead of tuples (for which there is a #define SWIG_PYTHON_OUTPUT_TUPLE
flag available to change.) But when this flag is used in combination with
outputting multiple std::vector<> returns, the behavior can be poor. The
returned tuples merge together strangely because std::vectors get mapped to
(1, 2, 3, 4 (1, 2, 3, 4))
when two lists ((1, 2, 3, 4), (1, 2, 3, 4)) would be the desired return.
Specifically, the SWIG_PYTHON_OUTPUT_TUPLE flag doesn't play well with the
std::vector<> wrapping. One way I could think of to solve this is to allow
an option for SWIG to wrap std::vectors as Python lists instead of Python
tuples. The place that seemed obvious to me to allow this is in
python/pycontainer.swg, by modifying the template to have a similar
template <class Seq, class T = typename Seq::value_type >
struct traits_from_stdseq {
typedef Seq sequence;
typedef T value_type;
typedef typename Seq::size_type size_type;
typedef typename sequence::const_iterator const_iterator;
static PyObject *from(const sequence& seq) {
%#ifdef SWIG_PYTHON_EXTRA_NATIVE_CONTAINERS
swig_type_info *desc = swig::type_info<sequence>();
if (desc && desc->clientdata) {
return SWIG_NewPointerObj(new sequence(seq), desc, SWIG_POINTER_OWN);
}
%#endif
size_type size = seq.size();
if (size <= (size_type)INT_MAX) {
%#ifdef SWIG_PYTHON_SEQUENCE_AS_LIST
PyObject *obj = PyList_New((Py_ssize_t)size);
Py_ssize_t i = 0;
for (const_iterator it = seq.begin(); it != seq.end(); ++it, ++i) {
PyList_SetItem(obj,i,swig::from<value_type>(*it));
}
%#else
PyObject *obj = PyTuple_New((Py_ssize_t)size);
Py_ssize_t i = 0;
for (const_iterator it = seq.begin(); it != seq.end(); ++it, ++i) {
PyTuple_SetItem(obj,i,swig::from<value_type>(*it));
}
%#endif
return obj;
} else {
PyErr_SetString(PyExc_OverflowError,"sequence size not valid in python");
return NULL;
}
}
};
}
}
Modifying the code like this appears to work and provide the expected
results. Is there a way in which a modification like this to
pycontainer.swg might break some other behavior of SWIG that I am not
seeing? Would this be a useful extension to pycontainer.swg?
I can submit code to show an example of the usage, but I felt that this
email was already code heavy, and I don't know if this is the appropriate
place for that :-)

Sorry for late response Nick. A patch and additional tests are okay by me,
just submit it all to Github.

I wonder though if template specialization would allow you to completely
customize the code. I've found it frustratingly hard to get specializations
of the STL wrapper templates working though whenever I try due to
complications in getting the specializations generated in the correct part
of the wrapper code, so am not going to experiment. Just a thought though.

William

[Vadim Zeitlin]

On Fri, 25 Nov 2016 20:47:17 +0000 William S Fulton <***@fultondesigns.co.uk> wrote:

WSF> On 25 June 2016 at 21:37, Nicholas Cox <***@gmail.com> wrote:
...
WSF> > Specifically, the SWIG_PYTHON_OUTPUT_TUPLE flag doesn't play well with the
WSF> > std::vector<> wrapping. One way I could think of to solve this is to allow
WSF> > an option for SWIG to wrap std::vectors as Python lists instead of Python
WSF> > tuples. The place that seemed obvious to me to allow this is in
WSF> > python/pycontainer.swg, by modifying the template to have a similar
WSF> > compiler #if/#else statement. ie.:
...
WSF> Sorry for late response Nick. A patch and additional tests are okay by me,
WSF> just submit it all to Github.

Sorry for being late to this discussion, but I have strong reservations
about the wisdom of adding yet another compile-time option to the
container/vector Python typemaps. They're already terribly confusing IMO,
and the -extranative option (a.k.a SWIG_PYTHON_EXTRA_NATIVE_CONTAINERS) is
very hard to discover, yet crucial for returning objects of types for which
their own custom typemaps are defined (see my message, which unfortunately
seems to have gone unnoticed, about it at
https://sourceforge.net/p/swig/mailman/message/35280877/) and another
option, SWIG_NO_EXPORT_ITERATOR_METHODS, which is currently needed to make
iterating over vectors of such objects work (see this question at SO
http://stackoverflow.com/q/22026035/15275 and my answer to it), is even
less discoverable.

I think ideal would be to get rid of the existing options and just
always return a C++ object representing the sequence (i.e. make the "extra
native" case the default and only one). And perhaps this would fix
Nicholas' original problem too?

Regards,
VZ