[Swig-devel] Can we take the decision about simplifying Python generation options?

[Vadim Zeitlin]

Hello,

This has been discussed several times in the past, but nothing has been
really decided, AFAIK, so I'd like to return to this again in the hope that
we finally do something about the problem of having too many Python build
variants.

Right now we have the following major options:

- builtin
- classic
- modern

without speaking about minor ones such as -extranative or a bunch of
-fastxxx.

According to the documentation, -builtin provides a significant
performance improvement and the limitations listed there don't seem
significant at all, so I propose making -builtin the default and only way
of generating wrappers.

It's less clear which benefits does -modern bring as the documentation,
but its only requirement seems to be Python 2.2, so it seems perfectly
reasonable to turn it on by default and remove the option. Similarly,
-classic is only useful for pre-2.2, apparently, so let's drop it and
forget it ever existed.

Could we please agree to do this? I'd do my best to make the changes doing
this that should help a lot with comprehensibility of the Python module
code and facilitate further changes in the future as currently it's just
too annoying to play whack-a-mole with all these options (which is a game
when you do some change working with default options, run the unit test
that finds a failure with some non default combination, fix this failure,
but this now results in a failure of another test for another options
combinations and after fixing this one, things don't work with the default
options again).

Thanks in advance,
VZ

[William S Fulton]

Post by Vadim Zeitlin
Hello,
This has been discussed several times in the past, but nothing has been
really decided, AFAIK, so I'd like to return to this again in the hope that
we finally do something about the problem of having too many Python build
variants.
- builtin
- classic
- modern
without speaking about minor ones such as -extranative or a bunch of
-fastxxx.
According to the documentation, -builtin provides a significant
performance improvement and the limitations listed there don't seem
significant at all, so I propose making -builtin the default and only way
of generating wrappers.
It's less clear which benefits does -modern bring as the documentation,
but its only requirement seems to be Python 2.2, so it seems perfectly
reasonable to turn it on by default and remove the option. Similarly,
-classic is only useful for pre-2.2, apparently, so let's drop it and
forget it ever existed.
Agreed. All code prior to 2.6 should be dropped, as seems to be the

conclusion on https://github.com/swig/swig/issues/701.

Post by Vadim Zeitlin
Could we please agree to do this? I'd do my best to make the changes doing
this that should help a lot with comprehensibility of the Python module
code and facilitate further changes in the future as currently it's just
too annoying to play whack-a-mole with all these options (which is a game
when you do some change working with default options, run the unit test
that finds a failure with some non default combination, fix this failure,
but this now results in a failure of another test for another options
combinations and after fixing this one, things don't work with the default
options again).

Yes we should. The page at
https://github.com/swig/swig/wiki/SWIG-3.1-Development was started to
discuss which options to remove and which should be the default. I suggest
we keep this up to date with any suggestions and it will form the
documentation/release notes too.

I don't think -builtin should be the default though. I've been using it
recently on a large code base that has previously used non-builtin for some
time. I've been fixing builtin bugs for swig-3.0.11, but there are still
more that need fixing. There are also a number of features that don't work
with builtin. I'm not sure I have any great solutions for all of them. A
lot of customisations needs to be implemented differently with -builtin and
so in total I think the pain for users would be too much to make it the
default, and certainly not while it is not quite up to par. I think for
swig-3.1 non-builtin should remain the default.

I would like to see all the options that are used by -O turned on by
default and no way to turn them off. That will result in quite a long list
of removed options. I think that would keep someone busy for quite a while
removing the resulting dead code. How about putting it in a branch if you
want to work on this?

William

[Vadim Zeitlin]

On Fri, 25 Nov 2016 19:22:35 +0000 William S Fulton <***@fultondesigns.co.uk> wrote:

WSF> Yes we should. The page at
WSF> https://github.com/swig/swig/wiki/SWIG-3.1-Development was started to
WSF> discuss which options to remove and which should be the default. I suggest
WSF> we keep this up to date with any suggestions and it will form the
WSF> documentation/release notes too.

Ah, this is where it was. I looked for quite some time among the issues as
I completely forgot it was in the wiki instead, thanks for the reminder.

WSF> I don't think -builtin should be the default though.

I admit I don't have much experience with -builtin, but why not? Having
two ways to do something doesn't seem to be ideal, even if it's much better
than having 17 (or whatever, I lost count) different ways to do something
as is the case now. Using -builtin is supposed to bring clear performance
improvements, so we need to keep it for people for whom performance is the
top concern, but what is the reason for keeping the Python proxy classes
variant?

WSF> There are also a number of features that don't work with builtin.

What are they? The ones mentioned in the documentation don't seem to be of
utmost importance, is there something else/more?

WSF> I'm not sure I have any great solutions for all of them. A lot of
WSF> customisations needs to be implemented differently with -builtin

Hmm, do you mean extending Python generation classes or is there even more
to it?

WSF> so in total I think the pain for users would be too much to make it
WSF> the default, and certainly not while it is not quite up to par. I
WSF> think for swig-3.1 non-builtin should remain the default.

I guess I could live with this, getting rid of modern/classic/etc would
already be a big gain. But the holy grail would still be having just one
version...

WSF> I would like to see all the options that are used by -O turned on by
WSF> default and no way to turn them off. That will result in quite a long list
WSF> of removed options. I think that would keep someone busy for quite a while
WSF> removing the resulting dead code. How about putting it in a branch if you
WSF> want to work on this?

My experience of working on long-lived SWIG branches hasn't been uniformly
positive so far, to put it mildly, so I'd prefer to make individual
small(er) PRs instead in the hope that they could be accepted faster. Maybe
we could fork a 3.0.x branch from master instead right now and start
preparing for 3.1 on master?

Thanks,
VZ

[William S Fulton]

Post by Vadim Zeitlin
On Fri, 25 Nov 2016 19:22:35 +0000 William S Fulton <
WSF> Yes we should. The page at
WSF> https://github.com/swig/swig/wiki/SWIG-3.1-Development was started to
WSF> discuss which options to remove and which should be the default. I suggest
WSF> we keep this up to date with any suggestions and it will form the
WSF> documentation/release notes too.
Ah, this is where it was. I looked for quite some time among the issues as
I completely forgot it was in the wiki instead, thanks for the reminder.
WSF> I don't think -builtin should be the default though.
I admit I don't have much experience with -builtin, but why not? Having
two ways to do something doesn't seem to be ideal, even if it's much better
than having 17 (or whatever, I lost count) different ways to do something
as is the case now. Using -builtin is supposed to bring clear performance
improvements, so we need to keep it for people for whom performance is the
top concern, but what is the reason for keeping the Python proxy classes
variant?

Python is the largest user base and while -builtin is not up to scratch I
think it would be crazy to piss off a huge portion of SWIG's users with
something substandard. A long term goal may be to have this on as default
for swig-3.2 perhaps once -builtin has fully matured?

Post by Vadim Zeitlin
WSF> There are also a number of features that don't work with builtin.
What are they? The ones mentioned in the documentation don't seem to be of
utmost importance, is there something else/more?

I've probably forgotten some, but native exceptions, %shadow, %pythoncode
don't work. Any special method, eg __str__ will be ignored unless a
suitable %feature("python:slot") is added. Also all builtin types are
convertible to int which is wrong. Builtin types are also immutable unlike
proxy classes.

Post by Vadim Zeitlin
WSF> I'm not sure I have any great solutions for all of them. A lot of
WSF> customisations needs to be implemented differently with -builtin
WSF> I would like to see all the options that are used by -O turned on by
WSF> default and no way to turn them off. That will result in quite a long list
WSF> of removed options. I think that would keep someone busy for quite a while
WSF> removing the resulting dead code. How about putting it in a branch if you
WSF> want to work on this?
My experience of working on long-lived SWIG branches hasn't been uniformly
positive so far, to put it mildly, so I'd prefer to make individual
small(er) PRs instead in the hope that they could be accepted faster. Maybe
we could fork a 3.0.x branch from master instead right now and start
preparing for 3.1 on master?
Yes great. This new version 3.1 isn't going to happen without a branch for

it. I'm afraid I'm not got too much time to focus on 3.1, but happy for
others to get going. Anything that is in
https://github.com/swig/swig/wiki/SWIG-3.1-Development ought to go into the
branch. so you may want to put the doxygen stuff in first as that
introduces a pile of code changes.

Small patches are great, but if they are at all risky or change default
behaviour then they won't be any good for the stable swig-3.0.x branch, but
will be suitable for 3.1. I'd say the main reason for not merging big long
lived branches is the amount of change is too great for a stable branch and
we don't have enough resources to focus on big new releases.

William

[Vadim Zeitlin]

On Fri, 25 Nov 2016 23:29:44 +0000 William S Fulton <***@fultondesigns.co.uk> wrote:

WSF> > WSF> There are also a number of features that don't work with builtin.
WSF> >
WSF> > What are they? The ones mentioned in the documentation don't seem to be of
WSF> > utmost importance, is there something else/more?
WSF> >
WSF> I've probably forgotten some, but native exceptions, %shadow, %pythoncode
WSF> don't work. Any special method, eg __str__ will be ignored unless a
WSF> suitable %feature("python:slot") is added. Also all builtin types are
WSF> convertible to int which is wrong. Builtin types are also immutable unlike
WSF> proxy classes.

OK, this does look like a lot of potential breakage. The documentation
probably should mention those instead of speaking about compatibility
problems with Python 2.[234] that nobody cares about any more...

WSF> > My experience of working on long-lived SWIG branches hasn't been uniformly
WSF> > positive so far, to put it mildly, so I'd prefer to make individual
WSF> > small(er) PRs instead in the hope that they could be accepted faster. Maybe
WSF> > we could fork a 3.0.x branch from master instead right now and start
WSF> > preparing for 3.1 on master?
WSF> >
WSF> Yes great. This new version 3.1 isn't going to happen without a branch for
WSF> it. I'm afraid I'm not got too much time to focus on 3.1, but happy for
WSF> others to get going. Anything that is in
WSF> https://github.com/swig/swig/wiki/SWIG-3.1-Development ought to go into the
WSF> branch. so you may want to put the doxygen stuff in first as that
WSF> introduces a pile of code changes.

I really don't want to predicate merging anything else on the merge of the
Doxygen branch. I have no idea when, or if, it's going to happen, so why
take the risk of not merging the other changes too?

WSF> Small patches are great, but if they are at all risky or change default
WSF> behaviour then they won't be any good for the stable swig-3.0.x branch, but
WSF> will be suitable for 3.1. I'd say the main reason for not merging big long
WSF> lived branches is the amount of change is too great for a stable branch and
WSF> we don't have enough resources to focus on big new releases.

But if this is so, what hope do we have for merging your proposed 3.1
branch which will *combine* even more changes? I was worried about merging
just the branch with Python options changes because I am very much afraid
that when it's ready to be merged in a few months time, there will be too
many changes in it to review. But reviewing it and Doxygen together is even
more impossible. Sorry for being direct, but do you realistically believe
in being able to do it yourself?

Why not create a 3.0 branch as I'm proposing for the future, stopgap 3.0.x
releases and start merging stuff into master, testing it and release once
it's in a good enough state? Like this it would allow to merge the pieces
in reasonably reviewable parts and sooner or later we will make a new
release from master.

Again, I realize that we all have limited time and I am not trying to
create more work for you (or anybody else). But knowing this, I just don't
see how would this putative 3.1 branch ever get merged. OTOH I do think
that releasing from master, after a period of stabilization, is feasible.

Do you still insist on creating a separate branch for all these changes?
VZ

[William S Fulton]

Post by Vadim Zeitlin
On Fri, 25 Nov 2016 23:29:44 +0000 William S Fulton <
WSF> > My experience of working on long-lived SWIG branches hasn't been uniformly
WSF> > positive so far, to put it mildly, so I'd prefer to make individual
WSF> > small(er) PRs instead in the hope that they could be accepted faster. Maybe
WSF> > we could fork a 3.0.x branch from master instead right now and start
WSF> > preparing for 3.1 on master?
WSF> >
WSF> Yes great. This new version 3.1 isn't going to happen without a branch for
WSF> it. I'm afraid I'm not got too much time to focus on 3.1, but happy for
WSF> others to get going. Anything that is in
WSF> https://github.com/swig/swig/wiki/SWIG-3.1-Development ought to go into the
WSF> branch. so you may want to put the doxygen stuff in first as that
WSF> introduces a pile of code changes.
I really don't want to predicate merging anything else on the merge of the
Doxygen branch. I have no idea when, or if, it's going to happen, so why
take the risk of not merging the other changes too?
If I havn't made it clear enough elsewhere, then I'll make it clear now

that the doxygen branch is currently in a good enough state for inclusion
in 3.1. I don't see any risk of basing other changes on it.

Post by Vadim Zeitlin
WSF> Small patches are great, but if they are at all risky or change default
WSF> behaviour then they won't be any good for the stable swig-3.0.x branch, but
WSF> will be suitable for 3.1. I'd say the main reason for not merging big long
WSF> lived branches is the amount of change is too great for a stable branch and
WSF> we don't have enough resources to focus on big new releases.
But if this is so, what hope do we have for merging your proposed 3.1
branch which will *combine* even more changes? I was worried about merging
just the branch with Python options changes because I am very much afraid
that when it's ready to be merged in a few months time, there will be too
many changes in it to review. But reviewing it and Doxygen together is even
more impossible. Sorry for being direct, but do you realistically believe
in being able to do it yourself?
Why not create a 3.0 branch as I'm proposing for the future, stopgap 3.0.x
releases and start merging stuff into master, testing it and release once
it's in a good enough state? Like this it would allow to merge the pieces
in reasonably reviewable parts and sooner or later we will make a new
release from master.
Again, I realize that we all have limited time and I am not trying to
create more work for you (or anybody else). But knowing this, I just don't
see how would this putative 3.1 branch ever get merged. OTOH I do think
that releasing from master, after a period of stabilization, is feasible.
Do you still insist on creating a separate branch for all these changes?
Yes. It is all about priorities and emphasis given the limited number of

people willing to get involved. master is for the stable 3.0.x releases,
which I will continue to release every few months until 3.1 goes out. With
regards to priorities, I want the constant stream of numerous small bug
fixes and improvement patches to be pushed out asap. The reality of things
means it is best done with the Travis testing and a stable master and 3.0.x
is the only thing we have that resembles stable. When 3.1 stops being
vapourware and a real branch with the planned and discussed features
actually implemented, then, we can do as you suggest and switch master to
3.1.x releases. I'm keen on 3.1 happening soon, but we need active
participants, other than Vadim and myself, so anyone else who is willing to
help make this happen please let us know.

William

[Nikita Simonian]

William, Vadim,

Thank you both for your leads on the C++ to C++ swig module. At the moment
I do not have an immediate plan to jump in and write the module. My
curiosity was mostly about the complexity of such undertaking. Should there
be a justified necessity for this, you will hear more from me :-)

Best, Nikita

Post by William S Fulton

Post by Vadim Zeitlin
On Fri, 25 Nov 2016 23:29:44 +0000 William S Fulton <
WSF> > My experience of working on long-lived SWIG branches hasn't been uniformly
WSF> > positive so far, to put it mildly, so I'd prefer to make individual
WSF> > small(er) PRs instead in the hope that they could be accepted faster. Maybe
WSF> > we could fork a 3.0.x branch from master instead right now and start
WSF> > preparing for 3.1 on master?
WSF> >
WSF> Yes great. This new version 3.1 isn't going to happen without a branch for
WSF> it. I'm afraid I'm not got too much time to focus on 3.1, but happy for
WSF> others to get going. Anything that is in
WSF> https://github.com/swig/swig/wiki/SWIG-3.1-Development ought to go into the
WSF> branch. so you may want to put the doxygen stuff in first as that
WSF> introduces a pile of code changes.
I really don't want to predicate merging anything else on the merge of the
Doxygen branch. I have no idea when, or if, it's going to happen, so why
take the risk of not merging the other changes too?
If I havn't made it clear enough elsewhere, then I'll make it clear now

that the doxygen branch is currently in a good enough state for inclusion
in 3.1. I don't see any risk of basing other changes on it.

Post by Vadim Zeitlin
WSF> Small patches are great, but if they are at all risky or change default
WSF> behaviour then they won't be any good for the stable swig-3.0.x branch, but
WSF> will be suitable for 3.1. I'd say the main reason for not merging big long
WSF> lived branches is the amount of change is too great for a stable branch and
WSF> we don't have enough resources to focus on big new releases.
But if this is so, what hope do we have for merging your proposed 3.1
branch which will *combine* even more changes? I was worried about merging
just the branch with Python options changes because I am very much afraid
that when it's ready to be merged in a few months time, there will be too
many changes in it to review. But reviewing it and Doxygen together is even
more impossible. Sorry for being direct, but do you realistically believe
in being able to do it yourself?
Why not create a 3.0 branch as I'm proposing for the future, stopgap 3.0.x
releases and start merging stuff into master, testing it and release once
it's in a good enough state? Like this it would allow to merge the pieces
in reasonably reviewable parts and sooner or later we will make a new
release from master.
Again, I realize that we all have limited time and I am not trying to
create more work for you (or anybody else). But knowing this, I just don't
see how would this putative 3.1 branch ever get merged. OTOH I do think
that releasing from master, after a period of stabilization, is feasible.
Do you still insist on creating a separate branch for all these changes?
Yes. It is all about priorities and emphasis given the limited number of

people willing to get involved. master is for the stable 3.0.x releases,
which I will continue to release every few months until 3.1 goes out. With
regards to priorities, I want the constant stream of numerous small bug
fixes and improvement patches to be pushed out asap. The reality of things
means it is best done with the Travis testing and a stable master and 3.0.x
is the only thing we have that resembles stable. When 3.1 stops being
vapourware and a real branch with the planned and discussed features
actually implemented, then, we can do as you suggest and switch master to
3.1.x releases. I'm keen on 3.1 happening soon, but we need active
participants, other than Vadim and myself, so anyone else who is willing to
help make this happen please let us know.
William
------------------------------------------------------------
------------------
_______________________________________________
Swig-devel mailing list
https://lists.sourceforge.net/lists/listinfo/swig-devel