[Swig-devel] swig modules in python3 subpackages

[m***@comcast.net]

I've been reading the recent archives of posts here and saw the
discussion of removing support for older versions of python in
swig-3.1.0. I think this area of thought overlaps a current problem
I'm having with swig and python3.

In python3 the way for a module (say a swig shadow file) to import
another module in the same package (the C wrappers generated by swig)
is:

# foo.py
from . import _foo

Here foo.py and _foo.so live in some subpackage. The
"from . import _foo" syntax works even with python3's new implicit
namespace packages (more on these later).

So, for just python3 (only) all that is needed is::

from . import _foo

And what swig spits out to support every version of python is
(-c++ -python -py3 -modern -O -naturalvar):

from sys import version_info
if version_info >= (3, 0, 0):
new_instancemethod = lambda func, inst, cls: _foo.SWIG_PyInstanceMethod_New(func)
else:
from new import instancemethod as new_instancemethod
if version_info >= (2, 6, 0):
def swig_import_helper():
from os.path import dirname
import imp
fp = None
try:
fp, pathname, description = imp.find_module('_foo', [dirname(__file__)])
except ImportError:
import _foo
return _foo
if fp is not None:
try:
_mod = imp.load_module('_foo', fp, pathname, description)
finally:
fp.close()
return _mod
_foo = swig_import_helper()
del swig_import_helper
else:
import _foo
del version_info


The above code generated by swig will not work with python's
implicit namespace packages:

https://www.python.org/dev/peps/pep-0420/

Implicit namespace packages seem great from a swig perspective
because they allow for portions of a package to be loaded from
different paths. So, for example the foo.py file could be under
/path1/pkg1/pkg2/foo.py and _foo.so could be under
/path2/pkg1/pkg2/_foo.so

Even more useful is that the foo.py file could be placed into a zip
file, frozen, or stored in some other way and the _foo.so could be on
some file system where the os can use it. With implicit namespace
packages all of these will work.

But not with the code swig is currently generating. The path
argument in find_module() forces a fail with implicit namespace
packages and the fallback to "import _foo" is looking for _foo.so in
the global namespace. My workaround works:

%pythonbegin %{
def impKludge(set=True):
import sys
if set:
from . import _foo
sys.modules['_foo'] = _foo
else:
del sys.modules['_foo']
impKludge()
%}

%pythoncode %{
impKludge(False)
del impKludge
%}

But this is ugly beyond description. So, I was going to try and
make a patch for swig that would cause it to simply spit out:

from . import _foo

if say -py3 and -modern were both used. Somehow I think there
should be a way to tell swig "Hey I'm using python3 and have no
interest at all in going back. Just generate code for python3".
Maybe there is already a way to do this that I'm missing? Otherwise,
I'm willing to help fix this issue in whatever way I can.

Mike

[William S Fulton]

Post by m***@comcast.net
I've been reading the recent archives of posts here and saw the
discussion of removing support for older versions of python in
swig-3.1.0. I think this area of thought overlaps a current problem
I'm having with swig and python3.
In python3 the way for a module (say a swig shadow file) to import
another module in the same package (the C wrappers generated by swig)
# foo.py
from . import _foo
Here foo.py and _foo.so live in some subpackage. The
"from . import _foo" syntax works even with python3's new implicit
namespace packages (more on these later).
from . import _foo
And what swig spits out to support every version of python is
from sys import version_info
new_instancemethod = lambda func, inst, cls: _foo.SWIG_PyInstanceMethod_New(func)
from new import instancemethod as new_instancemethod
from os.path import dirname
import imp
fp = None
fp, pathname, description = imp.find_module('_foo', [dirname(__file__)])
import _foo
return _foo
_mod = imp.load_module('_foo', fp, pathname, description)
fp.close()
return _mod
_foo = swig_import_helper()
del swig_import_helper
import _foo
del version_info
The above code generated by swig will not work with python's
https://www.python.org/dev/peps/pep-0420/
Implicit namespace packages seem great from a swig perspective
because they allow for portions of a package to be loaded from
different paths. So, for example the foo.py file could be under
/path1/pkg1/pkg2/foo.py and _foo.so could be under
/path2/pkg1/pkg2/_foo.so
Even more useful is that the foo.py file could be placed into a zip
file, frozen, or stored in some other way and the _foo.so could be on
some file system where the os can use it. With implicit namespace
packages all of these will work.
But not with the code swig is currently generating. The path
argument in find_module() forces a fail with implicit namespace
packages and the fallback to "import _foo" is looking for _foo.so in
%pythonbegin %{
import sys
from . import _foo
sys.modules['_foo'] = _foo
del sys.modules['_foo']
impKludge()
%}
%pythoncode %{
impKludge(False)
del impKludge
%}
But this is ugly beyond description. So, I was going to try and
from . import _foo
if say -py3 and -modern were both used. Somehow I think there
should be a way to tell swig "Hey I'm using python3 and have no
interest at all in going back. Just generate code for python3".
Maybe there is already a way to do this that I'm missing? Otherwise,
I'm willing to help fix this issue in whatever way I can.

Hi Mike,

Have you looked at the -relativeimport option to see if this fixes
what you are seeing? See
http://swig.org/Doc3.0/Python.html#Python_nn72. There are lots of
examples and tests for this under Examples/python/import_packages.

William

[m***@comcast.net]

[snip]

Post by William S Fulton
Have you looked at the -relativeimport option to see if this
fixes what you are seeing? See
http://swig.org/Doc3.0/Python.html#Python_nn72. There are lots
of examples and tests for this under
Examples/python/import_packages.

No, I'd not seen or tried -relativeimport. Thanks for pointing it
out! Unfortunately it does not do anything to un-break this problem.
I'll summarize the problem again briefly:

--- foo.i ---
%module(package="bar") foo
...

swig -outdir bar -Wall -c++ -python -py3 -modern -O -naturalvar \
-relativeimport foo.i

foo.py gets put into a zip file and _foo.so is left on the
filesytem. So, this is the configuration:

/some/path/stuff.zip
[inside of stuff.zip]
bar/
bar/foo.py

/some/real/fs/path/bar/_foo.so

PYTHONPATH="/some/path/stuff.zip:/usr/real/fs/path:$PYTHONPATH"

Python's zipimporter finds foo.py just fine with a "import
bar.foo". But the swig generated code found in
Source/Modules/python.cxx fails to load the _foo.so module. It looks
like none of the code concerning this in python.cxx pays any attention
to -relativeimport. And spits out this relevant block:

if version_info >= (2, 6, 0):
def swig_import_helper():
from os.path import dirname
import imp
fp = None
try:
fp, pathname, description = imp.find_module('_foo', [dirname(__file__)])
except ImportError:
import _foo
return _foo

Here find_module will fail because it is using dirname(__file__)
which was generated by zipimport and points inside the zip file. Even
it were to find the .so inside the zipfile loading it would of course
be impossible. Next "import _foo" is tried (and not "from . import
foo" or better yet "from bar import foo". This too fails because
_foo.so is in the package bar.

I think what needs to happen is to modify Source/Modules/python.cxx
to deal with this case. Right now it seems it is kinda assuming the
dirname() for the _foo.so will always match the shadow file (foo.py). In
python3 with implicit packages it should be the package name that
matches. Not the directory name. The package name is an abstraction
while the path name of __file__ is a concrete location and
incompatible with implicit namespace packages.

I'll go ahead and try to create a patch to do this. I just need to
figure out how to access the command line options from inside
python.cxx (I've not looked into this yet). Plus I'll need to figure
out how to create and integrate some tests.

Mike

[m***@comcast.net]

...

The more I look at this page:

https://docs.python.org/3/reference/import.html

The more I'm convinced that (for python3 at least) swig should be
loading the shared module with importlib and __package__ rather than
__file__. Note that the docs have a phrase "If __file__ is set...".
So, it is not required for __file__ to even be an attribute.

I think swig should be doing something along these lines (for
python3):


if version_info >= (3, 0, 0):
def swig_import_helper():
import importlib
if len(__package__):
return importlib.import_module('.'.join((__package__, "_foo")))
else:
return importlib.import_module("_foo")

I think this code would handle foo.py finding _foo.so no matter
which package you installed them in. Thus one could build the swig
wrappers and later install them into whatever python package you
wanted.

Mike

[William S Fulton]

Post by m***@comcast.net
...
https://docs.python.org/3/reference/import.html
The more I'm convinced that (for python3 at least) swig should be
loading the shared module with importlib and __package__ rather than
__file__. Note that the docs have a phrase "If __file__ is set...".
So, it is not required for __file__ to even be an attribute.
I think swig should be doing something along these lines (for
import importlib
return importlib.import_module('.'.join((__package__, "_foo")))
return importlib.import_module("_foo")
I think this code would handle foo.py finding _foo.so no matter
which package you installed them in. Thus one could build the swig
wrappers and later install them into whatever python package you
wanted.

If your patch is backwards compatible we can add include it into
3.0.9. Otherwise we can target 3.1.

The developer documentation starting point is
http://www.swig.org/Doc3.0/Extending.html#Extending and covers adding
tests. It shouldn't be hard to just dive straight into python.cxx
though for what you want to do.

William

[m***@comcast.net]

Post by William S Fulton
If your patch is backwards compatible we can add include it
into 3.0.9. Otherwise we can target 3.1.

I've included the "meat and taters" part of the patch below just in
case anyone is curious. In a nutshell it will have the python shadow
class use importlib.import_module() using the "dotted" package name
found in the __name__ attribute of the shadow class file. The use of
importlib to do this sorta thing seems to be the preferred way the
python devs do it in their test cases. The patch only uses importlib
for python 2.7 and newer (I don't have pythons older than this so I
just left those alon).

Post by William S Fulton
The developer documentation starting point is
http://www.swig.org/Doc3.0/Extending.html#Extending and covers
adding tests. It shouldn't be hard to just dive straight into
python.cxx though for what you want to do.

I will create an example/test. (No existing tests broke with this
patch). The new example/test will depend upon implicit namespace
packages. So, it will only apply to python3. I'll look in the above docs
to see if I can run the tests only for python3.

Finally, I'll see if I can create a proper pull requests. The
following is just for the curious.

Mike

[m***@comcast.net]

Post by William S Fulton
If your patch is backwards compatible we can add include it
into 3.0.9. Otherwise we can target 3.1.

I think I've figured out the whole github pull request thing. My
changes are here:

https://github.com/swig/swig/pull/612

The patch should be backwards compatible as it only changes the
import logic for python implementations 2.7 and newer.

I created an example/test under Examples/python/import_packages/namespace_pkg.
I used an example rather than something in the test-suite because I
needed to create some directories to hold packages and modules for the
test. It looked like the examples were the way to go. However, I'm
completely open to modifying anything and everything about the tests I
supplied. Lemme know if they need adjustments.

The example/tests only run under python3 as namespace packages were
added then. The patch to python.cxx is still valid for python's back
to 2.7 as it makes swig modules work with non-filesystem based loaders.

Mike