[Swig-devel] Potential concern regarding swig 3.0.9 change

[m***@comcast.net]

Hi Mike, Apologies for contacting you like this, but I am still
not 100% sure swig is really to blame, but wanted to at least
share what I know.

It is also my guess that #621 is involved with what is going on
here. The change I made did pass all the existing swig tests
(including all of the swig python import and package tests). So, I
think there must be a case which is not covered.

I think the case is the following:

- The swig wrapper (core.i) was built to put it into the package
"internal".

- The _core.so in *not* placed in the internal package but is
instead in the global namespace.

The old swig code always looked for _core.so in the global namespace
(near as I can tell). But the new code looks for it in the package
swig was told to find it. I guess there is not a test for falling
back to look for the C wrapper in the global namespace.

I'm not sure if this is even a good idea or not.

To confirm this could you supply the locations of where core.py and
_core.so (along with any __init__.py files)? The contents of your
PYTHONPATH (or sys.path) factor into this as well.

After upgrading to swig 3.0.9 I appear to have swig wrapper
modules that can no longer be found with Python 2.7. If I
revert your commit 23d2eb555fc41230bf68953007b72c08982a5386 it
https://github.com/swig/swig/commit/23d2eb555fc41230bf68953007b72c08982a5386
It seems the swig wrappers have a SWIG_name that does not
include the package (despite it being specified in the
module). In my example I have a core.i that specifies a package
“internal” that generates core.py and core_wrap.cc. The latter
has the _core module, which at run-time cannot be found as
internal._core. The Python 2.6 code that was there used to fall
back on just importing _core (after failing find_module and
load_module). The Python 2.7 code you added tries to import
internal._core, and that does not exist.
I can either stick with the Python 2.7 code you added and
override SWIG_name in the generated code to include the
package, or I can bypass the Python 2.7 code and use the 2.6
code without issues.
Does any of this sound reasonable? A genuine bug?
Thanks,
Andreas
IMPORTANT NOTICE: The contents of this email and any
attachments are confidential and may also be privileged. If you
are not the intended recipient, please notify the sender
immediately and do not disclose the contents to any other
person, use it for any purpose, or store or copy the
information in any medium. Thank you.

--

[m***@comcast.net]

Hi Mike, I should add that our specific setup relies on static
linking (for ease-of-distribution reasons). Thus, there is no
_core.so file. The specific setup has worked fine for quite
some years, but with 3.0.9 we face the issue mentioned in my
previous mail. A brute force hack is to change the SWIG_name in
the generated swig wrapper to include the package name, but it
would be good if we could avoid this hackery.

In this case there is 1/2 of the swig generated module installed in
a package (the pure python .py part). And then the other 1/2 of the
module (the C/C++ part) is *not* installed in the same package. This
is the part that does not seem good to me.

It seems to make more sense that both parts of the module should be
in the same package. If the _foo.so part is statically linked into an
interpreter (and loaded via freeze or something) then whatever loads it
still needs to put it into the same package as the other 1/2. At
least that is what makes the most sense to me.

But I guess swig always worked the other way and fell back to (or
always) loaded the C part as a global module. So, I'm working on a
fix that falls back to this global search location. The namespace
package support should make things easier for folks who would like
both parts of the swig module to really be in the same package. So,
both ways of doing things will work.


Mike

Let me know if you need any further information.
Thanks,
Andreas

Hi Mike, Apologies for contacting you like this, but I am

still > not 100% sure swig is really to blame, but wanted to at
least > share what I know.
It is also my guess that #621 is involved with what is going on
here. The change I made did pass all the existing swig tests
(including all of the swig python import and package tests).
So, I think there must be a case which is not covered.
- The swig wrapper (core.i) was built to put it into the
package "internal".
- The _core.so in *not* placed in the internal package but is
instead in the global namespace.
The old swig code always looked for _core.so in the global
namespace (near as I can tell). But the new code looks for it
in the package swig was told to find it. I guess there is not
a test for falling back to look for the C wrapper in the global
namespace.
I'm not sure if this is even a good idea or not.
To confirm this could you supply the locations of where core.py
and _core.so (along with any __init__.py files)? The contents
of your PYTHONPATH (or sys.path) factor into this as well.

After upgrading to swig 3.0.9 I appear to have swig wrapper >

modules that can no longer be found with Python 2.7. If I >
revert your commit 23d2eb555fc41230bf68953007b72c08982a5386 it
https://github.com/swig/swig/commit/23d2eb555fc41230bf68953007b72c08982a53
86

It seems the swig wrappers have a SWIG_name that does not >

include the package (despite it being specified in the >
module). In my example I have a core.i that specifies a package

“internal” that generates core.py and core_wrap.cc. The

latter > has the _core module, which at run-time cannot be
found as > internal._core. The Python 2.6 code that was there
used to fall > back on just importing _core (after failing
find_module and > load_module). The Python 2.7 code you added
tries to import > internal._core, and that does not exist.

I can either stick with the Python 2.7 code you added and >

override SWIG_name in the generated code to include the >
package, or I can bypass the Python 2.7 code and use the 2.6 >
code without issues.

Does any of this sound reasonable? A genuine bug?
Thanks,
Andreas
IMPORTANT NOTICE: The contents of this email and any >

attachments are confidential and may also be privileged. If you

are not the intended recipient, please notify the sender >

immediately and do not disclose the contents to any other >
person, use it for any purpose, or store or copy the >
information in any medium. Thank you.
--

IMPORTANT NOTICE: The contents of this email and any
attachments are confidential and may also be privileged. If you
are not the intended recipient, please notify the sender
immediately and do not disclose the contents to any other
person, use it for any purpose, or store or copy the
information in any medium. Thank you.

--

[Andreas Hansson]

Hi Mike,

We simply load the statically linked part by calling the SWIG_init
function of the wrapper. If this needs to be extended to also manipulate
the context somehow please let me know and I can try it out.

Thanks,

Andreas

Post by m***@comcast.net

Hi Mike, I should add that our specific setup relies on static
linking (for ease-of-distribution reasons). Thus, there is no
_core.so file. The specific setup has worked fine for quite
some years, but with 3.0.9 we face the issue mentioned in my
previous mail. A brute force hack is to change the SWIG_name in
the generated swig wrapper to include the package name, but it
would be good if we could avoid this hackery.

In this case there is 1/2 of the swig generated module installed in
a package (the pure python .py part). And then the other 1/2 of the
module (the C/C++ part) is *not* installed in the same package. This
is the part that does not seem good to me.
It seems to make more sense that both parts of the module should be
in the same package. If the _foo.so part is statically linked into an
interpreter (and loaded via freeze or something) then whatever loads it
still needs to put it into the same package as the other 1/2. At
least that is what makes the most sense to me.
But I guess swig always worked the other way and fell back to (or
always) loaded the C part as a global module. So, I'm working on a
fix that falls back to this global search location. The namespace
package support should make things easier for folks who would like
both parts of the swig module to really be in the same package. So,
both ways of doing things will work.
Mike

Let me know if you need any further information.
Thanks,
Andreas

Hi Mike, Apologies for contacting you like this, but I am

still > not 100% sure swig is really to blame, but wanted to at
least > share what I know.
It is also my guess that #621 is involved with what is going on
here. The change I made did pass all the existing swig tests
(including all of the swig python import and package tests).
So, I think there must be a case which is not covered.
- The swig wrapper (core.i) was built to put it into the
package "internal".
- The _core.so in *not* placed in the internal package but is
instead in the global namespace.
The old swig code always looked for _core.so in the global
namespace (near as I can tell). But the new code looks for it
in the package swig was told to find it. I guess there is not
a test for falling back to look for the C wrapper in the global
namespace.
I'm not sure if this is even a good idea or not.
To confirm this could you supply the locations of where core.py
and _core.so (along with any __init__.py files)? The contents
of your PYTHONPATH (or sys.path) factor into this as well.

After upgrading to swig 3.0.9 I appear to have swig wrapper >

modules that can no longer be found with Python 2.7. If I >
revert your commit 23d2eb555fc41230bf68953007b72c08982a5386 it

https://github.com/swig/swig/commit/23d2eb555fc41230bf68953007b72c08982a53

86

It seems the swig wrappers have a SWIG_name that does not >

include the package (despite it being specified in the >
module). In my example I have a core.i that specifies a package

“internal” that generates core.py and core_wrap.cc. The

latter > has the _core module, which at run-time cannot be
found as > internal._core. The Python 2.6 code that was there
used to fall > back on just importing _core (after failing
find_module and > load_module). The Python 2.7 code you added
tries to import > internal._core, and that does not exist.

I can either stick with the Python 2.7 code you added and >

override SWIG_name in the generated code to include the >
package, or I can bypass the Python 2.7 code and use the 2.6 >
code without issues.

Does any of this sound reasonable? A genuine bug?
Thanks,
Andreas
IMPORTANT NOTICE: The contents of this email and any >

attachments are confidential and may also be privileged. If you

are not the intended recipient, please notify the sender >

immediately and do not disclose the contents to any other >
person, use it for any purpose, or store or copy the >
information in any medium. Thank you.
--

IMPORTANT NOTICE: The contents of this email and any
attachments are confidential and may also be privileged. If you
are not the intended recipient, please notify the sender
immediately and do not disclose the contents to any other
person, use it for any purpose, or store or copy the
information in any medium. Thank you.

--

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.