Post by Jitka PlesnÃkováPost by Jitka PlesnÃkováI built nordugrid-arc from Fedora source rpm. You can find
complete build.log here.
https://jplesnik.fedorapeople.org/nordugrid-arc/
I attached the list of dependencies requires by source rpm. I
also attached the list of some commands which are running
during the build and you should be able to reproduce the
failure with it.
Jitka
The test failed when option "--with-altpython=python3" is used
for configure.
I've been able to build nordugrid-arc from source found on the
projects main site. The tests must have some run-time dependency I
don't have installed. But looking at the source, I can tell a bit
about what is going on.
They have a python package called arc. All of the swig generated
C wrapper modules are combined into a single shared object called
_arc.so that the packages __init__.py is loading. _arc.so bypasses
the normal python import machinery and calls each swig wrappers
PyModule_init() method. It also injects attributes (_common, _loader,
_message, etc) into the arc package module.
Right before it (_arc.so) initializes the embedded modules it
"imports" a reference to it's package with the following line of code:
// Initialise all the SWIG low level modules
PyObject *package = PyImport_AddModule((char *)"arc"); // a means to get a ha\
ndle to the package, not sure if this is a great idea but it works
The documentation for PyImport_AddModule has this to say:
Note: This function does not load or import the module; if the module
wasn’t already loaded, you will get an empty module object. Use
PyImport_ImportModule() or one of its variants to import a
module. Package structures implied by a dotted name for name are not
created if not already present.
What concerns me is that all this is going on while the package arc
is still being initialized. arc/__init__.py is the one which loads
_arc. This module is loaded as a global by python since (like with
SVN) it is not in the arc package. This in itself is fine and should
not cause a problem now.
However __init__.py (the arc package itself) is not done being
loaded by python and _arc then begins messing around with the python
loading machinery as mentioned above. The end result is that the new
code in swig:
importlib.import_module('_common') # fails
but the older way apparently works:
from arc import _common
I was told by Brett Cannon (core python developer) just a few months
ago that importlib.import_module() was the preferred way to import
named modules. My guess is that what _arc.so is doing is messing up
the normal python import machinery.
As I read the code, _arc.so initializes _common but goes totally
around the python import system. It then adds this module (not part
of a package and not in sys.modules as far as I can tell) as an attribute
to arc. So, 'from arc import _common' "works" because the import
statement is sucking in an attribute from arc and not really loading a
module at all.
Here is a little experiment simply loading _arc (as arc/__init__.py
does):
lrd > pwd
/home/romberg/kde/Downloads/nordugrid-arc-5.1.1/python/altpython
lrd > setenv PYTHONPATH arc/.libs
lrd > python3
Python 3.4.3 (default, Mar 31 2016, 20:42:37)
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
dict_keys(['_locale', 'marshal', '_warnings', '__main__', 'sys',
'abc', '_codecs', '_arc', 'encodings.utf_8', 'readline', '_thread',
'encodings.aliases', 'sysconfig', 'io', 'abrt_exception_handler3',
'_collections_abc', 'codecs', 'os.path', 'builtins', '_imp',
'posixpath', '_frozen_importlib', 'errno', 'swig_runtime_data4',
'genericpath', 'posix', 'stat', 'site', 'zipimport', '_weakref', 'os',
'atexit', '_weakrefset', 'encodings.latin_1', '_io', 'encodings',
'_sitebuiltins', 'arc', 'signal', '_stat', 'rlcompleter',
'_bootlocale', '_sysconfigdata'])
So, _common and the other modules have had their init() methods
called. But they are not in sys.modules. So, none of the normal
python import machinery can find them.
In a nutshell, I think the method arc is using to combine their swig
modules works with swig-3.0.8 because of a strange undocumented side
case of how 'from arc import X' works when arc is still under
construction. As it is being constructed _arc is adding attributes to
it. And also counting on 'from arc import X' going directly into the
currently under construction module and pulling out attributes to
use. The whole thing looks somewhat sketchy. The arc developer
mentions this in the comment above.
Swig could simply pull out all my changes and the above "method" of
how arc imports C modules will continue to work (until someone tweaks
the python machinery :). Switching to using importlib was/is intended to make
things easier for folks like the arc developers. With the changes in
swig-3.0.10 they can now actually put all the swig .so objects into
the arc package like they are clearly trying to do. The swig-3.0.8
code prevents them from doing so because all _foo.so modules are being
loaded as globals.
Or I can add another test case to swig and this could become another
supported method of module/package loading.
Mike
P.S. Sorry for the wall of text. But it is sorta complicated.