[Swig-devel] Got MATLAB working for Octave

[Joel Andersson]

Hi all,

If you are familiar with Octave, you know that it's designed to be syntax
compatible with MATLAB. This doesn't apply to the internal API, which is
why it has never been an option to use SWIG's Octave module for MATLAB.
SWIG's Octave module relies on Octave's internal C++ API which has no
MATLAB counterpart. The MATLAB module that we've been working on uses a
different design, relying on MATLAB's support for object oriented
programming and builds up a class hierarchy in MATLAB that serves to
redirect all function calls to a single entry-point MEX function.

Now, I was curious if it would be possible to use the MATLAB module working
from Octave, given that Octave now supports both MEX and MATLAB-style
object oriented programming. And after a bit of work it turned out that the
answer is yes! I had to modify the MATLAB module a bit - (cf.
https://github.com/jaeandersson/swig/tree/octave_new and
https://github.com/jaeandersson/swig/issues/67) - to work around some
MATLAB features that are not (yet) supported by Octave, but after that I
managed to compile and run SWIG's "class" example as well as the generated
MATLAB interface to CasADi, which uses a large subset of SWIG.

I find this exciting for a number of reasons:

1. By using the same module for both Octave and MATLAB, there is no need to
duplicate the work to e.g. write typemaps and to implement
language-specific features for both Octave and MATLAB. We have successfully
generated a proof-of-concept Octave interface to CasADi using SWIG's Octave
module, but to implement all these language-specific things has been a
daunting task for a (compared to MATLAB) relatively small number of users.
By making the modules MATLAB/Octave bilingual, one can be pretty sure that
if it works for one, it will work for the other.

2. I would argue that the MATLAB module offers a much more natural user
syntax than the Octave module. SWIG's Octave module was written at a time
when Octave didn't support object oriented programming, but manages to work
around this fact. But it feels weird e.g. to have classes be variables in
Octave and not being able to call "help" for a class. The MATLAB module, on
the other hand, uses Octave's newly added support for OOP.

3. If the MATLAB module is Octave compatible, we can use Octave for testing
it e.g. on Travis. And you won't need to obtain a MATLAB license to run the
test suite.

Thoughts, comments?

Best regards,
Joel

[Kris Thielemans]

Hi Joel
I think this is great stuff. The swig-Octave interface is quite outdated.

thanks!
Kris

Post by Joel Andersson
Hi all,
If you are familiar with Octave, you know that it's designed to be syntax
compatible with MATLAB. This doesn't apply to the internal API, which is
why it has never been an option to use SWIG's Octave module for MATLAB.
SWIG's Octave module relies on Octave's internal C++ API which has no
MATLAB counterpart. The MATLAB module that we've been working on uses a
different design, relying on MATLAB's support for object oriented
programming and builds up a class hierarchy in MATLAB that serves to
redirect all function calls to a single entry-point MEX function.
Now, I was curious if it would be possible to use the MATLAB module
working from Octave, given that Octave now supports both MEX and
MATLAB-style object oriented programming. And after a bit of work it turned
out that the answer is yes! I had to modify the MATLAB module a bit - (cf.
https://github.com/jaeandersson/swig/tree/octave_new and
https://github.com/jaeandersson/swig/issues/67) - to work around some
MATLAB features that are not (yet) supported by Octave, but after that I
managed to compile and run SWIG's "class" example as well as the generated
MATLAB interface to CasADi, which uses a large subset of SWIG.
1. By using the same module for both Octave and MATLAB, there is no need
to duplicate the work to e.g. write typemaps and to implement
language-specific features for both Octave and MATLAB. We have successfully
generated a proof-of-concept Octave interface to CasADi using SWIG's Octave
module, but to implement all these language-specific things has been a
daunting task for a (compared to MATLAB) relatively small number of users.
By making the modules MATLAB/Octave bilingual, one can be pretty sure that
if it works for one, it will work for the other.
2. I would argue that the MATLAB module offers a much more natural user
syntax than the Octave module. SWIG's Octave module was written at a time
when Octave didn't support object oriented programming, but manages to work
around this fact. But it feels weird e.g. to have classes be variables in
Octave and not being able to call "help" for a class. The MATLAB module, on
the other hand, uses Octave's newly added support for OOP.
3. If the MATLAB module is Octave compatible, we can use Octave for
testing it e.g. on Travis. And you won't need to obtain a MATLAB license to
run the test suite.
Thoughts, comments?
Best regards,
Joel
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning
reports.
http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381
_______________________________________________
Swig-devel mailing list
https://lists.sourceforge.net/lists/listinfo/swig-devel

[William S Fulton]

Post by Joel Andersson
Hi all,
If you are familiar with Octave, you know that it's designed to be syntax
compatible with MATLAB. This doesn't apply to the internal API, which is why
it has never been an option to use SWIG's Octave module for MATLAB. SWIG's
Octave module relies on Octave's internal C++ API which has no MATLAB
counterpart. The MATLAB module that we've been working on uses a different
design, relying on MATLAB's support for object oriented programming and
builds up a class hierarchy in MATLAB that serves to redirect all function
calls to a single entry-point MEX function.
Now, I was curious if it would be possible to use the MATLAB module working
from Octave, given that Octave now supports both MEX and MATLAB-style object
oriented programming. And after a bit of work it turned out that the answer
is yes! I had to modify the MATLAB module a bit - (cf.
https://github.com/jaeandersson/swig/tree/octave_new and
https://github.com/jaeandersson/swig/issues/67) - to work around some MATLAB
features that are not (yet) supported by Octave, but after that I managed to
compile and run SWIG's "class" example as well as the generated MATLAB
interface to CasADi, which uses a large subset of SWIG.
1. By using the same module for both Octave and MATLAB, there is no need to
duplicate the work to e.g. write typemaps and to implement language-specific
features for both Octave and MATLAB. We have successfully generated a
proof-of-concept Octave interface to CasADi using SWIG's Octave module, but
to implement all these language-specific things has been a daunting task for
a (compared to MATLAB) relatively small number of users. By making the
modules MATLAB/Octave bilingual, one can be pretty sure that if it works for
one, it will work for the other.
2. I would argue that the MATLAB module offers a much more natural user
syntax than the Octave module. SWIG's Octave module was written at a time
when Octave didn't support object oriented programming, but manages to work
around this fact. But it feels weird e.g. to have classes be variables in
Octave and not being able to call "help" for a class. The MATLAB module, on
the other hand, uses Octave's newly added support for OOP.
3. If the MATLAB module is Octave compatible, we can use Octave for testing
it e.g. on Travis. And you won't need to obtain a MATLAB license to run the
test suite.
Thoughts, comments?

Looking to the future, I think you are saying that it will be possible
to support just one language module for use from Octave and MATLAB. If
so that seems okay to me in principle and would be preferable thank
supporting two modules. If we can test this using Octave on Travis or
any OS supporting Octave that seems good enough to incorporate it into
master for proper maintenance going forwards. Would you like to
outline any plans in this area for discussion? Perhaps replace the
Octave module with the MATLAB module in SWIG-3.1?

William

[Joel Andersson]

Hi William, all,

Right, I do think that it is possible to have one language module for
supporting both Octave and MATLAB. It might require some compromises on the
design side, but so far these were very minor. After the fixes I made this
week (now merged with my SWIG fork), I'm only aware of one thing that
doesn't work from Octave: the "import" command, which is not necessarily a
blocker.

Given that Octave is an open-source project, free and readily available,
and already used within SWIG, I think it's natural to use Octave to test
the now bilingual Octave/MATLAB module. I also think that if a feature
works in Octave, one can be pretty sure it works for MATLAB too. The other
way around is less certain.

If everyone agrees that this is the way forward, I'd be great to have
someone to help changing the build system from MATLAB to Octave for the new
module. I'm pretty much illiterate when it comes to automake stuff...

About replacing the existing Octave module, I think it makes sense to have
both modules live side-by-side for at least one SWIG release. The Octave
module uses Octave's internal C++ API, whereas the MATLAB module uses
MATLAB's C API, which Octave supports via a MATLAB compatibility layer.
This means that things like typemaps all have to be rewritten, which can be
disruptive. While keeping them both does not make sense long-term, I'd say
that when to remove the Octave module is s a question that needs to be
posed to existing users of the module. If they are able and willing to
migrate to the MATLAB module.

Best regards,
Joel

Post by Joel Andersson

Post by Joel Andersson
Hi all,
If you are familiar with Octave, you know that it's designed to be syntax
compatible with MATLAB. This doesn't apply to the internal API, which is

why

Post by Joel Andersson
it has never been an option to use SWIG's Octave module for MATLAB.

SWIG's

Post by Joel Andersson
Octave module relies on Octave's internal C++ API which has no MATLAB
counterpart. The MATLAB module that we've been working on uses a

different

Post by Joel Andersson
design, relying on MATLAB's support for object oriented programming and
builds up a class hierarchy in MATLAB that serves to redirect all

function

Post by Joel Andersson
calls to a single entry-point MEX function.
Now, I was curious if it would be possible to use the MATLAB module

working

Post by Joel Andersson
from Octave, given that Octave now supports both MEX and MATLAB-style

object

Post by Joel Andersson
oriented programming. And after a bit of work it turned out that the

answer

Post by Joel Andersson
is yes! I had to modify the MATLAB module a bit - (cf.
https://github.com/jaeandersson/swig/tree/octave_new and
https://github.com/jaeandersson/swig/issues/67) - to work around some

MATLAB

Post by Joel Andersson
features that are not (yet) supported by Octave, but after that I

managed to

Post by Joel Andersson
compile and run SWIG's "class" example as well as the generated MATLAB
interface to CasADi, which uses a large subset of SWIG.
1. By using the same module for both Octave and MATLAB, there is no need

to

Post by Joel Andersson
duplicate the work to e.g. write typemaps and to implement

language-specific

Post by Joel Andersson
features for both Octave and MATLAB. We have successfully generated a
proof-of-concept Octave interface to CasADi using SWIG's Octave module,

but

Post by Joel Andersson
to implement all these language-specific things has been a daunting task

for

Post by Joel Andersson
a (compared to MATLAB) relatively small number of users. By making the
modules MATLAB/Octave bilingual, one can be pretty sure that if it works

for

Post by Joel Andersson
one, it will work for the other.
2. I would argue that the MATLAB module offers a much more natural user
syntax than the Octave module. SWIG's Octave module was written at a time
when Octave didn't support object oriented programming, but manages to

work

Post by Joel Andersson
around this fact. But it feels weird e.g. to have classes be variables in
Octave and not being able to call "help" for a class. The MATLAB module,

on

Post by Joel Andersson
the other hand, uses Octave's newly added support for OOP.
3. If the MATLAB module is Octave compatible, we can use Octave for

testing

Post by Joel Andersson
it e.g. on Travis. And you won't need to obtain a MATLAB license to run

the

Post by Joel Andersson
test suite.
Thoughts, comments?

Looking to the future, I think you are saying that it will be possible
to support just one language module for use from Octave and MATLAB. If
so that seems okay to me in principle and would be preferable thank
supporting two modules. If we can test this using Octave on Travis or
any OS supporting Octave that seems good enough to incorporate it into
master for proper maintenance going forwards. Would you like to
outline any plans in this area for discussion? Perhaps replace the
Octave module with the MATLAB module in SWIG-3.1?
William

[William S Fulton]

Post by Joel Andersson
Hi William, all,
Right, I do think that it is possible to have one language module for
supporting both Octave and MATLAB. It might require some compromises on the
design side, but so far these were very minor. After the fixes I made this
week (now merged with my SWIG fork), I'm only aware of one thing that
doesn't work from Octave: the "import" command, which is not necessarily a
blocker.
Given that Octave is an open-source project, free and readily available, and
already used within SWIG, I think it's natural to use Octave to test the now
bilingual Octave/MATLAB module. I also think that if a feature works in
Octave, one can be pretty sure it works for MATLAB too. The other way around
is less certain.
If everyone agrees that this is the way forward, I'd be great to have
someone to help changing the build system from MATLAB to Octave for the new
module. I'm pretty much illiterate when it comes to automake stuff...
About replacing the existing Octave module, I think it makes sense to have
both modules live side-by-side for at least one SWIG release. The Octave
module uses Octave's internal C++ API, whereas the MATLAB module uses
MATLAB's C API, which Octave supports via a MATLAB compatibility layer. This
means that things like typemaps all have to be rewritten, which can be
disruptive. While keeping them both does not make sense long-term, I'd say
that when to remove the Octave module is s a question that needs to be posed
to existing users of the module. If they are able and willing to migrate to
the MATLAB module.

Having both together for a while makes sense, then we can ask Octave
users if they can switch to the MATLAB module.

How about we aim to merge the MATLAB branch into swig-3.1.0 and
release the 3.1.x series with both MATLAB and Octave with a view to
seeing if we can drop Octave in swig-3.2?

William

[Joel Andersson]

Post by William S Fulton
How about we aim to merge the MATLAB branch into swig-3.1.0 and
release the 3.1.x series with both MATLAB and Octave with a view to
seeing if we can drop Octave in swig-3.2?

That sounds good to me. I'll aim at making the MATLAB branch ready-to-merge
within the next couple of weeks.

Joel

[Ian Bell]

I like this idea, especially the idea of octave/MATLAB interchangeability.
Two issues that have bedeviled me with the MATLAB wrapper that I would
(ideally) like to see addressed before merging into master:

* The MATLAB wrapper code is SLOW:
https://github.com/jaeandersson/swig/issues/18
* Would be nice to have standard typemaps working:
https://github.com/jaeandersson/swig/issues/29 - this *does* work in Octave

I know neither are required, but they are non-starters for me in further
development with MATLAB+swig.

Kind Regards,
Ian

Post by William S Fulton
How about we aim to merge the MATLAB branch into swig-3.1.0 and

Post by William S Fulton
release the 3.1.x series with both MATLAB and Octave with a view to
seeing if we can drop Octave in swig-3.2?

That sounds good to me. I'll aim at making the MATLAB branch
ready-to-merge within the next couple of weeks.
Joel
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning
reports. http://sdm.link/zohomanageengine
_______________________________________________
Swig-devel mailing list
https://lists.sourceforge.net/lists/listinfo/swig-devel

[Joel Andersson]

Post by Ian Bell
https://github.com/jaeandersson/swig/issues/18

The code was refactored for speed as part of that issue, which was then
closed. For us, speed is essentially a non-issue currently, completely
dwarfed by the time spent in the C++ code. I don't see how the code can be
made much faster with the current design. And I tend to think that it's
more of a MATLAB-vs-Python issue than an issue with the bindings. For-loops
in MATLAB, without the just-in-time compiler, are well known to be slow.

Post by Ian Bell
https://github.com/jaeandersson/swig/issues/29 - this *does* work in Octave

OK. Someone else would take the lead on this, in the extent that it's still
an issue.

I know neither are required, but they are non-starters for me in further

Post by Ian Bell
development with MATLAB+swig.

OK. I don't have any intention of working on either of these issues. So if
it's made a precondition for merging with master, and no one volunteers to
take the lead, the branch simply won't get merged.

Joel

[William S Fulton]

Post by Joel Andersson

Post by Ian Bell
https://github.com/jaeandersson/swig/issues/18

The code was refactored for speed as part of that issue, which was then
closed. For us, speed is essentially a non-issue currently, completely
dwarfed by the time spent in the C++ code. I don't see how the code can be
made much faster with the current design. And I tend to think that it's more
of a MATLAB-vs-Python issue than an issue with the bindings. For-loops in
MATLAB, without the just-in-time compiler, are well known to be slow.

Post by Ian Bell
https://github.com/jaeandersson/swig/issues/29 - this does work in Octave

OK. Someone else would take the lead on this, in the extent that it's still
an issue.

Post by Ian Bell
I know neither are required, but they are non-starters for me in further
development with MATLAB+swig.

OK. I don't have any intention of working on either of these issues. So if
it's made a precondition for merging with master, and no one volunteers to
take the lead, the branch simply won't get merged.

I havn't looked at the MATLAB module, but I would expect a modern
module like MATLAB to have been implemented following the same common
style, principles, features and typemaps as the other language
modules. If not, there'd need to be a good reason. Given Octave uses
the common typemaps in the UTL, which offers a lot of functionality
for free, I'd also hope that the MATLAB module has. Is this the case
or not?

William

[Joel Andersson]

Post by William S Fulton
I havn't looked at the MATLAB module, but I would expect a modern
module like MATLAB to have been implemented following the same common
style, principles, features and typemaps as the other language
modules. If not, there'd need to be a good reason. Given Octave uses
the common typemaps in the UTL, which offers a lot of functionality
for free, I'd also hope that the MATLAB module has. Is this the case
or not?

I'm not 100 % sure of the current status, but if I recall correctly,
typemaps are mostly working. As you say, this functionality is for free
anyway. I think the main issue has been the auto-conversion between
built-in MATLAB types and std types. For example, an std::vector<double> is
probably best mapped to a 1-by-n MATLAB matrix whereas a
std::vector<MyClass> needs to be mapped to an 1-by-n MATLAB cell array.
"struct" in MATLAB is similar to "dict" in Python, but it has also major
differences, so a mapping to std::map<std::string, typename> is not trivial.

Anyway, I don't think that sweeping statements such as "typemaps are not
working" or "the interface is slow" are warranted. The MATLAB module works
great for us and produces high-quality interface code (I'd argue superior
to the Octave module thanks to the use of modern language features in
MATLAB/Octave). We build, test and distribute this generated code in binary
form. In the event that something is not working, it would be good to point
at a particular unit test or construct a failing example. And preferably
make a fix and a pull request.

As for style, principles etc, I'm not aware of any departures from the
coding principles used for other modules.

Looking forward, I'm volunteering to migrate the unit tests to Octave,
report back on current test suite coverage and make an iteration on the
documentation for the module. IMO, the next natural step would be a code
review and then take it from there.

Joel

[Ian Bell]

My comment was precisely about the auto-conversion between STL types and
MATLAB types. For instance, as I noted in my original issue, wrapping a
function like this taking STL types gets messy without automatic typemaps:

std::vector<std::vector<double> > PropsSImulti(const
std::vector<std::string> &Outputs,
const std::string &Name1,
const std::vector<double> &Prop1,
const std::string &Name2,
const
std::vector<double> &Prop2,
const std::string &backend,
const
std::vector<std::string> &fluids,
const
std::vector<double> &fractions);

and if it understood STL types natively, it would *just work, *as is the
case in python, amongst other SWIG-supported languages.

Sorry for my wording about the speed. But it matters. I should update my
speed benchmark and see what the speed difference between python and MATLAB
bindings is now...

Ian

Post by William S Fulton
I havn't looked at the MATLAB module, but I would expect a modern

Post by William S Fulton
module like MATLAB to have been implemented following the same common
style, principles, features and typemaps as the other language
modules. If not, there'd need to be a good reason. Given Octave uses
the common typemaps in the UTL, which offers a lot of functionality
for free, I'd also hope that the MATLAB module has. Is this the case
or not?

I'm not 100 % sure of the current status, but if I recall correctly,
typemaps are mostly working. As you say, this functionality is for free
anyway. I think the main issue has been the auto-conversion between
built-in MATLAB types and std types. For example, an std::vector<double> is
probably best mapped to a 1-by-n MATLAB matrix whereas a
std::vector<MyClass> needs to be mapped to an 1-by-n MATLAB cell array.
"struct" in MATLAB is similar to "dict" in Python, but it has also major
differences, so a mapping to std::map<std::string, typename> is not trivial.
Anyway, I don't think that sweeping statements such as "typemaps are not
working" or "the interface is slow" are warranted. The MATLAB module works
great for us and produces high-quality interface code (I'd argue superior
to the Octave module thanks to the use of modern language features in
MATLAB/Octave). We build, test and distribute this generated code in binary
form. In the event that something is not working, it would be good to point
at a particular unit test or construct a failing example. And preferably
make a fix and a pull request.
As for style, principles etc, I'm not aware of any departures from the
coding principles used for other modules.
Looking forward, I'm volunteering to migrate the unit tests to Octave,
report back on current test suite coverage and make an iteration on the
documentation for the module. IMO, the next natural step would be a code
review and then take it from there.
Joel