Hello, good evening and welcome.
For the next few months I will be using this blog to help document and publicise my "Ctypes for Perl" project. The project is being carried out for TPF under the auspices of the Google Summer of Code programme, and mentored by Reini Urban.
What's a ctypes?
'ctypes' is the Foreign Function Interface (FFI) library distributed with the Python core. It basically allows native C libraries to be called easily from Python; for module authors, it allows the wrapping of C libraries in pure Python.
This is obviously a powerful concept. Imagine a world where Perl module authors didn't need to use XS, and module consumers don't need to have a correctly configured compiler set up on their system. This is the purpose of the project: to create an easy, cross-platform, pure-Perl interface to native C libraries.
Implementations
ctypes is based on libffi. It's small, supports a wide range of systems, and has a very liberal license. It's been distributed with GCC for a number of years, used by gcj for interfacing between interpreted and compiled code.
From what I can gather, Python set the trend in dynamic languages using libffi. Looking at the success of the Python module, developers at Mozilla chose libffi to develop ctypes.jsm. Ruby-FFI uses it too, so there's plenty of prior art which will hopefully help me out.
The FFI problem hasn't been ignored in the Perl world. There's FFI.pm, the biggest disadvantage of which in my view is being built on libffcall, a library analogous to libffi but under the GPL (I don't think libffi was around at the time FFI.pm was written). It also sets out to provide a 'low-level' interface. P5NCI, on the other hand, is all about the lovely interfaces, but only allows up to four arguments passed to C functions, and doesn't yet support passing in pointers. C::Dynalib provides similar functionality to the other two modules; click here for the latest updates on its development. It's worth pointing out that none of these modules worked out of the box on Strawberry 5.10.1.
My proposed API rolls in features of several of the above implementations, particularly P5NCI and FFI.pm. I have indeed copied and pasted swathes from their POD pages (So what? Wanna fight about it?). I plan to also mimic C::DynaLib's acceptance of both positional & named parameters; examples are omitted below for succinctness.
1. Functional
use Ptypes;
# FFI.pm's interface of Absolute Freedom...
my $addr = (address of a C function)
my $signature = (function signature)
my $ret = Ptypes::call($addr, $signature, ...);
# Keeping things where you can see them...
my $library_path = Ptypes::find_lib( 'mathtastic' );
my $library = Ptypes::load_lib( $library_path );
my $double_func = Ptypes::load_func( $library, 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );
# Supplying a Perl callback...
$ret = Ptypes::call($addr, $signature, $subref, ...)
2. Objectionable
use Ptypes;
my $lib = Ptypes->new( library => 'mathtastic' [, package => 'MyPackage' ] );
my $double_func = $lib->load_function( 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );
# Exposing funcs directly in Perl namespaces...
$lib->install_function( 'double_int', 'ii' [, 'perl_sub_name', 'AnotherPackage' ] );
my $ret = AnotherPackage::double_int( 1 );
# or simply...
package AnotherPackage;
my $ret = double_int( 3 );
All fairly self-explanatory, perhaps apart from arguments like 'ii' or 'dd' - these strings describe return values and arguments for C functions in the same notation used by pack. In addition to the above, the module may provide mechanisms for manipulating C data types directly from Perl ($c = new Ptypes::char). To start off with though there'll be a fairly straight-forward / 'stupid' translation based on seeing what kind of data's in your SV*, for initial experimentation.
There's also some exciting stuff to do with GCC::TranslationUnit to tell you about, but details of that will wait till later. For now, I have some questions for you, the community:
Don't want to gush here, but I'm so chuffed* to be working on this. I'm already learning loads, and I think it will save a lot of blood, sweat & tears for module authors and users in the future. I want to thank rurban for his guidance & help so far, and dukeleto and others for organising the The Perl Foundation's participation in GSoC and letting me participate!
I like to work log, so follow @doubious_code on Twitter to get far more information than you want about this project. I hope to be blogging pretty regularly too.
* For American-speakers, 'chuffed' is kinda equivalent to 'stoked'
Because ctypes has such a good reputation, I'd hijack their brand as much as possible.
So, distribution name "ctypes".
Main namespace, "CDLL".
So in python...
cdll.LoadLibrary('libc.so.6');
CDLL->load_library('libc.so.6');
Re:Steal their brand
doubi on 2010-05-25T00:19:26
Good point about the branding.
Also, in the effort of articulating why I was never keen on the Python API, it's just 'clicked' with me. Thanks for that;)
I take it you can't just simply ask for 'libc' on *nix 'cause you might have several libc.so.x lying around.
Why is that? Could there be a way to mitigate for it? I do like the ol' automatic accessor style:cdll->libc->function('arg').
If that wasn't available cross platform I wouldn't even bother making it an optional syntax for Windows.
Another reason not to use "ptypes" - you're not replacing the C with Perl, you're replacing the Python with Perl.
I agree with Alias that somehow we should piggy-back off the existing "ctypes" brand, so that (e.g.) when someone googles for "ctypes" they find the new implementation too. I'm not experienced enough with it to know if this is the correct term to piggyback off of, but some pig should be backed.
Do you envision any special-purpose stuff for creating wrapper modules? One thing that's been fairly clear in the Inline::C world is that the needs of people writing wrapper modules & people writing little scripts that call libraries are pretty different.
Very nice work you're proposing. I'd totally use it.
Re:Name change
doubi on 2010-05-25T00:29:25
Another reason not to use "ptypes" - you're not replacing the C with Perl, you're replacing the Python with Perl.
Granted. In my head, it was meant to be some kind of weird pun. I don't think it worked
:F some pig should be backed.
As they say, when the world gives you pigs, all you can do is back.
Do you envision any special-purpose stuff for creating wrapper modules? One thing that's been fairly clear in the Inline::C world is that the needs of people writing wrapper modules & people writing little scripts that call libraries are pretty different.
If it's not useful for fully wrapping libs, I will consider it a bit of a failure. As you say, there are already a few modules on CPAN you could probably coerce into working for you for the purposes of tinkering, so I do want to go well beyond that.
Hi doubi,
I'm glad you have the enthusiasm to work on this project. As you know (and some other readers here may), I've also proposed a CTypes for Perl project to the TPF two or three times, but it wasn't accepted, so I'm glad it was accepted as part of GSoC.
I'd like to note that when I requested help with some ctypes (in Python) problems on #python on Freenode, some of the people there told me that they now prefer to use Cython over ctypes and that "it also has a strict subset so that a cython module can be used directly in Python, rather than compiled" (but "idea how good that is though.") so it may be worth to investigate. Cython is not a core Python module though, which has been the case for ctypes for a few years now.
Otherwise, note that your first interface is procedural rather than functional (see Functional programming on the wikipedia, which is not the opposite of Object Oriented programming). The interface looks fine, but I think we may eventually overgrow the pack/unpack like notation because ctypes and the C programming language supports quite a bit more than what unpack gives. Also see the Data-ParseBinary CPAN module, which also has origins in the Python "construct" module, and which is far superior to pack/unpack.
I agree with the other people here that we should call it "ctypes" or something and not "ptypes". I tried to log-in into twitter and follow your tweets, but it said that "Twitter is over capacity." and wouldn't let me. I'm not visiting Twitter a lot often. In any case, I'm subscribed to the use.perl.org master Perl feed, and you should see about getting it syndicated on the various Perl planets.
Good luck with perlifying ctypes.
