New patches: [manual: convert-* interface, optimizing translators overview optimized Stephen Compall **20060301043349 - (Wrapper generators): Add comment about the 30-90% figure. - Reword `Pointers' to be shorter/clearer. - Promote `Optimizing Type Translators' to a full section, a few other things here, most conspicuously expanding on the note about expand-* method definition time. - (Tutorial-Conclusion): Not a first draft anymore. - Document convert-to-foreign, convert-from-foreign, and free-converted-object. ] < > { hunk ./doc/cffi-manual.texinfo 1619 likely that either code or a good reason for lack of code is already present. -@impnote{This is something of a first draft; there are some other -things in @cffi{} that might deserve tutorial sections. Let us know -which ones you care about. define-foreign-type, -free-translated-object, structs.} +@impnote{There are some other things in @cffi{} that might deserve +tutorial sections, such as define-foreign-type, +free-translated-object, or structs. Let us know which ones you care +about.} @c =================================================================== hunk ./doc/cffi-manual.texinfo 1683 incremental development style enabled by @cffi{} generally reduces this proportion below that for languages like Python. +@c Where I got the above 30-90% figures: +@c 30%: lemonodor's post about SWIG +@c 90%: Balooga on #lisp. He said 99%, but that's probably an +@c exaggeration (leave it to me to pass judgement :) +@c -stephen + @c =================================================================== @c CHAPTER: Foreign Types hunk ./doc/cffi-manual.texinfo 1705 * Other Types:: * Defining Typedefs:: * Foreign Type Translators:: +* Optimizing Type Translators:: * Foreign Structure Types:: * Operations on Types:: * Allocating Foreign Objects:: hunk ./doc/cffi-manual.texinfo 1712 Dictionary +* convert-from-foreign:: +* convert-to-foreign:: * defbitfield:: * defcstruct:: * defcunion:: hunk ./doc/cffi-manual.texinfo 1732 * foreign-slot-value:: * foreign-type-alignment:: * foreign-type-size:: +* free-converted-object:: * free-translated-object:: * translate-from-foreign:: * translate-to-foreign:: hunk ./doc/cffi-manual.texinfo 1957 will prevent some pointer errors when calling foreign functions that cannot handle null pointers. -@strong{Please note:} these methods are meant as extensible hooks only -and you should not call them directly. Use @code{convert-to-foreign}, -@code{convert-from-foreign} and @code{free-converted-object} -instead. These will take care of following the typedef chain, for -example, calling all the applicable translators. It will also work -for non user-defined types like enums. +@strong{Please note:} these methods are meant as extensible hooks +only, and you should not call them directly. Use +@code{convert-to-foreign}, @code{convert-from-foreign} and +@code{free-converted-object} instead. These will take care of +following the typedef chain, for example, calling all the applicable +translators. They will also work for @cffi{}'s built-in types, such +as enums. @xref{Tutorial-Types,, Defining new types}, for a more involved tutorial example of type translators. hunk ./doc/cffi-manual.texinfo 1968 -@heading Optimizing Type Translators +@node Optimizing Type Translators +@section Optimizing Type Translators hunk ./doc/cffi-manual.texinfo 1971 +@cindex type translators, optimizing +@cindex compiler macros for type translation +@cindex defining type-translation compiler macros Being based on generic functions, the type translation mechanism hunk ./doc/cffi-manual.texinfo 1975 -described above can add a bit of overhead. Most of the times, this is -not significant but we provide a way of getting rid of the overhead -for the cases where it matters. +described above can add a bit of overhead. This is usually not +significant, but we nevertheless provide a way of getting rid of the +overhead for the cases where it matters. A good way to understand this issue is to look at the code generated by @code{defcfun}. Consider the following example using the hunk ./doc/cffi-manual.texinfo 1999 @end lisp In order to get rid of those generic function calls, @cffi{} has -another set of extensible generic functions that provide an interface +another set of extensible generic functions that provide functionality similar to @acronym{CL}'s compiler macros: @code{expand-to-foreign-dyn}, @code{expand-to-foreign} and @code{expand-from-foreign}. Here's how one could define hunk ./doc/cffi-manual.texinfo 2013 `(not (zerop ,value))) @end lisp +@noindent And here's what the macroexpansion of @code{foo} now looks like: @lisp hunk ./doc/cffi-manual.texinfo 2023 (NOT (ZEROP (%FOREIGN-FUNCALL "foo" :INT #:G3182 :INT))))) @end lisp +@noindent Much better. hunk ./doc/cffi-manual.texinfo 2026 -The third generic function in this set is @code{expand-to-foreign-dyn} -which can be used when it's necessary to allocate something with -dynamic extent. Consider the following example: +The third generic function in this interface is +@code{expand-to-foreign-dyn} which can be used when it's necessary to +allocate something with dynamic extent. Consider the following +example: @lisp ;;; This type inherits :string's translators. hunk ./doc/cffi-manual.texinfo 2041 ,@@body)) @end lisp -Not unlike @code{define-compiler-macro}'s @code{&whole} argument, -@cffi{} binds the special variable -@strong{@code{*runtime-translator-form*}} to a form you can return -from any of the @code{expand-*} methods. +To provide short-circuiting functionality similar to that provided by +@code{define-compiler-macro}'s @code{&whole} argument, @cffi{} binds +the special variable @strong{@code{*runtime-translator-form*}}. +Return it instead in cases where your method cannot generate an +appropriate replacement for it. The @code{expand-*} methods have precedence over their @code{translate-*} counterparts and are guaranteed to be used in hunk ./doc/cffi-manual.texinfo 2050 @code{defcfun}, @code{foreign-funcall}, @code{defcvar} and -@code{defcallback}. They are also used in compiler macros for the -following @cffi{} operators: @code{mem-ref}, @code{mem-aref}, -@code{foreign-slot-value}, @code{convert-to-foreign} and -@code{convert-from-foreign}, in which cases its use is not guaranteed -and depends on whether the Lisp compiler calls the respective compiler -macros or not. +@code{defcallback}. They may or may not be used in other @cffi{} +operators that need to translate between Lisp and C data; you may only +assume that @code{expand-*} methods will probably only be called +during Lisp compilation. @code{expand-to-foreign-dyn} has precedence over @code{expand-to-foreign} and is only used in @code{defcfun} and hunk ./doc/cffi-manual.texinfo 2057 -@code{foreign-funcall}. +@code{foreign-funcall}, only making sense in those contexts. @strong{Important note:} this set of generic functions is called at hunk ./doc/cffi-manual.texinfo 2060 -macroexpansion time and therefore you will need to either load the -method definitions earlier in a seperate file from, say, -@code{defcfun}s that will use them or wrap them in an -@code{eval-when}. Otherwise, they will not be found and the runtime -translators will be used instead. +macroexpansion time. Methods are defined when loaded or evaluated, +not compiled. You are responsible for ensuring that your +@code{expand-*} methods are defined when the @code{foreign-funcall} or +other forms that use them are compiled. One way to do this is to put +the method definitions earlier in the file and inside an appropriate +@code{eval-when} form; another way is to always load a separate Lisp +or @acronym{FASL} file containing your @code{expand-*} definitions +before compiling files with forms that ought to use them. Otherwise, +they will not be found and the runtime translators will be used +instead. @node Foreign Structure Types @section Foreign Structure Types hunk ./doc/cffi-manual.texinfo 2121 @xref{Allocating Foreign Memory}. +@c =================================================================== +@c CONVERT-FROM-FOREIGN + +@node convert-from-foreign +@unnumberedsec convert-from-foreign +@subheading Syntax +@Function{convert-from-foreign foreign-value type @result{} value} + +@subheading Arguments and Values + +@table @var +@item foreign-value +The primitive C value as returned from a primitive foreign function or +from @code{convert-to-foreign}. + +@item type +A @cffi{} type specifier. + +@item value +The Lisp value translated from @var{foreign-value}. +@end table + +@subheading Description + +This is an external interface to the type translation facility. In +the implementation, all foreign functions are ultimately defined as +type translation wrappers around primitive foreign function +invocations. + +This function is available mostly for inspection of the type +translation process, and possibly optimization of special cases of +your foreign function calls. + +Its behavior is better described under @code{translate-from-foreign}'s +documentation. + +@subheading Examples + +@lisp +CFFI-USER> (convert-to-foreign "a boat" :string) +@result{} # +@result{} (T) +CFFI-USER> (convert-from-foreign * :string) +@result{} "a boat" +@end lisp + +@subheading See Also +@seealso{convert-to-foreign} @* +@seealso{translate-from-foreign} + + +@c =================================================================== +@c CONVERT-TO-FOREIGN + +@node convert-to-foreign +@unnumberedsec convert-to-foreign +@subheading Syntax +@Function{convert-to-foreign value type @result{} foreign-value, alloc-params} + +@subheading Arguments and Values + +@table @var +@item value +The Lisp object to be translated to a foreign object. + +@item type +A @cffi{} type specifier. + +@item foreign-value +The primitive C value, ready to be passed to a primitive foreign +function. + +@item alloc-params +Something of a translation state; you must pass it to +@code{free-converted-object} along with the foreign value for that to +work. +@end table + +@subheading Description + +This is an external interface to the type translation facility. In +the implementation, all foreign functions are ultimately defined as +type translation wrappers around primitive foreign function +invocations. + +This function is available mostly for inspection of the type +translation process, and possibly optimization of special cases of +your foreign function calls. + +Its behavior is better described under @code{translate-to-foreign}'s +documentation. + +@subheading Examples + +@lisp +CFFI-USER> (convert-to-foreign t :boolean) +@result{} 1 +@result{} (NIL) +CFFI-USER> (convert-to-foreign "hello, world" :string) +@result{} # +@result{} (T) +CFFI-USER> (code-char (mem-aref * :char 5)) +@result{} #\, +@end lisp + +@subheading See Also +@seealso{convert-from-foreign} @* +@seealso{free-converted-object} @* +@seealso{translate-to-foreign} + + @c =================================================================== @c DEFBITFIELD hunk ./doc/cffi-manual.texinfo 3145 @seealso{foreign-type-alignment} +@c =================================================================== +@c FREE-CONVERTED-OBJECT + +@node free-converted-object +@unnumberedsec free-converted-object +@subheading Syntax +@Function{free-converted-object foreign-value type params} + +@subheading Arguments and Values + +@table @var +@item foreign-value +The C object to be freed. + +@item type +A @cffi{} type specifier. + +@item params +The state returned as the second value from @code{convert-to-foreign}; +used to implement the third argument to @code{free-translated-object}. +@end table + +@subheading Description + +The return value is unspecified. + +This is an external interface to the type translation facility. In +the implementation, all foreign functions are ultimately defined as +type translation wrappers around primitive foreign function +invocations. + +This function is available mostly for inspection of the type +translation process, and possibly optimization of special cases of +your foreign function calls. + +Its behavior is better described under @code{free-translated-object}'s +documentation. + +@subheading Examples + +@lisp +CFFI-USER> (convert-to-foreign "a boat" :string) +@result{} # +@result{} (T) +CFFI-USER> (free-converted-object * :string '(t)) +@result{} NIL +@end lisp + +@subheading See Also +@seealso{convert-from-foreign} @* +@seealso{convert-to-foreign} @* +@seealso{free-translated-object} + + @c =================================================================== @c FREE-TRANSLATED-OBJECT hunk ./doc/cffi-manual.texinfo 3409 the C @acronym{ABI}. Lisp has no such constraint on its fixnums; therefore, it only makes sense to think of fixnums as C integers if you assume that @cffi{} converts them when necessary, such as when -storing it in C format for use in a function call, or as the value of -a C variable. This, of course, can only be done by defining an area -of memory@footnote{The definition of @dfn{memory} includes the -@acronym{CPU} registers.}, represented in the abstract by an address, -and storing it there. +storing one for use in a C function call, or as the value of a C +variable. This requires defining an area of memory@footnote{The +definition of @dfn{memory} includes the @acronym{CPU} registers.}, +represented through an effective address, and storing it there. Due to this compartmentalization, it only makes sense to manipulate raw C data in Lisp through pointers to it. For example, while there hunk ./doc/cffi-manual.texinfo 3418 may be a Lisp representation of a @code{struct} that is converted to C at store time, you may only manipulate its raw data through a pointer. -Speaking abstractly, the C compiler does this also. +The C compiler does this also, albeit informally. @menu * Basic Pointer Operations:: } Context: [Remove known issues sub-section from the manual. Luis Oliveira **20060228193623 - Listing the expected failures in the manual is too much trouble. Removed those. - Also, renamed "The Scieneer Common Lisp" to "Scieneer CL" for consistency. ] [TODO item about a pointer type, suggested by Jörg Höhle Luis Oliveira **20060228191642] [More testing Luis Oliveira **20060227194854 - Make defcfun.undefined an expected failure for SBCL on non linkage-table platforms. - New file: tests/misc.lisp. - Added a couple of tests for cffi-features. ] [TODO item about our use of EVAL Luis Oliveira **20060227194813] [manual: strings, foreign allocation, add `Wrapper generators' Stephen Compall **20060228162638 - Explain the difference between mem-aref and mem-ref by analogy with C operators. - Use ::= instead of = in with-foreign-object's syntax. - (Strings): Explain that it is portable code. - (Other Types): Don't use metasyntactic variables in Lisp examples. - Add `Wrapper generators' section on Verrazano et al. - (Foreign Type Translators): Use Please note: instead of Note: to suppress makeinfo warning. ] [Try to improve the wording in foreign-alloc's description.. Luis Oliveira **20060225155627] [Minor change to foreign-alloc's documentation Luis Oliveira **20060225041058 - make it slightly clearer that count can be omitted when initial-contents is supplied. - add reference to with-foreign-object. ] [foreign-alloc changes Luis Oliveira **20060225034634 - bugfix, foreign-alloc doesn't need to call translate-type-to-foreign explicitly since mem-aref already does. Bug reported by Greg Pfeil. - new keyword argument: NULL-TERMINATE-P. - new regression tests for the bug described above and new tests for the new keyword argument. - document new argument. ] [More tests Luis Oliveira **20060224192231 - A couple of new tests involving defcfun/foreign-funcall/defcallback and lots of doubles and floats. These were written to figure out what exactly was going on with the CALLBACKS.BFF.[12] failures. ] [Fix cffi-lispworks bug in foreign-funcall. Luis Oliveira **20060224131735 - Add same declarations to define-foreign-funcallable as those used for define-foreign-callable. - Makes FUNCALL.FLOAT pass on linux/x86. ] [SCL port, courtesy of Douglas Crosher Luis Oliveira **20060223061757 - Makefile: new test-scl target. Add SCL's fasl file extensions to the clean target. - Remove SCL TODO item. - New file: cffi-scl.lisp. - New primitive type :long-double. Since it's only supported by SCL, it's not worth adding a no-long-double feature just yet. - New tests for :long-double. - Add information about SCL and the new :long-double type to the user manual. ] [Update TODO item about fixnums. Luis Oliveira **20060223181831] [Implement and use my_llabs instead of llabs in tests Luis Oliveira **20060223081406] [Make CLISP not throw an error on undefined functions Luis Oliveira **20060220044813 - Catch the error in %foreign-funcall and throw a warning instead. - Regression test: DEFCFUN.UNDEFINED. (CMUCL fails) ] [Fix LOOP indentation in backends. James Bielman **20060217204457 - Thanks for Luís for showing me how to configure Emacs to get this right. ] [Fix and add tests for %MEM-REF and %MEM-SET evaluation order. James Bielman **20060216083448 - Some minor reformatting of LOOP forms to pacify cl-indent. - Add new regression tests to check the evaluation order of %MEM-REF and %MEM-SET. - Add the necessary ONCE-ONLY forms to the compiler macros. - Have the CFFI-TESTS package use CFFI-SYS for testing primitives. ] [Fix typo. extend -> extent Luis Oliveira **20060216051544] [Minor change to tests/defcfun.lisp Luis Oliveira **20060216050735 - Remove unnecessary (setf (mem-ref s :char) 0) forms. ] [CFFI Manual update Luis Oliveira **20060216045547 - Update known issues (callback.bff.[12] failures). - defbitfield, foreign-bitfield-symbols and foreign-bitfield-value. - Fix typo in tutorial. defenum -> defcenum. - New section "Other Types" documenting :string, :boolean and :wrapper. - Add note about the translate-* methods not being meant to be called directly and suggesting convert-* instead. - Document the macroexpansion-time translators in a new "Optimizing Type Translators" sub-section. - Document defcenum's base-type option. - Document defcfun's varargs support. ] [New functions convert-to/from-foreign and free-converted-object Luis Oliveira **20060216045335 - These functions basically export the functionality of translate-type-to-foreign, translate-type-from-foreign and free-translated-object. - TODO: document these. ] [New TODO item: defcfun compiler macros Luis Oliveira **20060216045240] [Implement compiler macros for %MEM-REF and %MEM-SET in LispWorks. James Bielman **20060215222559 - Use FLI:FOREIGN-TYPED-AREF if available and dereferencing a number. - Avoid calling INC-POINTER when the offset is a multiple of the type size allowing direct use of the INDEX argument to FOREIGN-TYPED-AREF. - Fall back to open-coding the call to FLI:DEREFERENCE otherwise. ] [Delete LispWorks/Linux .ufsl files on make clean. James Bielman **20060215213442] [Use C limits for foreign floats and doubles instead of Lisp's. James Bielman **20060215201514 - Export 'float_min', 'float_max', 'double_min', and 'double_max' from the libtest shared library with the values of FLT_MIN, FLT_MAX, DBL_MIN, and DBL_MAX, respectively. - Use *FLOAT-MIN*, *FLOAT-MAX*, *DOUBLE-MIN*, and *DOUBLE-MAX* as test values for foreign floats/doubles instead of the Lisp constants -POSITIVE--FLOAT. ] [Oops, fix tests. Luis Oliveira **20060215174316 - MISC-TYPES.BOOLEAN.2 was using the wrong ff name. - MISC-TYPES.EXPAND.* need the expand-* methods defined at macroexpansion-time. ] [Change +REQUIRED-DLL-VERSION+ to *REQUIRED-DLL-VERSION*. James Bielman **20060215171631] [Remove completed TODO item about building 32 and 64 bit libtest. James Bielman **20060215091337] [A few more tests Luis Oliveira **20060215163551 - Tests for the new macroexpansion-time type translator interface: MISC-TYPES.EXPAND.[1234] - New tests: DEREF.LONG-LONG and DEREF.UNSIGNED-LONG-LONG. - Tests for mem-ref with non-constant type arguments: DEREF.NONCONST.* - Fix some comments in tests/memory.lisp. - New test: MISC-TYPES.BOOLEAN.2 (accepting typedefs to integer types) ] [Minor test changes Luis Oliveira **20060215030152 - Use long instead of int in the big C functions in libtest.c. - Mark callbacks.bff.[12] as expect failures for a couple of lisps. ] [Make (%callback 'non-existant-callback) signal an error Luis Oliveira **20060215025906 - Make %callback signal an error for non-existing callbacks. (allegro, sbcl and openmcl) - Regression test: callbacks.non-existant ] [Implement %MEM-REF and %MEM-SET compiler macros for Allegro CL. James Bielman **20060215025257] [Ignore RETTYPE in Allegro CL %DEFCALLBACK. James Bielman **20060215022651] [Fix float varargs promotion. Luis Oliveira **20060214042928 - Convert floats to doubles in foreign-funcall-varargs. Test DEFCFUN.VARARGS.FLOAT now passes. ] [Preliminary random tester. Luis Oliveira **20060214042733 - Push random-tester.lisp. This was used to generate the BFF tests, but is still not automated. - Update respective TODO item. ] [MORE TESTS Luis Oliveira **20060214032830 - New tests: callbacks.funcall.2, callbacks.bff.[12], defcfun.bff.[12]. - Re-enable defcfun.varargs.double. - Use #'float instead of #'coerce in the defcfun.varargs.* tests. - Fix dll version in libtest.c. ] [Change defconstant to defparameter in bindings.lisp Luis Oliveira **20060214002839] [Macroexpansion-time translators Luis Oliveira **20060214001931 - New interface. expand-to-foreign-dyn, expand-to-foreign and expand-from-foreign. - defcfun/foreign-funcall/defcallback/defcvar use this as well as the compiler macros and setf-expanders for mem-ref, mem-aref, and foreign-slot-value. - Also, parse ignore declarations and use them to avoid translating such types (thus avoiding breaking the ignore declaration). ] [Implement simple version check for libtest DLL Jack Unrue **20060212182709] [Push CFFI-FEATURES:X86-64 on Allegro/amd64. James Bielman **20060211003123] [Removed outdated comment from the colorize script. Luis Oliveira **20060210220055] [Use slanted instead of italic for VAR in style.css Luis Oliveira **20060210195248] [use lispcmt for code comments instead of @r Stephen Compall **20060210160507 - Add ROMANCOMMENTS option to manual. - Change all instances of @r to @lispcmt. ] [make Texinfo @r and colorize play nice Stephen Compall **20060210035038 - Comment out master menu entry for explain-foreign-slot-value so manual builds again. - Add @r to all Lisp comments in manual. - (Tutorial-easy_setopt): Revert previous @dots{}; it is literal `...', not an ellipsis. Oops. - In colorize-lisp-examples.lisp, remove the output generated by @r from Lisp blocks. Also shove around the process-file code some. ] [Remove @r{} in manual's code comments Luis Oliveira **20060210021117 (at least until the colorize script can cope with these) ] [More minor doc changes. Luis Oliveira **20060210005707 - @emph -> @var. - change var's style to something more sensible in the CSS file. - remove (setf callback) from the docs. - fix some typos. ] [Make :boolean accept typedefs to integer types. Luis Oliveira **20060210003650] [Minor documentation changes Luis Oliveira **20060209225109 - Have the Makefile's clean target delete some more of the temporary texinfo files. - Comment out reference to explain-foreign-slot-value. ] [reference prose in Pointers and Strings manual chapters Stephen Compall **20060209183938 - (Tutorial-easy_setopt): Use @dots{}. - (seealso): Remove old comment about @code formatting. - Pointers chapter: Describe the theory of foreign data, dealing with pointers per se, and expand on the allocation description. - (Strings): A short paragraph by way of introduction. ] [Remove note about unsupported FOREIGN-FUNCALL on Lispworks. James Bielman **20060206185439] [TAG 0.9.0 James Bielman **20060204082912] Patch bundle hash: ab7949489ec2619d50cde27034c27d9aa201e1f1