guile/doc/ref/compiler.texi

@c -*-texinfo-*-
@c This is part of the GNU Guile Reference Manual.
@c Copyright (C)  2008
@c   Free Software Foundation, Inc.
@c See the file guile.texi for copying conditions.

@node Compiling to the Virtual Machine
@section Compiling to the Virtual Machine

Compilers have a mystique about them that is attractive and
off-putting at the same time. They are attractive because they are
magical -- they transform inert text into live results, like throwing
the switch on Frankenstein's monster. However, this magic is perceived
by many to be impenetrable.

This section aims to pay attention to the small man behind the
curtain.

@xref{Read/Load/Eval/Compile}, if you're lost and you just wanted to
know how to compile your .scm file.

@menu
* Compiler Tower::                   
* The Scheme Compiler::                   
* Tree-IL::                 
* GLIL::                
* Assembly::                   
* Bytecode and Objcode::                   
* Extending the Compiler::
@end menu

@node Compiler Tower
@subsection Compiler Tower

Guile's compiler is quite simple, actually -- its @emph{compilers}, to
put it more accurately. Guile defines a tower of languages, starting
at Scheme and progressively simplifying down to languages that
resemble the VM instruction set (@pxref{Instruction Set}).

Each language knows how to compile to the next, so each step is simple
and understandable. Furthermore, this set of languages is not
hardcoded into Guile, so it is possible for the user to add new
high-level languages, new passes, or even different compilation
targets.

Languages are registered in the module, @code{(system base language)}:

@example
(use-modules (system base language))
@end example

They are registered with the @code{define-language} form.

@deffn {Scheme Syntax} define-language @
name title version reader printer @
[parser=#f] [compilers='()] [decompilers='()] [evaluator=#f]
Define a language.

This syntax defines a @code{#<language>} object, bound to @var{name}
in the current environment. In addition, the language will be added to
the global language set. For example, this is the language definition
for Scheme:

@example
(define-language scheme
  #:title       "Guile Scheme"
  #:version     "0.5"
  #:reader      read
  #:compilers   `((tree-il . ,compile-tree-il)
                  (ghil . ,compile-ghil))
  #:decompilers `((tree-il . ,decompile-tree-il))
  #:evaluator   (lambda (x module) (primitive-eval x))
  #:printer     write)
@end example
@end deffn

The interesting thing about having languages defined this way is that
they present a uniform interface to the read-eval-print loop. This
allows the user to change the current language of the REPL:

@example
$ guile
Guile Scheme interpreter 0.5 on Guile 1.9.0
Copyright (C) 2001-2008 Free Software Foundation, Inc.

Enter `,help' for help.
scheme@@(guile-user)> ,language tree-il
Tree Intermediate Language interpreter 1.0 on Guile 1.9.0
Copyright (C) 2001-2008 Free Software Foundation, Inc.

Enter `,help' for help.
tree-il@@(guile-user)> 
@end example

Languages can be looked up by name, as they were above.

@deffn {Scheme Procedure} lookup-language name
Looks up a language named @var{name}, autoloading it if necessary.

Languages are autoloaded by looking for a variable named @var{name} in
a module named @code{(language @var{name} spec)}.

The language object will be returned, or @code{#f} if there does not
exist a language with that name.
@end deffn

Defining languages this way allows us to programmatically determine
the necessary steps for compiling code from one language to another.

@deffn {Scheme Procedure} lookup-compilation-order from to
Recursively traverses the set of languages to which @var{from} can
compile, depth-first, and return the first path that can transform
@var{from} to @var{to}. Returns @code{#f} if no path is found.

This function memoizes its results in a cache that is invalidated by
subsequent calls to @code{define-language}, so it should be quite
fast.
@end deffn

There is a notion of a ``current language'', which is maintained in
the @code{*current-language*} fluid. This language is normally Scheme,
and may be rebound by the user. The run-time compilation interfaces
(@pxref{Read/Load/Eval/Compile}) also allow you to choose other source
and target languages.

The normal tower of languages when compiling Scheme goes like this:

@itemize
@item Scheme, which we know and love
@item Tree Intermediate Language (Tree-IL)
@item Guile Low Intermediate Language (GLIL)
@item Assembly
@item Bytecode
@item Objcode
@end itemize

Object code may be serialized to disk directly, though it has a cookie
and version prepended to the front. But when compiling Scheme at run
time, you want a Scheme value: for example, a compiled procedure. For
this reason, so as not to break the abstraction, Guile defines a fake
language at the bottom of the tower:

@itemize
@item Value
@end itemize

Compiling to @code{value} loads the object code into a procedure, and
wakes the sleeping giant.

Perhaps this strangeness can be explained by example:
@code{compile-file} defaults to compiling to object code, because it
produces object code that has to live in the barren world outside the
Guile runtime; but @code{compile} defaults to compiling to
@code{value}, as its product re-enters the Guile world.

Indeed, the process of compilation can circulate through these
different worlds indefinitely, as shown by the following quine:

@example
((lambda (x) ((compile x) x)) '(lambda (x) ((compile x) x)))
@end example

@node The Scheme Compiler
@subsection The Scheme Compiler

The job of the Scheme compiler is to expand all macros and all of
Scheme to its most primitive expressions. The definition of
``primitive'' is given by the inventory of constructs provided by
Tree-IL, the target language of the Scheme compiler: procedure
applications, conditionals, lexical references, etc. This is described
more fully in the next section.

The tricky and amusing thing about the Scheme-to-Tree-IL compiler is
that it is completely implemented by the macro expander. Since the
macro expander has to run over all of the source code already in order
to expand macros, it might as well do the analysis at the same time,
producing Tree-IL expressions directly.

Because this compiler is actually the macro expander, it is
extensible. Any macro which the user writes becomes part of the
compiler.

The Scheme-to-Tree-IL expander may be invoked using the generic
@code{compile} procedure:

@lisp
(compile '(+ 1 2) #:from 'scheme #:to 'tree-il)
@result{}
 #<<application> src: #f
                 proc: #<<toplevel-ref> src: #f name: +>
                 args: (#<<const> src: #f exp: 1>
                        #<<const> src: #f exp: 2>)>
@end lisp

Or, since Tree-IL is so close to Scheme, it is often useful to expand
Scheme to Tree-IL, then translate back to Scheme. For that reason the
expander provides two interfaces. The former is equivalent to calling
@code{(sc-expand '(+ 1 2) 'c)}, where the @code{'c} is for
``compile''. With @code{'e} (the default), the result is translated
back to Scheme:

@lisp
(sc-expand '(+ 1 2))
@result{} (+ 1 2)
(sc-expand '(let ((x 10)) (* x x)))
@result{} (let ((x84 10)) (* x84 x84))
@end lisp

The second example shows that as part of its job, the macro expander
renames lexically-bound variables. The original names are preserved
when compiling to Tree-IL, but can't be represented in Scheme: a
lexical binding only has one name. It is for this reason that the
@emph{native} output of the expander is @emph{not} Scheme. There's too
much information we would lose if we translated to Scheme directly:
lexical variable names, source locations, and module hygiene.

Note however that @code{sc-expand} does not have the same signature as
@code{compile-tree-il}. @code{compile-tree-il} is a small wrapper
around @code{sc-expand}, to make it conform to the general form of
compiler procedures in Guile's language tower.

Compiler procedures take two arguments, an expression and an
environment. They return three values: the compiled expression, the
corresponding environment for the target language, and a
``continuation environment''. The compiled expression and environment
will serve as input to the next language's compiler. The
``continuation environment'' can be used to compile another expression
from the same source language within the same module.

For example, you might compile the expression, @code{(define-module
(foo))}. This will result in a Tree-IL expression and environment. But
if you compiled a second expression, you would want to take into
account the compile-time effect of compiling the previous expression,
which puts the user in the @code{(foo)} module. That is purpose of the
``continuation environment''; you would pass it as the environment
when compiling the subsequent expression.

For Scheme, an environment may be one of two things:
@itemize
@item @code{#f}, in which case compilation is performed in the context
of the current module; or
@item a module, which specifies the context of the compilation.
@end itemize

@node Tree-IL
@subsection Tree-IL

Tree Intermediate Language (Tree-IL) is a structured intermediate
language that is close in expressive power to Scheme. It is an
expanded, pre-analyzed Scheme.

Tree-IL is ``structured'' in the sense that its representation is
based on records, not S-expressions. This gives a rigidity to the
language that ensures that compiling to a lower-level language only
requires a limited set of transformations. Practically speaking,
consider the Tree-IL type, @code{<const>}, which has two fields,
@code{src} and @code{exp}. Instances of this type are records created
via @code{make-const}, and whose fields are accessed as
@code{const-src}, and @code{const-exp}. There is also a predicate,
@code{const?}. @xref{Records}, for more information on records.

@c alpha renaming

All Tree-IL types have a @code{src} slot, which holds source location
information for the expression. This information, if present, will be
residualized into the compiled object code, allowing backtraces to
show source information. The format of @code{src} is the same as that
returned by Guile's @code{source-properties} function. @xref{Source
Properties}, for more information.

Although Tree-IL objects are represented internally using records,
there is also an equivalent S-expression external representation for
each kind of Tree-IL. For example, an the S-expression representation
of @code{#<const src: #f exp: 3>} expression would be:

@example
(const 3)
@end example

Users may program with this format directly at the REPL:

@example
scheme@@(guile-user)> ,language tree-il
Tree Intermediate Language interpreter 1.0 on Guile 1.9.0
Copyright (C) 2001-2008 Free Software Foundation, Inc.

Enter `,help' for help.
tree-il@@(guile-user)> (apply (primitive +) (const 32) (const 10))
@result{} 42
@end example

The @code{src} fields are left out of the external representation.

@deftp {Scheme Variable} <void> src
@deftpx {External Representation} (void)
An empty expression. In practice, equivalent to Scheme's @code{(if #f
#f)}.
@end deftp
@deftp {Scheme Variable} <const> src exp
@deftpx {External Representation} (const @var{exp})
A constant.
@end deftp
@deftp {Scheme Variable} <primitive-ref> src name
@deftpx {External Representation} (primitive @var{name})
A reference to a ``primitive''. A primitive is a procedure that, when
compiled, may be open-coded. For example, @code{cons} is usually
recognized as a primitive, so that it compiles down to a single
instruction.

Compilation of Tree-IL usually begins with a pass that resolves some
@code{<module-ref>} and @code{<toplevel-ref>} expressions to
@code{<primitive-ref>} expressions. The actual compilation pass
has special cases for applications of certain primitives, like
@code{apply} or @code{cons}.
@end deftp
@deftp {Scheme Variable} <lexical-ref> src name gensym
@deftpx {External Representation} (lexical @var{name} @var{gensym})
A reference to a lexically-bound variable. The @var{name} is the
original name of the variable in the source program. @var{gensym} is a
unique identifier for this variable.
@end deftp
@deftp {Scheme Variable} <lexical-set> src name gensym exp
@deftpx {External Representation} (set! (lexical @var{name} @var{gensym}) @var{exp})
Sets a lexically-bound variable.
@end deftp
@deftp {Scheme Variable} <module-ref> src mod name public?
@deftpx {External Representation} (@@ @var{mod} @var{name})
@deftpx {External Representation} (@@@@ @var{mod} @var{name})
A reference to a variable in a specific module. @var{mod} should be
the name of the module, e.g. @code{(guile-user)}.

If @var{public?} is true, the variable named @var{name} will be looked
up in @var{mod}'s public interface, and serialized with @code{@@};
otherwise it will be looked up among the module's private bindings,
and is serialized with @code{@@@@}.
@end deftp
@deftp {Scheme Variable} <module-set> src mod name public? exp
@deftpx {External Representation} (set! (@@ @var{mod} @var{name}) @var{exp})
@deftpx {External Representation} (set! (@@@@ @var{mod} @var{name}) @var{exp})
Sets a variable in a specific module.
@end deftp
@deftp {Scheme Variable} <toplevel-ref> src name
@deftpx {External Representation} (toplevel @var{name})
References a variable from the current procedure's module.
@end deftp
@deftp {Scheme Variable} <toplevel-set> src name exp
@deftpx {External Representation} (set! (toplevel @var{name}) @var{exp})
Sets a variable in the current procedure's module.
@end deftp
@deftp {Scheme Variable} <toplevel-define> src name exp
@deftpx {External Representation} (define (toplevel @var{name}) @var{exp})
Defines a new top-level variable in the current procedure's module.
@end deftp
@deftp {Scheme Variable} <conditional> src test then else
@deftpx {External Representation} (if @var{test} @var{then} @var{else})
A conditional. Note that @var{else} is not optional.
@end deftp
@deftp {Scheme Variable} <application> src proc args
@deftpx {External Representation} (apply @var{proc} . @var{args})
A procedure call.
@end deftp
@deftp {Scheme Variable} <sequence> src exps
@deftpx {External Representation} (begin . @var{exps})
Like Scheme's @code{begin}.
@end deftp
@deftp {Scheme Variable} <lambda> src names vars meta body
@deftpx {External Representation} (lambda @var{names} @var{vars} @var{meta} @var{body})
A closure. @var{names} is original binding form, as given in the
source code, which may be an improper list. @var{vars} are gensyms
corresponding to the @var{names}. @var{meta} is an association list of
properties. The actual @var{body} is a single Tree-IL expression.
@end deftp
@deftp {Scheme Variable} <let> src names vars vals exp
@deftpx {External Representation} (let @var{names} @var{vars} @var{vals} @var{exp})
Lexical binding, like Scheme's @code{let}. @var{names} are the
original binding names, @var{vars} are gensyms corresponding to the
@var{names}, and @var{vals} are Tree-IL expressions for the values.
@var{exp} is a single Tree-IL expression.
@end deftp
@deftp {Scheme Variable} <letrec> src names vars vals exp
@deftpx {External Representation} (letrec @var{names} @var{vars} @var{vals} @var{exp})
A version of @code{<let>} that creates recursive bindings, like
Scheme's @code{letrec}.
@end deftp

@c FIXME -- need to revive this one
@c @deftp {Scheme Variable} <ghil-mv-bind> src vars rest producer . body
@c Like Scheme's @code{receive} -- binds the values returned by
@c applying @code{producer}, which should be a thunk, to the
@c @code{lambda}-like bindings described by @var{vars} and @var{rest}.
@c @end deftp

Tree-IL implements a compiler to GLIL that recursively traverses
Tree-IL expressions, writing out GLIL expressions into a linear list.
The compiler also keeps some state as to whether the current
expression is in tail context, and whether its value will be used in
future computations. This state allows the compiler not to emit code
for constant expressions that will not be used (e.g. docstrings), and
to perform tail calls when in tail position.

In the future, there will be a pass at the beginning of the
Tree-IL->GLIL compilation step to perform inlining, copy propagation,
dead code elimination, and constant folding.

Interested readers are encouraged to read the implementation in
@code{(language tree-il compile-glil)} for more details.

@node GLIL
@subsection GLIL

Guile Low Intermediate Language (GLIL) is a structured intermediate
language whose expressions more closely approximate Guile's VM
instruction set.

Its expression types are defined in @code{(language glil)}, and as
with GHIL, some of its fields parse as rest arguments.

@deftp {Scheme Variable} <glil-program> nargs nrest nlocs nexts meta . body
A unit of code that at run-time will correspond to a compiled
procedure. @var{nargs} @var{nrest} @var{nlocs}, and @var{nexts}
collectively define the program's arity; see @ref{Compiled
Procedures}, for more information. @var{meta} should be an alist of
properties, as in Tree IL's @code{<lambda>}. @var{body} is a list of
GLIL expressions.
@end deftp
@deftp {Scheme Variable} <glil-bind> . vars
An advisory expression that notes a liveness extent for a set of
variables. @var{vars} is a list of @code{(@var{name} @var{type}
@var{index})}, where @var{type} should be either @code{argument},
@code{local}, or @code{external}.

@code{<glil-bind>} expressions end up being serialized as part of a
program's metadata and do not form part of a program's code path.
@end deftp
@deftp {Scheme Variable} <glil-mv-bind> vars rest
A multiple-value binding of the values on the stack to @var{vars}. Iff
@var{rest} is true, the last element of @var{vars} will be treated as
a rest argument.

In addition to pushing a binding annotation on the stack, like
@code{<glil-bind>}, an expression is emitted at compilation time to
make sure that there are enough values available to bind. See the
notes on @code{truncate-values} in @ref{Procedural Instructions}, for
more information.
@end deftp
@deftp {Scheme Variable} <glil-unbind>
Closes the liveness extent of the most recently encountered
@code{<glil-bind>} or @code{<glil-mv-bind>} expression. As GLIL
expressions are compiled, a parallel stack of live bindings is
maintained; this expression pops off the top element from that stack.

Bindings are written into the program's metadata so that debuggers and
other tools can determine the set of live local variables at a given
offset within a VM program.
@end deftp
@deftp {Scheme Variable} <glil-source> loc
Records source information for the preceding expression. @var{loc}
should be an association list of containing @code{line} @code{column},
and @code{filename} keys, e.g. as returned by
@code{source-properties}.
@end deftp
@deftp {Scheme Variable} <glil-void>
Pushes the unspecified value on the stack.
@end deftp
@deftp {Scheme Variable} <glil-const> obj
Pushes a constant value onto the stack. @var{obj} must be a number,
string, symbol, keyword, boolean, character, the empty list, or a pair
or vector of constants.
@end deftp
@deftp {Scheme Variable} <glil-local> op index
Accesses a lexically bound variable from the stack. If @var{op} is
@code{ref}, the value is pushed onto the stack; if it is @code{set},
the variable is set from the top value on the stack, which is popped
off. @xref{Stack Layout}, for more information.
@end deftp
@deftp {Scheme Variable} <glil-external> op depth index
Accesses a heap-allocated variable, addressed by @var{depth}, the nth
enclosing environment, and @var{index}, the variable's position within
the environment. @var{op} is @code{ref} or @code{set}.
@end deftp
@deftp {Scheme Variable} <glil-toplevel> op name
Accesses a toplevel variable. @var{op} may be @code{ref}, @code{set},
or @code{define}.
@end deftp
@deftp {Scheme Variable} <glil-module> op mod name public?
Accesses a variable within a specific module. See Tree-IL's
@code{<module-ref>}, for more information.
@end deftp
@deftp {Scheme Variable} <glil-label> label
Creates a new label. @var{label} can be any Scheme value, and should
be unique.
@end deftp
@deftp {Scheme Variable} <glil-branch> inst label
Branch to a label. @var{label} should be a @code{<ghil-label>}.
@code{inst} is a branching instruction: @code{br-if}, @code{br}, etc.
@end deftp
@deftp {Scheme Variable} <glil-call> inst nargs
This expression is probably misnamed, as it does not correspond to
function calls. @code{<glil-call>} invokes the VM instruction named
@var{inst}, noting that it is called with @var{nargs} stack arguments.
The arguments should be pushed on the stack already. What happens to
the stack afterwards depends on the instruction.
@end deftp
@deftp {Scheme Variable} <glil-mv-call> nargs ra
Performs a multiple-value call. @var{ra} is a @code{<glil-label>}
corresponding to the multiple-value return address for the call. See
the notes on @code{mv-call} in @ref{Procedural Instructions}, for more
information.
@end deftp

Users may enter in GLIL at the REPL as well, though there is a bit
more bookkeeping to do. Since GLIL needs the set of variables to be
declared explicitly in a @code{<glil-program>}, GLIL expressions must
be wrapped in a thunk that declares the arity of the expression:

@example
scheme@@(guile-user)> ,language glil
Guile Lowlevel Intermediate Language (GLIL) interpreter 0.3 on Guile 1.9.0
Copyright (C) 2001-2008 Free Software Foundation, Inc.

Enter `,help' for help.
glil@@(guile-user)> (program 0 0 0 0 () (const 3) (call return 0))
@result{} 3
@end example

Just as in all of Guile's compilers, an environment is passed to the
GLIL-to-object code compiler, and one is returned as well, along with
the object code.

@node Assembly
@subsection Assembly

Assembly is an S-expression-based, human-readable representation of
the actual bytecodes that will be emitted for the VM. As such, it is a
useful intermediate language both for compilation and for
decompilation.

Besides the fact that it is not a record-based language, assembly
differs from GLIL in four main ways:

@itemize
@item Labels have been resolved to byte offsets in the program.
@item Constants inside procedures have either been expressed as inline
instructions, and possibly cached in object arrays.
@item Procedures with metadata (source location information, liveness
extents, procedure names, generic properties, etc) have had their
metadata serialized out to thunks.
@item All expressions correspond directly to VM instructions -- i.e.,
there is no @code{<glil-local>} which can be a ref or a set.
@end itemize

Assembly is isomorphic to the bytecode that it compiles to. You can
compile to bytecode, then decompile back to assembly, and you have the
same assembly code.

The general form of assembly instructions is the following:

@lisp
(@var{inst} @var{arg} ...)
@end lisp

The @var{inst} names a VM instruction, and its @var{arg}s will be
embedded in the instruction stream. The easiest way to see assembly is
to play around with it at the REPL, as can be seen in this annotated
example:

@example
scheme@@(guile-user)> (compile '(lambda (x) (+ x x)) #:to 'assembly)
(load-program 0 0 0 0
  () ; Labels
  60 ; Length
  #f ; Metadata
  (make-false) ; object table for the returned lambda
  (nop)
  (nop) ; Alignment. Since assembly has already resolved its labels
  (nop) ; to offsets, and programs must be 8-byte aligned since their
  (nop) ; object code is mmap'd directly to structures, assembly
  (nop) ; has to have the alignment embedded in it.
  (nop) 
  (load-program 1 0 0 0 
    ()
    6
    ; This is the metadata thunk for the returned procedure.
    (load-program 0 0 0 0 () 21 #f
      (load-symbol "x")  ; Name and liveness extent for @code{x}.
      (make-false)
      (make-int8:0) ; Some instruction+arg combinations
      (make-int8:0) ; have abbreviations.
      (make-int8 6)
      (list 0 5)
      (list 0 1)
      (make-eol)
      (list 0 2)
      (return))
    ; And here, the actual code.
    (local-ref 0)
    (local-ref 0)
    (add)
    (return))
  ; Return our new procedure.
  (return))
@end example

Of course you can switch the REPL to assembly and enter in assembly
S-expressions directly, like with other languages, though it is more
difficult, given that the length fields have to be correct.

@node Bytecode and Objcode
@subsection Bytecode and Objcode

Finally, the raw bytes. There are actually two different ``languages''
here, corresponding to two different ways to represent the bytes.

``Bytecode'' represents code as uniform byte vectors, useful for
structuring and destructuring code on the Scheme level. Bytecode is
the next step down from assembly:

@example
scheme@@(guile-user)> (compile '(+ 32 10) #:to 'assembly)
@result{} (load-program 0 0 0 0 () 6 #f
       (make-int8 32) (make-int8 10) (add) (return))
scheme@@(guile-user)> (compile '(+ 32 10) #:to 'bytecode)
@result{} #u8(0 0 0 0 6 0 0 0 0 0 0 0 10 32 10 10 100 48)
@end example

``Objcode'' is bytecode, but mapped directly to a C structure,
@code{struct scm_objcode}:

@example
struct scm_objcode @{
  scm_t_uint8 nargs;
  scm_t_uint8 nrest;
  scm_t_uint8 nlocs;
  scm_t_uint8 nexts;
  scm_t_uint32 len;
  scm_t_uint32 metalen;
  scm_t_uint8 base[0];
@};
@end example

As one might imagine, objcode imposes a minimum length on the
bytecode. Also, the multibyte fields are in native endianness, which
makes objcode (and bytecode) system-dependent. Indeed, in the short
example above, all but the last 5 bytes were the program's header.

Objcode also has a couple of important efficiency hacks. First,
objcode may be mapped directly from disk, allowing compiled code to be
loaded quickly, often from the system's disk cache, and shared among
multiple processes. Secondly, objcode may be embedded in other
objcode, allowing procedures to have the text of other procedures
inlined into their bodies, without the need for separate allocation of
the code. Of course, the objcode object itself does need to be
allocated.

Procedures related to objcode are defined in the @code{(system vm
objcode)} module.

@deffn {Scheme Procedure} objcode? obj
@deffnx {C Function} scm_objcode_p (obj)
Returns @code{#f} iff @var{obj} is object code, @code{#f} otherwise.
@end deffn

@deffn {Scheme Procedure} bytecode->objcode bytecode
@deffnx {C Function} scm_bytecode_to_objcode (bytecode,)
Makes a bytecode object from @var{bytecode}, which should be a
@code{u8vector}.
@end deffn

@deffn {Scheme Variable} load-objcode file
@deffnx {C Function} scm_load_objcode (file)
Load object code from a file named @var{file}. The file will be mapped
into memory via @code{mmap}, so this is a very fast operation.

On disk, object code has an eight-byte cookie prepended to it, to
prevent accidental loading of arbitrary garbage.
@end deffn

@deffn {Scheme Variable} write-objcode objcode file
@deffnx {C Function} scm_write_objcode (objcode)
Write object code out to a file, prepending the eight-byte cookie.
@end deffn

@deffn {Scheme Variable} objcode->u8vector objcode
@deffnx {C Function} scm_objcode_to_u8vector (objcode)
Copy object code out to a @code{u8vector} for analysis by Scheme.
@end deffn

The following procedure is actually in @code{(system vm program)}, but
we'll mention it here:

@deffn {Scheme Variable} make-program objcode objtable [external='()]
@deffnx {C Function} scm_make_program (objcode, objtable, external)
Load up object code into a Scheme program. The resulting program will
have @var{objtable} as its object table, which should be a vector or
@code{#f}, and will capture the closure variables from @var{external}.
@end deffn

Object code from a file may be disassembled at the REPL via the
meta-command @code{,disassemble-file}, abbreviated as @code{,xx}.
Programs may be disassembled via @code{,disassemble}, abbreviated as
@code{,x}.

Compiling object code to the fake language, @code{value}, is performed
via loading objcode into a program, then executing that thunk with
respect to the compilation environment. Normally the environment
propagates through the compiler transparently, but users may specify
the compilation environment manually as well:

@deffn {Scheme Procedure} make-objcode-env module externals
Make an object code environment. @var{module} should be a Scheme
module, and @var{externals} should be a list of external variables.
@code{#f} is also a valid object code environment.
@end deffn

@node Extending the Compiler
@subsection Extending the Compiler

At this point, we break with the impersonal tone of the rest of the
manual, and make an intervention. Admit it: if you've read this far
into the compiler internals manual, you are a junkie. Perhaps a course
at your university left you unsated, or perhaps you've always harbored
a sublimated desire to hack the holy of computer science holies: a
compiler. Well you're in good company, and in a good position. Guile's
compiler needs your help.

There are many possible avenues for improving Guile's compiler.
Probably the most important improvement, speed-wise, will be some form
of native compilation, both just-in-time and ahead-of-time. This could
be done in many ways. Probably the easiest strategy would be to extend
the compiled procedure structure to include a pointer to a native code
vector, and compile from bytecode to native code at run-time after a
procedure is called a certain number of times.

The name of the game is a profiling-based harvest of the low-hanging
fruit, running programs of interest under a system-level profiler and
determining which improvements would give the most bang for the buck.
There are many well-known efficiency hacks in the literature: Dybvig's
letrec optimization, individual boxing of heap-allocated values (and
then store the boxes on the stack directly), optimized case-lambda
expressions, stack underflow and overflow handlers, etc. Highly
recommended papers: Dybvig's HOCS, Ghuloum's compiler paper.

The compiler also needs help at the top end, enhancing the Scheme that
it knows to also understand R6RS, and adding new high-level compilers:
Emacs Lisp, Lua, JavaScript...