Compiling to the Virtual Machine
================================

Compilers have a mystique about them that is attractive and off-putting at the same time. They are attractive because they are magical -- they transform inert text into live results, like throwing the switch on Frankenstein's monster. However, this magic is perceived by many to be impenetrable.

This section aims to pay attention to the small man behind the curtain.

See Read/Load/Eval/Compile, if you're lost and you just wanted to know how to compile your .scm file.

Compiler Tower
--------------

Guile's compiler is quite simple, actually -- its compilers, to put it more accurately. Guile defines a tower of languages, starting at Scheme and progressively simplifying down to languages that resemble the VM instruction set (see Instruction Set).

Each language knows how to compile to the next, so each step is simple and understandable. Furthermore, this set of languages is not hardcoded into Guile, so it is possible for the user to add new high-level languages, new passes, or even different compilation targets.

Languages are registered in the module, (system base language):

     (use-modules (system base language))

They are registered with the define-language form.

 -- Scheme Syntax: define-language
     name title version reader printer
     [parser=#f] [read-file=#f] [compilers='()] [evaluator=#f]
     Define a language.

     This syntax defines a #<language> object, bound to NAME in the current environment. In addition, the language will be added to the global language set. For example, this is the language definition for Scheme:

          (define-language scheme
            #:title	"Guile Scheme"
            #:version	"0.5"
            #:reader	read
            #:read-file	read-file
            #:compilers   `((,ghil . ,compile-ghil))
            #:evaluator	(lambda (x module) (primitive-eval x))
            #:printer	write)

     In this example, from (language scheme spec), read-file reads expressions from a port and wraps them in a begin block.

The interesting thing about having languages defined this way is that they present a uniform interface to the read-eval-print loop. This allows the user to change the current language of the REPL:

     $ guile
     Guile Scheme interpreter 0.5 on Guile 1.9.0
     Copyright (C) 2001-2008 Free Software Foundation, Inc.

     Enter `,help' for help.
     scheme@(guile-user)> ,language ghil
     Guile High Intermediate Language (GHIL) interpreter 0.3 on Guile 1.9.0
     Copyright (C) 2001-2008 Free Software Foundation, Inc.

     Enter `,help' for help.
     ghil@(guile-user)>

Languages can be looked up by name, as they were above.

 -- Scheme Procedure: lookup-language name
     Looks up a language named NAME, autoloading it if necessary.

     Languages are autoloaded by looking for a variable named NAME in a module named (language NAME spec).

     The language object will be returned, or #f if there does not exist a language with that name.

Defining languages this way allows us to programmatically determine the necessary steps for compiling code from one language to another.

 -- Scheme Procedure: lookup-compilation-order from to
     Recursively traverses the set of languages to which FROM can compile, depth-first, and return the first path that can transform FROM to TO. Returns #f if no path is found.

     This function memoizes its results in a cache that is invalidated by subsequent calls to define-language, so it should be quite fast.

There is a notion of a ``current language'', which is maintained in the *current-language* fluid. This language is normally Scheme, and may be rebound by the user. The run-time compilation interfaces (see Read/Load/Eval/Compile) also allow you to choose other source and target languages.

The normal tower of languages when compiling Scheme goes like this:

  * Scheme, which we know and love

  * Guile High Intermediate Language (GHIL)

  * Guile Low Intermediate Language (GLIL)

  * Object code

Object code may be serialized to disk directly, though it has a cookie and version prepended to the front. But when compiling Scheme at run time, you want a Scheme value, e.g. a compiled procedure. For this reason, so as not to break the abstraction, Guile defines a fake language, value. Compiling to value loads the object code into a procedure, and wakes the sleeping giant.

Perhaps this strangeness can be explained by example: compile-file defaults to compiling to object code, because it produces object code that has to live in the barren world outside the Guile runtime; but compile defaults to compiling to value, as its product re-enters the Guile world.

Indeed, the process of compilation can circulate through these different worlds indefinitely, as shown by the following quine:

     ((lambda (x) ((compile x) x)) '(lambda (x) ((compile x) x)))

The Scheme Compiler
-------------------

The job of the Scheme compiler is to expand all macros and to resolve all symbols to lexical variables. Its target language, GHIL, is fairly close to Scheme itself, so this process is not very complicated.

The Scheme compiler is driven by a table of translators, declared with the define-scheme-translator form, defined in the module, (language scheme compile-ghil).

 -- Scheme Syntax: define-scheme-translator head clause1 clause2...
     The best documentation of this form is probably an example. Here is the translator for if:

          (define-scheme-translator if
            ;; (if TEST THEN [ELSE])
            ((,test ,then)
             (make-ghil-if e l (retrans test) (retrans then) (retrans '(begin))))
            ((,test ,then ,else)
             (make-ghil-if e l (retrans test) (retrans then) (retrans else))))

     The match syntax is from the pmatch macro, defined in (system base pmatch). The result of a clause should be a valid GHIL value. If no clause matches, a syntax error is signalled.

     In the body of the clauses, the following bindings are introduced:

       * e, the current environment

       * l, the current source location (or #f)

       * retrans, a procedure that may be called to compile subexpressions

     Note that translators are looked up by value, not by name. That is to say, the translator is keyed under the value of if, which normally prints as #<primitive-builtin-macro! if>.

Users can extend the compiler by defining new translators. Additionally, some forms can be inlined directly to instructions -- see Inlined Scheme Instructions, for a list. The actual inliners are defined in (language scheme inline):

 -- Scheme Syntax: define-inline head arity1 result1 arity2 result2...
     Defines an inliner for head. As in define-scheme-translator, inliners are keyed by value and not by name.

     Expressions are matched on their arities. For example:

          (define-inline eq?
            (x y) (eq? x y))

     This inlines calls to the Scheme procedure, eq?, to the instruction eq?.

     A more complicated example would be:

          (define-inline +
            () 0
            (x) x
            (x y) (add x y)
            (x y . rest) (add x (+ y . rest)))

Compilers take two arguments, an expression and an environment, and return two values as well: an expression in the target language, and an environment suitable for the target language. The format of the environment is language-dependent.

For Scheme, an environment may be one of three things:

  * #f, in which case compilation is performed in the context of the current module;

  * a module, which specifies the context of the compilation; or

  * a compile environment, which specifies lexical variables as well.

The format of a compile environment for scheme is (MODULE LEXICALS . EXTERNALS), though users are strongly discouraged from constructing these environments themselves. Instead, if you need this functionality -- as in GOOPS' dynamic method compiler -- capture an environment with compile-time-environment, then pass that environment to compile.

 -- Scheme Procedure: compile-time-environment
     A special function known to the compiler that, when compiled, will return a representation of the lexical environment in place at compile time. Useful for supporting some forms of dynamic compilation. Returns #f if called from the interpreter.

GHIL
----

Guile High Intermediate Language (GHIL) is a structured intermediate language that is close in expressive power to Scheme. It is an expanded, pre-analyzed Scheme.

GHIL is ``structured'' in the sense that its representation is based on records, not S-expressions. This gives a rigidity to the language that ensures that compiling to a lower-level language only requires a limited set of transformations. Practically speaking, consider the GHIL type, <ghil-quote>, which has fields named env, loc, and exp. Instances of this type are records created via make-ghil-quote, and whose fields are accessed as ghil-quote-env, ghil-quote-loc, and ghil-quote-exp. There is also a predicate, ghil-quote?. See Records, for more information on records.

Expressions of GHIL name their environments explicitly, and all variables are referenced by identity in addition to by name. (language ghil) defines a number of routines to deal explicitly with variables and environments:

 -- Scheme Variable: <ghil-toplevel-env> [table='()]
     A toplevel environment. The TABLE holds all toplevel variables that have been resolved in this environment.

 -- Scheme Variable: <ghil-env> parent [table='()] [variables='()]
     A lexical environment. PARENT will be the enclosing lexical environment, or a toplevel environment. TABLE holds an alist mapping symbols to variables bound in this environment, while VARIABLES holds a cumulative list of all variables ever defined in this environment.

     Lexical environments correspond to procedures. Bindings introduced e.g. by Scheme's let add to the bindings in a lexical environment. An example of a case in which a variable might be in VARIABLES but not in TABLE would be a variable that is in the same procedure, but is out of scope.

 -- Scheme Variable: <ghil-var> env name kind [index=#f]
     A variable. KIND is one of argument, local, external, toplevel, public, or private; see the procedures below for more information. INDEX is used in compilation.

 -- Scheme Procedure: ghil-var-is-bound? env sym
     Recursively look up a variable named SYM in ENV, and return it or #f if none is found.

 -- Scheme Procedure: ghil-var-for-ref! env sym
     Recursively look up a variable named SYM in ENV, and return it. If the symbol was not bound, return a new toplevel variable.

 -- Scheme Procedure: ghil-var-for-set! env sym
     Like ghil-var-for-ref!, except that the returned variable will be marked as external. See Variables and the VM.

 -- Scheme Procedure: ghil-var-define! toplevel-env sym
     Return an existing or new toplevel variable named SYM. TOPLEVEL-ENV must be a toplevel environment.

 -- Scheme Procedure: ghil-var-at-module! env modname sym interface?
     Return a variable that will be resolved at run-time with respect to a specific module named MODNAME. If INTERFACE? is true, the variable will be of type public, otherwise private.

 -- Scheme Procedure: call-with-ghil-environment env syms func
     Bind SYMS to fresh variables within a new lexical environment whose parent is ENV, and call FUNC as (FUNC NEW-ENV NEW-VARS).

 -- Scheme Procedure: call-with-ghil-bindings env syms func
     Like call-with-ghil-environment, except the existing environment ENV is re-used. For that reason, FUNC is invoked as (FUNC NEW-VARS)

In the aforementioned <ghil-quote> type, the ENV slot holds a pointer to the environment in which the expression occurs. The LOC slot holds source location information, so that errors corresponding to this expression can be mapped back to the initial expression in the higher-level language, e.g. Scheme. See Compiled Procedures, for more information on source location objects.

GHIL also has a declarative serialization format, which makes writing and reading it a tractable problem for the human mind. Since all GHIL language constructs contain env and loc pointers, they are left out of the serialization. (Serializing env structures would be difficult, as they are often circular.) What is left is the type of expression, and the remaining slots defined in the expression type.

For example, an S-expression representation of the <ghil-quote> expression would be:

     (quote 3)

It's deceptively like Scheme. The general rule is, for a type defined as <ghil-FOO> env loc SLOT1 SLOT2..., the S-expression representation will be (FOO SLOT1 SLOT2...). Users may program with this format directly at the REPL:

     scheme@(guile-user)> ,language ghil
     Guile High Intermediate Language (GHIL) interpreter 0.3 on Guile 1.9.0
     Copyright (C) 2001-2008 Free Software Foundation, Inc.

     Enter `,help' for help.
     ghil@(guile-user)> (call (ref +) (quote 32) (quote 10))
     => 42

For convenience, some slots are serialized as rest arguments; those are noted below. The other caveat is that variables are serialized as their names only, and not their identities.

 -- Scheme Variable: <ghil-void> env loc
     The unspecified value.

 -- Scheme Variable: <ghil-quote> env loc exp
     A quoted expression.

     Note that unlike in Scheme, there are no self-quoting expressions; all constants must come from quote expressions.

 -- Scheme Variable: <ghil-quasiquote> env loc exp
     A quasiquoted expression. The expression is treated as a constant, except for embedded unquote and unquote-splicing forms.

 -- Scheme Variable: <ghil-unquote> env loc exp
     Like Scheme's unquote; only valid within a quasiquote.

 -- Scheme Variable: <ghil-unquote-splicing> env loc exp
     Like Scheme's unquote-splicing; only valid within a quasiquote.

 -- Scheme Variable: <ghil-ref> env loc var
     A variable reference. Note that for purposes of serialization, VAR is serialized as its name, as a symbol.

 -- Scheme Variable: <ghil-set> env loc var val
     A variable mutation. VAR is serialized as a symbol.

 -- Scheme Variable: <ghil-define> env loc var val
     A toplevel variable definition. See ghil-var-define!.

 -- Scheme Variable: <ghil-if> env loc test then else
     A conditional. Note that ELSE is not optional.

 -- Scheme Variable: <ghil-and> env loc . exps
     Like Scheme's and.

 -- Scheme Variable: <ghil-or> env loc . exps
     Like Scheme's or.

 -- Scheme Variable: <ghil-begin> env loc . body
     Like Scheme's begin.

 -- Scheme Variable: <ghil-bind> env loc vars exprs . body
     Like a deconstructed let: each element of VARS will be bound to the corresponding GHIL expression in EXPRS.

     Note that for purposes of the serialization format, EXPRS are evaluated before the new bindings are added to the environment. For letrec semantics, there also exists a bindrec parse flavor. This is useful for writing GHIL at the REPL, but the serializer does not currently have the cleverness needed to determine whether a <ghil-bind> has let or letrec semantics, and thus only serializes <ghil-bind> as bind.

 -- Scheme Variable: <ghil-mv-bind> env loc vars rest producer . body
     Like Scheme's receive -- binds the values returned by applying producer, which should be a thunk, to the lambda-like bindings described by VARS and REST.

 -- Scheme Variable: <ghil-lambda> env loc vars rest meta . body
     A closure. VARS is the argument list, serialized as a list of symbols. REST is a boolean, which is #t iff the last argument is a rest argument. META is an association list of properties. The actual BODY should be a list of GHIL expressions.

 -- Scheme Variable: <ghil-call> env loc proc . args
     A procedure call.

 -- Scheme Variable: <ghil-mv-call> env loc producer consumer
     Like Scheme's call-with-values.

 -- Scheme Variable: <ghil-inline> env loc op . args
     An inlined VM instruction. OP should be the instruction name as a symbol, and ARGS should be its arguments, as GHIL expressions.

 -- Scheme Variable: <ghil-values> env loc . values
     Like Scheme's values.

 -- Scheme Variable: <ghil-values*> env loc . values
     VALUES are as in the Scheme expression, (apply values . VALS).

 -- Scheme Variable: <ghil-reified-env> env loc
     Produces, at run-time, a reification of the environment at compile time. Used in the implementation of Scheme's compile-time-environment.

GHIL implements a compiler to GLIL that recursively traverses GHIL expressions, writing out GLIL expressions into a linear list. The compiler also keeps some state as to whether the current expression is in tail context, and whether its value will be used in future computations. This state allows the compiler not to emit code for constant expressions that will not be used (e.g. docstrings), and to perform tail calls when in tail position.

Just as the Scheme to GHIL compiler introduced new hidden state---the environment---the GHIL to GLIL compiler introduces more state, the stack. While not represented explicitly, the stack is present in the compilation of each GHIL expression: compiling a GHIL expression should leave the run-time value stack in the same state. For example, if the intermediate value stack has two elements before evaluating an if expression, it should have two elements after that expression.

Interested readers are encouraged to read the implementation in (language ghil compile-glil) for more details.

GLIL
----

Guile Low Intermediate Language (GLIL) is a structured intermediate language whose expressions closely mirror the functionality of Guile's VM instruction set.

Its expression types are defined in (language glil), and as with GHIL, some of its fields parse as rest arguments.

 -- Scheme Variable: <glil-program> nargs nrest nlocs nexts meta . body
     A unit of code that at run-time will correspond to a compiled procedure. NARGS NREST NLOCS, and NEXTS collectively define the program's arity; see Compiled Procedures, for more information. META should be an alist of properties, as in <ghil-lambda>. BODY is a list of GLIL expressions.

 -- Scheme Variable: <glil-bind> . vars
     An advisory expression that notes a liveness extent for a set of variables. VARS is a list of (NAME TYPE INDEX), where TYPE should be either argument, local, or external.

     <glil-bind> expressions end up being serialized as part of a program's metadata and do not form part of a program's code path.

 -- Scheme Variable: <glil-mv-bind> vars rest
     A multiple-value binding of the values on the stack to VARS. Iff REST is true, the last element of VARS will be treated as a rest argument.

     In addition to pushing a binding annotation on the stack, like <glil-bind>, an expression is emitted at compilation time to make sure that there are enough values available to bind. See the notes on truncate-values in Procedural Instructions, for more information.

 -- Scheme Variable: <glil-unbind>
     Closes the liveness extent of the most recently encountered <glil-bind> or <glil-mv-bind> expression. As GLIL expressions are compiled, a parallel stack of live bindings is maintained; this expression pops off the top element from that stack.

     Bindings are written into the program's metadata so that debuggers and other tools can determine the set of live local variables at a given offset within a VM program.

 -- Scheme Variable: <glil-source> loc
     Records source information for the preceding expression. LOC should be a vector, #(LINE COLUMN FILENAME).

 -- Scheme Variable: <glil-void>
     Pushes the unspecified value on the stack.

 -- Scheme Variable: <glil-const> obj
     Pushes a constant value onto the stack. OBJ must be a number, string, symbol, keyword, boolean, character, or a pair or vector or list thereof, or the empty list.

 -- Scheme Variable: <glil-argument> op index
     Accesses an argument on the stack. If OP is ref, the argument is pushed onto the stack; if it is set, the argument is set from the top value on the stack, which is popped off.

 -- Scheme Variable: <glil-local> op index
     Like <glil-argument>, but for local variables. See Stack Layout, for more information.

 -- Scheme Variable: <glil-external> op depth index
     Accesses a heap-allocated variable, addressed by DEPTH, the nth enclosing environment, and INDEX, the variable's position within the environment. OP is ref or set.

 -- Scheme Variable: <glil-toplevel> op name
     Accesses a toplevel variable. OP may be ref, set, or define.

 -- Scheme Variable: <glil-module> op mod name public?
     Accesses a variable within a specific module. See ghil-var-at-module!, for more information.

 -- Scheme Variable: <glil-label> label
     Creates a new label. LABEL can be any Scheme value, and should be unique.

 -- Scheme Variable: <glil-branch> inst label
     Branch to a label. LABEL should be a <ghil-label>. inst is a branching instruction: br-if, br, etc.

 -- Scheme Variable: <glil-call> inst nargs
     This expression is probably misnamed, as it does not correspond to function calls. <glil-call> invokes the VM instruction named INST, noting that it is called with NARGS stack arguments. The arguments should be pushed on the stack already. What happens to the stack afterwards depends on the instruction.

 -- Scheme Variable: <glil-mv-call> nargs ra
     Performs a multiple-value call. RA is a <glil-label> corresponding to the multiple-value return address for the call. See the notes on mv-call in Procedural Instructions, for more information.

Users may enter in GLIL at the REPL as well, though there is a bit more bookkeeping to do. Since GLIL needs the set of variables to be declared explicitly in a <glil-program>, GLIL expressions must be wrapped in a thunk that declares the arity of the expression:

     scheme@(guile-user)> ,language glil
     Guile Lowlevel Intermediate Language (GLIL) interpreter 0.3 on Guile 1.9.0
     Copyright (C) 2001-2008 Free Software Foundation, Inc.

     Enter `,help' for help.
     glil@(guile-user)> (program 0 0 0 0 () (const 3) (call return 0))
     => 3

Just as in all of Guile's compilers, an environment is passed to the GLIL-to-object code compiler, and one is returned as well, along with the object code.

Object Code
-----------

Object code is the serialization of the raw instruction stream of a program, ready for interpretation by the VM. Procedures related to object code are defined in the (system vm objcode) module.

 -- Scheme Procedure: objcode? obj
 -- C Function: scm_objcode_p (obj)
     Returns #f iff OBJ is object code, #f otherwise.

 -- Scheme Procedure: bytecode->objcode bytecode nlocs nexts
 -- C Function: scm_bytecode_to_objcode (bytecode, nlocs, nexts)
     Makes a bytecode object from BYTECODE, which should be a u8vector. NLOCS and NEXTS denote the number of stack and heap variables to reserve when this objcode is executed.

 -- Scheme Variable: load-objcode file
 -- C Function: scm_load_objcode (file)
     Load object code from a file named FILE. The file will be mapped into memory via mmap, so this is a very fast operation.

     On disk, object code has an eight-byte cookie prepended to it, so that we will not execute arbitrary garbage. In addition, two more bytes are reserved for NLOCS and NEXTS.

 -- Scheme Variable: objcode->u8vector objcode
 -- C Function: scm_objcode_to_u8vector (objcode)
     Copy object code out to a u8vector for analysis by Scheme. The ten-byte header is included.

 -- Scheme Variable: objcode->program objcode [external='()]
 -- C Function: scm_objcode_to_program (objcode, external)
     Load up object code into a Scheme program. The resulting program will be a thunk that captures closure variables from EXTERNAL.

Object code from a file may be disassembled at the REPL via the meta-command ,disassemble-file, abbreviated as ,xx. Programs may be disassembled via ,disassemble, abbreviated as ,x.

Compiling object code to the fake language, value, is performed via loading objcode into a program, then executing that thunk with respect to the compilation environment. Normally the environment propagates through the compiler transparently, but users may specify the compilation environment manually as well:

 -- Scheme Procedure: make-objcode-env module externals
     Make an object code environment. MODULE should be a Scheme module, and EXTERNALS should be a list of external variables. #f is also a valid object code environment.

Extending the Compiler
----------------------

At this point, we break with the impersonal tone of the rest of the manual, and make an intervention. Admit it: if you've read this far into the compiler internals manual, you are a junkie. Perhaps a course at your university left you unsated, or perhaps you've always harbored a sublimated desire to hack the holy of computer science holies: a compiler. Well you're in good company, and in a good position. Guile's compiler needs your help.

There are many possible avenues for improving Guile's compiler. Probably the most important improvement, speed-wise, will be some form of native compilation, both just-in-time and ahead-of-time. This could be done in many ways. Probably the easiest strategy would be to extend the compiled procedure structure to include a pointer to a native code vector, and compile from bytecode to native code at run-time after a procedure is called a certain number of times.

The name of the game is a profiling-based harvest of the low-hanging fruit, running programs of interest under a system-level profiler and determining which improvements would give the most bang for the buck. There are many well-known efficiency hacks in the literature: Dybvig's letrec optimization, individual boxing of heap-allocated values (and then store the boxes on the stack directory), optimized case-lambda expressions, stack underflow and overflow handlers, etc. Highly recommended papers: Dybvig's HOCS, Ghuloum's compiler paper.

The compiler also needs help at the top end, enhancing the Scheme that it knows to also understand R6RS, and adding new high-level compilers: Emacs Lisp, Lua, JavaScript...