write history.texi

* doc/ref/vm.texi: Flesh out the VM documentation, adding a rationale.

* doc/ref/history.texi: Write the Guile history.

* doc/ref/guile.texi (Top): Some tweaks.

doc/ref/guile.texi

@@ -177,7 +177,7 @@ x
 
 * Guile Modules::
 
-* History and Implementation Details::
+* Guile Implementation::
 
 * Autoconf Support::
 
@@ -367,15 +367,16 @@ available through both Scheme and C interfaces.
 
 @include autoconf.texi
 
-@node History and Implementation Details
+@node Guile Implementation
-@chapter History and Implementation Details
+@chapter Guile Implementation
 
-Some mumblings about Guile as an artifact of historical processes;
+Things that aren't necessary to know to use guile, but that are
-knowledge of this history useful when hacking the source code.
+interesting once you decide that Guile is interesting.
-Libguile as the end product of 
+
+Also Schemers as closet compiler writers.
 
 @menu
-* A Brief History of Guile::            Foo.
+* History::                             A brief history of Guile.
 * Data Representation in Scheme::       Why things aren't just totally
                                         straightforward, in general terms.
 * The Libguile Runtime Environment::    Low-level details on Guile's C

doc/ref/history.texi

@@ -4,43 +4,280 @@
 @c   Free Software Foundation, Inc.
 @c See the file guile.texi for copying conditions.
 
-@node A Brief History of Guile
+@node History
 @section A Brief History of Guile
 
+Guile is an artifact of historical processes, both as code and as a
+community of hackers. It is sometimes useful to know this history when
+hacking the source code, to know about past decisions and future
+directions.
+
+Of course, the real history of Guile is written by the hackers hacking
+and not the writers writing, so we round up the section with a note on
+current status and future directions.
+
 @menu
-* In the Beginning There Was Emacs::  
+* The Emacs Thesis::  
-* The Tcl Wars::                
 * Early Days::                  
-* Adolescence::                 
+* A Scheme of Many Maintainers::  
-* Maturity::                    
+* A Timeline of Selected Guile Releases::  
+* Status::
 @end menu
 
-@node In the Beginning There Was Emacs
+@node The Emacs Thesis
-@subsection In the Beginning, There Was Emacs
+@subsection The Emacs Thesis
 
-@node The Tcl Wars
+The story of Guile is the story of bringing the development experience
-@subsection The ``Tcl Wars''
+of Emacs to the mass of programs on a GNU system.
+
+Emacs, when it was first created in its GNU form in 1984, was a new
+take on the problem of ``how to make a program''. The Emacs thesis is
+that it is delightful to create composite programs based on an
+orthogonal kernel written in a low-level language together with a
+powerful, high-level extension language.
+
+Extension languages foster extensible programs, programs which adapt
+readily to different users and to changing times. Proof of this can be
+seen in Emacs' current and continued existence, spanning more than a
+quarter-century.
+
+Besides providing for modification of a program by others, extension
+languages are good for /intension/ as well. Programs built in ``the
+Emacs way'' are pleasurable and easy for their authors to flesh out
+with the features that they need.
+
+After the Emacs experience was appreciated more widely, a number of
+hackers started to consider how to spread this experience to the rest
+of the GNU system. It was clear that the easiest way to Emacsify a
+program would be to embed a shared language implementation into it.
 
 @node Early Days
 @subsection Early Days
 
-The naming (scheme, plan, connive)
+Tom Lord was the first to fully concentrate his efforts on an
+embeddable language runtime, which he named ``GEL'', the GNU Extension
+Language.
 
-GEL -> GUILE -> Guile
+GEL was the product of converting SCM, Aubrey Jaffer's implementation
+of Scheme, into something more appropriate to embedding as a library.
+(SCM was itself based on an implementation by George Carrette, SIOD).
 
-Multilingual vision
+Lord managed to convince Richard Stallman to dub GEL the official
+extension language for the GNU project. It was a natural fit, given
+that Scheme was a cleaner, more modern Lisp than Emacs Lisp. Part of
+the argument was that eventually when GEL became more capable, it
+could gain the ability to execute other languages, especially Emacs
+Lisp.
 
-@node Adolescence
+Due to a naming conflict with another programming language, Jim Blandy
-@subsection Adolescence
+suggested a new name for GEL: ``Guile''. Besides being a recursive
+acroymn, ``Guile'' craftily follows the naming of its ancestors,
+``Planner'', ``Conniver'', and ``Schemer''. (The latter was truncated
+to ``Scheme'' due to a 6-character file name limit on an old operating
+system.) Finally, ``Guile'' suggests ``guy-ell'', or ``Guy L.
+Steele'', who, together with Gerald Sussman, originally discovered
+Scheme.
 
-GOOPS
+Around the same time that Guile (then GEL) was readying itself for
+public release, another extension language was gaining in popularity,
+Tcl. Many developers found advantages in Tcl because of its shell-like
+syntax and its well-developed graphical widgets library, Tk. Also, at
+the time there was a large marketing push promoting Tcl as a
+``universal extension language''.
 
-the module system
+Richard Stallman, as the primary author of GNU Emacs, had a particular
+vision of what extension languages should be, and Tcl did not seem to
+him to be as capable as Emacs Lisp. He posted a criticism to the
+comp.lang.tcl newsgroup, sparking one of the internet's legendary
+flamewars. As part of these discussions, retrospectively dubbed the
+``Tcl Wars'', he announced the Free Software Foundation's intent to
+promote Guile as the extension language for the GNU project.
 
-@node Maturity
+It is a common misconception that Guile was created as a reaction to
-@subsection Maturity
+Tcl. While it is true that the public announcement of Guile happened
+at the same time as the ``Tcl wars'', Guile was created out of a
+condition that existed outside the polemic. Indeed, the need for a
+powerful language to bridge the gap between extension of existing
+applications and a more fully dynamic programming environment is still
+with us today.
 
-1.6, 1.8, ...
+@node A Scheme of Many Maintainers
+@subsection A Scheme of Many Mantainers
 
-pthreads
+Surveying the field, it seems that Scheme implementations correspond
+with their maintainers on an N-to-1 relationship. That is to say, that
+those people that implement Schemes might do so on a number of
+occasions, but that the lifetime of a given Scheme is tied to the
+maintainership of one individual.
 
+Guile is atypical in this regard.
+
+Tom Lord maintaned Guile for its first year and a half or so,
+corresponding to the end of 1994 through the middle of 1996. The
+releases made in this time constitute an arc from SCM as a standalone
+program to Guile as a reusable, embeddable library, but passing
+through a explosion of features: embedded Tcl and Tk, a toolchain for
+compiling and disassembling Java, addition of a C-like syntax,
+creation of a module system, and a start at a rich POSIX interface.
+
+Only some of those features remain in Guile. There were ongoing
+tensions between providing a small, embeddable language, and one which
+had all of the features (e.g. a graphical toolkit) that a modern Emacs
+might need. In the end, as Guile gained in uptake, the development
+team decided to focus on depth, documentation and orthogonality rather
+than on breadth. This has been the focus of Guile ever since, although
+there is a wide range of third-party libraries for Guile.
+
+Jim Blandy presided over that period of stabilization, in the three
+years until the end of 1999, when he too moved on to other projects.
+Since then, Guile has had a group maintainership. The first group was
+Maciej Stachowiak, Mikael Djurfeldt, and Marius Vollmer, with Vollmer
+staying on the longest. By late 2007, Vollmer had mostly moved on to
+other things, so Neil Jerram and Ludovic Courtès stepped up to take on
+the primary maintenance responsibility.
+
+Of course, a large part of the actual work on Guile has come from
+other contributors too numerous to mention, but without whom the world
+would be a poorer place.
+
+@node A Timeline of Selected Guile Releases
+@subsection A Timeline of Selected Guile Releases
+
+@table @asis
+@item guile-i --- 4 February 1995
+SCM, turned into a library.
+
+@item guile-ii --- 6 April 1995
+A low-level module system was added. Tcl/Tk support was added,
+allowing extension of Scheme by Tcl or vice versa. POSIX support was
+improved, and there was an experimental stab at Java integration.
+
+@item guile-iii --- 18 August 1995
+The C-like syntax, ctax, was improved, but mostly this release
+featured a start at the task of breaking Guile into pieces.
+
+@item 1.0 --- 5 January 1997
+@code{#f} was distinguished from @code{'()}. Green threads were added.
+Source-level debugging became more useful, and programmer's and user's
+manuals were begun. The module system gained a high-level interface,
+which is still used today in more or less the same form.
+
+@item 1.1 --- 16 May 1997
+@itemx 1.2 --- 24 June 1997
+Support for Tcl/Tk and ctax were split off as separate packages, and
+have remained there since. Guile became more compatible with SCSH, and
+more useful as a UNIX scripting language. Libguile can now be built as
+a shared library, and third-party extensions written in C became
+loadable via dynamic linking.
+
+@item 1.3.0 --- 19 October 1998
+Command-line editing became much more pleasant through the use of the
+readline library. The initial support for internationalization via
+multi-byte strings was removed, and has yet to be added back, though
+UTF-8 hacks are common. Modules gained the ability to have custom
+expanders, which is still used for syntax-case macros. Ports have
+better support for file descriptors, and fluids were added.
+
+@item 1.3.2 --- 20 August 1999
+@itemx 1.3.4 --- 25 September 1999
+@itemx 1.4 --- 21 June 2000
+A long list of lispy features were added: hooks, Common Lisp's
+@code{format}, optional and keyword procedure arguments,
+@code{getopt-long}, sorting, random numbers, and many other fixes and
+enhancements. Guile now has an interactive debugger, interactive help,
+and gives better backtraces.
+
+@item 1.6 --- 6 September 2002
+Guile gained support for the R5RS standard, and added a number of SRFI
+modules. The module system was expanded with programmatic support for
+identifier selection and renaming. The GOOPS object system was merged
+into Guile core.
+
+@item 1.8 --- 20 February 2006
+Guile's arbitrary-precision arithmetic switched to use the GMP
+library, and added support for exact rationals. Green threads were
+removed in favor of POSIX threads, providing true multiprocessing.
+Gettext support was added, and Guile's C API was cleaned up and
+orthogonalized in a massive way.
+
+@item 2.0 --- thus far, only unstable snapshots available
+A virtual machine was added to Guile, along with the associated
+compiler and toolchain. Support for locales was added. Running Guile
+instances became controllable and debuggable from within Emacs, via
+GDS. GDS was backported to 1.8.5. An SRFI-compatible interface to
+multithreading was added, including thread cancellation.
+@end table
+
+@node Status
+@subsection Status, or: Your Help Needed
+
+Guile has achieved much of what it set out to achieve, but there is
+much remaining to do.
+
+There is still the old problem of bringing existing applications into
+a more Emacs-like experience. Guile has had some successes in this
+respect, but still most applications in the GNU system are without
+Guile integration.
+
+Getting Guile to those applications takes an investment, the
+``hacktivation energy'' needed to wire Guile into a program that only
+pays off once it is good enough to enable new kinds of behavior. This
+would be a great way for new hackers to contribute: take an
+application that you use and that you know well, think of something
+that it can't yet do, and figure out a way to integrate Guile and
+implement that task in Guile.
+
+With time, perhaps this exposure can reverse itself, whereby programs
+can run under Guile instead of vice versa, eventually resulting in the
+Emacsification of the entire GNU system. Indeed, this is the reason
+for the naming of the many Guile modules that live in the @code{ice-9}
+namespace, a nod to the fictional substance in Kurt Vonnegut's
+novel, Cat's Cradle, capable of acting as a seed crystal to
+crystallize the mass of software.
+
+Implicit to this whole discussion is the idea that dynamic languages
+are somehow better than languages like C. While languages like C have
+their place, Guile's take on this question is that yes, Scheme is more
+expressive than C, and more fun to write. This realization carries an
+imperative with it to write as much code in Scheme as possible rather
+than in other languages.
+
+These days it is possible to write extensible applications almost
+entirely from high-level languages, through byte-code and native
+compilation, speed gains in the underlying hardware, and foreign call
+interfaces in the high-level language. Smalltalk systems are like
+this, as are Common Lisp-based systems. While there already are a
+number of pure-Guile applications out there, users still need to drop
+down to C for some tasks: interfacing to system libraries that don't
+have prebuilt Guile interfaces, and for some tasks requiring high
+performance.
+
+The addition of the virtual machine in Guile 2.0, together with the
+compiler infrastructure, should go a long way to addressing the speed
+issues. But there is much optimization to be done. Interested
+contributors will find lots of delightful low-hanging fruit, from
+simple profile-driven optimization to hacking a just-in-time compiler
+from VM bytecode to native code.
+
+Still, even with an all-Guile application, sometimes you want to
+provide an opportunity for users to extend your program from a
+language with a syntax that is closer to C, or to Python. Another
+interesting idea to consider is compiling e.g. Python to Guile. It's
+not that far-fetched of an idea: see for example IronPython or JRuby.
+
+And then there's Emacs itself. Though there is a somewhat-working
+Emacs Lisp translator for Guile, it cannot yet execute all of Emacs
+Lisp. A serious integration of Guile with Emacs would replace the
+Elisp virtual machine with Guile, and provide the necessary C shims so
+that Guile could emulate Emacs' C API. This would give lots of
+exciting things to Emacs: native threads, a real object system, more
+sophisticated types, cleaner syntax, and access to all of the Guile
+extensions.
+
+Finally, there is another axis of crystallization, the axis between
+different Scheme implementations. Guile does not yet support the
+latest Scheme standard, R6RS, and should do so. Like all standards,
+R6RS is imperfect, but supporting it will allow more code to run on
+Guile without modification, and will allow Guile hackers to produce
+code compatible with other schemes. Help in this regard would be much
+appreciated.

doc/ref/vm.texi

@@ -7,6 +7,19 @@
 @node A Virtual Machine for Guile
 @section A Virtual Machine for Guile
 
+Guile has both an interpreter and a compiler. To a user, the
+difference is largely transparent -- interpreted and compiled
+procedures can call each other as they please.
+
+The difference is that the compiler creates and interprets bytecode
+for a custom virtual machine, instead of interpreting the
+S-expressions directly. Running compiled code is faster than running
+interpreted code.
+
+The virtual machine that does the bytecode interpretation is a part of
+Guile itself. This section describes the nature of Guile's virtual
+machine.
+
 @menu
 * Why a VM?::                   
 * VM Concepts::                 
@@ -19,7 +32,47 @@
 @node Why a VM?
 @subsection Why a VM?
 
-asdfa
+For a long time, Guile only had an interpreter, called the evaluator.
+Guile's evaluator operates directly on the S-expression representation
+of Scheme source code.
+
+But while the evaluator is highly optimized and hand-tuned, and
+contains some extensive speed trickery (REFFIXME memoization), it
+still performs many needless computations during the course of
+evaluating an expression. For example, application of a function to
+arguments needlessly conses up the arguments in a list. Evaluation of
+an expression always has to figure out what the car of the expression
+is -- a procedure, a memoized form, or something else. All values have
+to be allocated on the heap. Et cetera.
+
+The solution to this problem is to compile the higher-level language,
+Scheme, into a lower-level language for which all of the checks and
+dispatching have already been done -- the code is instead stripped to
+the bare minimum needed to ``do the job''.
+
+The question becomes then, what low-level language to choose? There
+are many options. We could compile to native code directly, but that
+poses portability problems for Guile, as it is a highly cross-platform
+project.
+
+So we want the performance gains that compilation provides, but we
+also want to maintain the portability benefits of a single code path.
+The obvious solution is to compile to a virtual machine that is
+present on all Guile installations.
+
+The easiest (and most fun) way to depend on a virtual machine is to
+implement the virtual machine within Guile itself. This way the
+virtual machine provides what Scheme needs (tail calls, multiple
+values, call/cc) and can provide optimized inline instructions for
+Guile (cons, struct-ref, etc.).
+
+So this is what Guile does. The rest of this section describes that VM
+that Guile implements, and the compiled procedures that run on it.
+
+Note that this decision to implement a bytecode compiler does not
+preclude native compilation. We can compile from bytecode to native
+code at runtime, or even do ahead of time compilation. More
+possibilities are discussed in REFFIXME.
 
 @node VM Concepts
 @subsection VM Concepts
@@ -184,18 +237,17 @@ Within the lambda expression, "foo" is a top-level variable, "a" is a
 lexically captured variable, and "b" is a local variable.
 
 That is to say: @code{b} may safely be allocated on the stack, as
-there is no enclosing lexical environment that references it, nor is
+there is no enclosed procedure that references it, nor is it ever
-it ever mutated.
+mutated.
 
-@code{a}, on the other hand, is referenced by an enclosed lexical
+@code{a}, on the other hand, is referenced by an enclosed procedure,
-context, that of the lambda. Thus it must be allocated on the heap, as
+that of the lambda. Thus it must be allocated on the heap, as it may
-it may (and will) outlive the dynamic extent of the invocation of
+(and will) outlive the dynamic extent of the invocation of @code{foo}.
-@code{foo}.
 
 @code{foo} is a toplevel variable, as mandated by Scheme's semantics:
 
 @example
-  (define proc (foo 'bar))
+  (define proc (foo 'bar)) ; assuming prev. definition of @code{foo}
   (define foo 42)          ; redefinition
   (proc 'baz)
   @result{} (42 bar baz)
@@ -642,7 +694,7 @@ Create and fill a vector with the top @var{n} values from the stack,
 popping off those values and pushing on the resulting vector.
 @end deffn
 
-@deffn mark
+@deffn Instruction mark
 Pushes a special value onto the stack that other stack instructions
 like @code{list-mark} can use.
 @end deffn