Update documentation to incorporate read-syntax

* NEWS: Update a bit.
* doc/ref/api-debug.texi (Source Properties): Mention read-syntax.
* doc/ref/api-evaluation.texi (Annotated Scheme Read): New section.
* doc/ref/api-macros.texi (Syntax Case): Update for source vectors.

NEWS

@@ -43,16 +43,19 @@ documentation for a full discussion.
 
 ** New `read' implementation in Scheme
 
-The new `(ice-9 read)' module implements a reader, in Scheme.  Compared
-to the C reader (which still exists for bootstrapping reasons), the new
+Guile's `read' procedure has been rewritten in Scheme.  Compared to the
+C reader (which still exists for bootstrapping reasons), the new
 implementation is more maintainable, more secure, more debuggable, all
-while faithfully reproducing all quirks from Guile's legacy reader
+while faithfully reproducing all quirks from Guile's previous reader
 implemented in C.  Also, the Scheme reader is finally compatible with
 suspendable ports, allowing REPL implementations to be built with
 lightweight concurrency packages such as the third-party "Fibers"
-library.  The Scheme implementation is currently about 60% as fast as
-the C implementation, and we hope to close the gap over time.  Bug
-reports very welcome.
+library.  Calls to `read' from Scheme as well as calls to `scm_read'
+from C use the new reader.
+
+The Scheme implementation is currently about 60% as fast as the
+now-inaccessible C implementation, and we hope to close the gap over
+time.  Bug reports very welcome.
 
 ** Scheme compiler uses `read-syntax' for better debugging
 
@@ -110,10 +113,12 @@ These new interfaces replace `dynamic-link', `dynamic-pointer' and
 similar, which will eventually be deprecated.
 
 ** `read-syntax'
+
+See "Annotated Scheme Read" in the manual.
+
 ** `syntax-sourcev'
 ** `quote-syntax'
 
-
 ** Optimized "eof-object?"
 
 * Bug fixes

doc/ref/api-debug.texi

@@ -1,6 +1,6 @@
 @c -*-texinfo-*-
 @c This is part of the GNU Guile Reference Manual.
-@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2007, 2010, 2011, 2012, 2013, 2014, 2018
+@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2007, 2010, 2011, 2012, 2013, 2014, 2018, 2021
 @c   Free Software Foundation, Inc.
 @c See the file guile.texi for copying conditions.
 
@@ -245,35 +245,24 @@ frame. See its source code for more details.
 @node Source Properties
 @subsection Source Properties
 
+How best to associate source locations with datums parsed from a port?
+The right way to do this is to annotate all components of each parsed
+datum.  @xref{Annotated Scheme Read}, for more on @code{read-syntax}.
+
 @cindex source properties
-As Guile reads in Scheme code from file or from standard input, it
-remembers the file name, line number and column number where each
-expression begins. These pieces of information are known as the
-@dfn{source properties} of the expression. Syntax expanders and the
-compiler propagate these source properties to compiled procedures, so
-that, if an error occurs when evaluating the transformed expression,
-Guile's debugger can point back to the file and location where the
-expression originated.
+Guile only switched to use @code{read-syntax} in 2021, however.  For the
+previous thirty years, it used a mechanism known as @dfn{source
+properties}.
 
-The way that source properties are stored means that Guile cannot
-associate source properties with individual symbols, keywords,
-characters, booleans, or small integers.  This can be seen by typing
-@code{(xxx)} and @code{xxx} at the Guile prompt (where the variable
-@code{xxx} has not been defined):
+As Guile reads in Scheme code from file or from standard input, it can
+record remembers the file name, line number and column number where each
+expression begins in a side table.
 
-@example
-scheme@@(guile-user)> (xxx)
-<unnamed port>:4:1: In procedure module-lookup:
-<unnamed port>:4:1: Unbound variable: xxx
-
-scheme@@(guile-user)> xxx
-ERROR: In procedure module-lookup:
-ERROR: Unbound variable: xxx
-@end example
-
-@noindent
-In the latter case, no source properties were stored, so the error
-doesn't have any source information.
+The way that source properties are stored means that Guile can only
+associate source properties with freshly allocated objects.  This
+notably excludes individual symbols, keywords, characters, booleans, or
+small integers.  This limitation finally motivated the switch to
+@code{read-syntax}.
 
 @deffn {Scheme Procedure} supports-source-properties? obj
 @deffnx {C Function} scm_supports_source_properties_p (obj)
@@ -283,7 +272,9 @@ otherwise return #f.
 
 The recording of source properties is controlled by the read option
 named ``positions'' (@pxref{Scheme Read}).  This option is switched
-@emph{on} by default.
+@emph{on} by default.  Now that @code{read-syntax} is available,
+however, Guile may change the default for this flag to off in the
+future.
 
 The following procedures can be used to access and set the source
 properties of read expressions.

doc/ref/api-evaluation.texi

@@ -13,6 +13,7 @@ loading, evaluating, and compiling Scheme code at run time.
 @menu
 * Scheme Syntax::               Standard and extended Scheme syntax.
 * Scheme Read::                 Reading Scheme code.
+* Annotated Scheme Read::       Reading Scheme code, for the compiler.
 * Scheme Write::                Writing Scheme values to a port.
 * Fly Evaluation::              Procedures for on the fly evaluation.
 * Compilation::                 How to compile Scheme files and procedures.
@@ -386,6 +387,57 @@ For more information on the @code{r7rs-symbols} option, see
 (@pxref{Symbol Read Syntax}).
 
 
+@node Annotated Scheme Read
+@subsection Reading Scheme Code, For the Compiler
+
+When something goes wrong with a Scheme program, the user will want to
+know how to fix it.  This starts with identifying where the error
+occured: we want to associate a source location with each component part
+of source code, and propagate that source location information through
+to the compiler or interpreter.
+
+For that, Guile provides @code{read-syntax}.
+
+@deffn {Scheme Procedure} read-syntax [port]
+Read an s-expression from the input port @var{port}, or from the current
+input port if @var{port} is not specified.
+
+If, after skipping white space and comments, no more bytes are available
+from @var{port}, return the end-of-file object.  @xref{Binary I/O}.
+Otherwise, return an annotated datum.  An annotated datum is a syntax
+object which associates a source location with a datum.  For example:
+
+@example
+(call-with-input-string "  foo" read-syntax)
+; @result{} #<syntax:unknown file:1:2 foo>
+(call-with-input-string "(foo)" read-syntax)
+; @result{}
+; #<syntax:unknown file:1:0
+;   (#<syntax unknown file:1:1 foo>)>
+@end example
+
+As the second example shows, all fields of pairs and vectors are also
+annotated, recursively.
+@end deffn
+
+Most users are familiar with syntax objects in the context of macros,
+which use syntax objects to associate scope information with
+identifiers.  @xref{Macros}.  Here we use syntax objects to associate
+source location information with any datum, but without attaching scope
+information.  The Scheme compiler (@code{compile}) and the interpreter
+(@code{eval}) can accept syntax objects directly as input, allowing them
+to associate source information with resulting code.
+@xref{Compilation}, and @xref{Fly Evaluation}.
+
+Note that there is a legacy interface for getting source locations into
+the Scheme compiler or interpreter, which is to use a side table that
+associates ``source properties'' with each subdatum returned by
+@code{read}, instead of wrapping the datums directly as in
+@code{read-syntax}.  This has the disadvantage of not being able to
+annotate all kinds of datums.  @xref{Source Properties}, for more
+information.
+
+
 @node Scheme Write
 @subsection Writing Scheme Values
 

doc/ref/api-macros.texi

@@ -644,13 +644,19 @@ context corresponding to the identifier @var{template-id}.  If
 @var{template-id} is false, the datum will have no lexical context
 information.
 
-Syntax objects have an associated source location.  @xref{Source
-Properties}.  If a syntax object is passed as @var{source}, the
-resulting syntax object will have the source properties of @var{source}.
-Otherwise if @var{source} is a source properties alist, those will be
-the source properties of the resulting syntax object.  Otherwise if
-@var{source} is false, the source properties are computed as
-@code{(source-properties @var{datum})}.
+Syntax objects have an associated source location.  Internally this is
+represented as a 3-element vector of filename, line, and column.
+Usually this location ultimately is provided by @code{read-syntax};
+@xref{Annotated Scheme Read}.
+
+If a syntax object is passed as @var{source}, the resulting syntax
+object will have the source location of @var{source}.  Otherwise if
+@var{source} is a 3-element source location vector, that vector will be
+the source location of the resulting syntax object.  If @var{source} is
+a source properties alist, those will be parsed and set as the source
+location of the resulting syntax object.  Otherwise if @var{source} is
+false, the source properties are looked up from @code{(source-properties
+@var{datum})}.  @xref{Source Properties}.
 @end deffn
 
 For completeness, we should mention that it is possible to strip the metadata