Update documentation to incorporate read-syntax

* NEWS: Update a bit.
* doc/ref/api-debug.texi (Source Properties): Mention read-syntax.
* doc/ref/api-evaluation.texi (Annotated Scheme Read): New section.
* doc/ref/api-macros.texi (Syntax Case): Update for source vectors.

NEWS

@@ -43,16 +43,19 @@ documentation for a full discussion.
 
 ** New `read' implementation in Scheme
 
-The new `(ice-9 read)' module implements a reader, in Scheme.  Compared
+Guile's `read' procedure has been rewritten in Scheme.  Compared to the
-to the C reader (which still exists for bootstrapping reasons), the new
+C reader (which still exists for bootstrapping reasons), the new
 implementation is more maintainable, more secure, more debuggable, all
-while faithfully reproducing all quirks from Guile's legacy reader
+while faithfully reproducing all quirks from Guile's previous reader
 implemented in C.  Also, the Scheme reader is finally compatible with
 suspendable ports, allowing REPL implementations to be built with
 lightweight concurrency packages such as the third-party "Fibers"
-library.  The Scheme implementation is currently about 60% as fast as
+library.  Calls to `read' from Scheme as well as calls to `scm_read'
-the C implementation, and we hope to close the gap over time.  Bug
+from C use the new reader.
-reports very welcome.
+
+The Scheme implementation is currently about 60% as fast as the
+now-inaccessible C implementation, and we hope to close the gap over
+time.  Bug reports very welcome.
 
 ** Scheme compiler uses `read-syntax' for better debugging
 
@@ -110,10 +113,12 @@ These new interfaces replace `dynamic-link', `dynamic-pointer' and
 similar, which will eventually be deprecated.
 
 ** `read-syntax'
+
+See "Annotated Scheme Read" in the manual.
+
 ** `syntax-sourcev'
 ** `quote-syntax'
 
-
 ** Optimized "eof-object?"
 
 * Bug fixes

doc/ref/api-debug.texi

@@ -1,6 +1,6 @@
 @c -*-texinfo-*-
 @c This is part of the GNU Guile Reference Manual.
-@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2007, 2010, 2011, 2012, 2013, 2014, 2018
+@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2007, 2010, 2011, 2012, 2013, 2014, 2018, 2021
 @c   Free Software Foundation, Inc.
 @c See the file guile.texi for copying conditions.
 
@@ -245,35 +245,24 @@ frame. See its source code for more details.
 @node Source Properties
 @subsection Source Properties
 
+How best to associate source locations with datums parsed from a port?
+The right way to do this is to annotate all components of each parsed
+datum.  @xref{Annotated Scheme Read}, for more on @code{read-syntax}.
+
 @cindex source properties
-As Guile reads in Scheme code from file or from standard input, it
+Guile only switched to use @code{read-syntax} in 2021, however.  For the
-remembers the file name, line number and column number where each
+previous thirty years, it used a mechanism known as @dfn{source
-expression begins. These pieces of information are known as the
+properties}.
-@dfn{source properties} of the expression. Syntax expanders and the
-compiler propagate these source properties to compiled procedures, so
-that, if an error occurs when evaluating the transformed expression,
-Guile's debugger can point back to the file and location where the
-expression originated.
 
-The way that source properties are stored means that Guile cannot
+As Guile reads in Scheme code from file or from standard input, it can
-associate source properties with individual symbols, keywords,
+record remembers the file name, line number and column number where each
-characters, booleans, or small integers.  This can be seen by typing
+expression begins in a side table.
-@code{(xxx)} and @code{xxx} at the Guile prompt (where the variable
-@code{xxx} has not been defined):
 
-@example
+The way that source properties are stored means that Guile can only
-scheme@@(guile-user)> (xxx)
+associate source properties with freshly allocated objects.  This
-<unnamed port>:4:1: In procedure module-lookup:
+notably excludes individual symbols, keywords, characters, booleans, or
-<unnamed port>:4:1: Unbound variable: xxx
+small integers.  This limitation finally motivated the switch to
-
+@code{read-syntax}.
-scheme@@(guile-user)> xxx
-ERROR: In procedure module-lookup:
-ERROR: Unbound variable: xxx
-@end example
-
-@noindent
-In the latter case, no source properties were stored, so the error
-doesn't have any source information.
 
 @deffn {Scheme Procedure} supports-source-properties? obj
 @deffnx {C Function} scm_supports_source_properties_p (obj)
@@ -283,7 +272,9 @@ otherwise return #f.
 
 The recording of source properties is controlled by the read option
 named ``positions'' (@pxref{Scheme Read}).  This option is switched
-@emph{on} by default.
+@emph{on} by default.  Now that @code{read-syntax} is available,
+however, Guile may change the default for this flag to off in the
+future.
 
 The following procedures can be used to access and set the source
 properties of read expressions.

doc/ref/api-evaluation.texi

@@ -13,6 +13,7 @@ loading, evaluating, and compiling Scheme code at run time.
 @menu
 * Scheme Syntax::               Standard and extended Scheme syntax.
 * Scheme Read::                 Reading Scheme code.
+* Annotated Scheme Read::       Reading Scheme code, for the compiler.
 * Scheme Write::                Writing Scheme values to a port.
 * Fly Evaluation::              Procedures for on the fly evaluation.
 * Compilation::                 How to compile Scheme files and procedures.
@@ -386,6 +387,57 @@ For more information on the @code{r7rs-symbols} option, see
 (@pxref{Symbol Read Syntax}).
 
 
+@node Annotated Scheme Read
+@subsection Reading Scheme Code, For the Compiler
+
+When something goes wrong with a Scheme program, the user will want to
+know how to fix it.  This starts with identifying where the error
+occured: we want to associate a source location with each component part
+of source code, and propagate that source location information through
+to the compiler or interpreter.
+
+For that, Guile provides @code{read-syntax}.
+
+@deffn {Scheme Procedure} read-syntax [port]
+Read an s-expression from the input port @var{port}, or from the current
+input port if @var{port} is not specified.
+
+If, after skipping white space and comments, no more bytes are available
+from @var{port}, return the end-of-file object.  @xref{Binary I/O}.
+Otherwise, return an annotated datum.  An annotated datum is a syntax
+object which associates a source location with a datum.  For example:
+
+@example
+(call-with-input-string "  foo" read-syntax)
+; @result{} #<syntax:unknown file:1:2 foo>
+(call-with-input-string "(foo)" read-syntax)
+; @result{}
+; #<syntax:unknown file:1:0
+;   (#<syntax unknown file:1:1 foo>)>
+@end example
+
+As the second example shows, all fields of pairs and vectors are also
+annotated, recursively.
+@end deffn
+
+Most users are familiar with syntax objects in the context of macros,
+which use syntax objects to associate scope information with
+identifiers.  @xref{Macros}.  Here we use syntax objects to associate
+source location information with any datum, but without attaching scope
+information.  The Scheme compiler (@code{compile}) and the interpreter
+(@code{eval}) can accept syntax objects directly as input, allowing them
+to associate source information with resulting code.
+@xref{Compilation}, and @xref{Fly Evaluation}.
+
+Note that there is a legacy interface for getting source locations into
+the Scheme compiler or interpreter, which is to use a side table that
+associates ``source properties'' with each subdatum returned by
+@code{read}, instead of wrapping the datums directly as in
+@code{read-syntax}.  This has the disadvantage of not being able to
+annotate all kinds of datums.  @xref{Source Properties}, for more
+information.
+
+
 @node Scheme Write
 @subsection Writing Scheme Values
 

doc/ref/api-macros.texi

@@ -644,13 +644,19 @@ context corresponding to the identifier @var{template-id}.  If
 @var{template-id} is false, the datum will have no lexical context
 information.
 
-Syntax objects have an associated source location.  @xref{Source
+Syntax objects have an associated source location.  Internally this is
-Properties}.  If a syntax object is passed as @var{source}, the
+represented as a 3-element vector of filename, line, and column.
-resulting syntax object will have the source properties of @var{source}.
+Usually this location ultimately is provided by @code{read-syntax};
-Otherwise if @var{source} is a source properties alist, those will be
+@xref{Annotated Scheme Read}.
-the source properties of the resulting syntax object.  Otherwise if
+
-@var{source} is false, the source properties are computed as
+If a syntax object is passed as @var{source}, the resulting syntax
-@code{(source-properties @var{datum})}.
+object will have the source location of @var{source}.  Otherwise if
+@var{source} is a 3-element source location vector, that vector will be
+the source location of the resulting syntax object.  If @var{source} is
+a source properties alist, those will be parsed and set as the source
+location of the resulting syntax object.  Otherwise if @var{source} is
+false, the source properties are looked up from @code{(source-properties
+@var{datum})}.  @xref{Source Properties}.
 @end deffn
 
 For completeness, we should mention that it is possible to strip the metadata