HTTP, the Web, and All That
===========================

It has always been possible to connect computers together and share information between them, but the rise of the World-Wide Web over the last couple of decades has made it much easier to do so.  The result is a richly connected network of computation, in which Guile forms a part.

By ``the web'', we mean the HTTP protocol (Yes, the P is for protocol, but this phrase appears repeatedly in RFC 2616.) as handled by servers, clients, proxies, caches, and the various kinds of messages and message components that can be sent and received by that protocol, notably HTML.

On one level, the web is text in motion: the protocols themselves are textual (though the payload may be binary), and it's possible to create a socket and speak text to the web.  But such an approach is obviously primitive.  This section details the higher-level data types and operations provided by Guile: URIs, HTTP request and response records, and a conventional web server implementation.

The material in this section is arranged in ascending order, in which later concepts build on previous ones.  If you prefer to start with the highest-level perspective, see Web Examples, and work your way back.

Types and the Web
-----------------

It is a truth universally acknowledged, that a program with good use of data types, will be free from many common bugs.  Unfortunately, the common practice in web programming seems to ignore this maxim.  This subsection makes the case for expressive data types in web programming.

By ``expressive data types'', we mean that the data types say something about how a program solves a problem.  For example, if we choose to represent dates using SRFI 19 date records (see SRFI-19), this indicates that there is a part of the program that will always have valid dates.  Error handling for a number of basic cases, like invalid dates, occurs on the boundary in which we produce a SRFI 19 date record from other types, like strings.

With regards to the web, data types are help in the two broad phases of HTTP messages: parsing and generation.

Consider a server, which has to parse a request, and produce a response. Guile will parse the request into an HTTP request object (see Requests), with each header parsed into an appropriate Scheme data type.  This transition from an incoming stream of characters to typed data is a state change in a program---the strings might parse, or they might not, and something has to happen if they do not.  (Guile throws an error in this case.)  But after you have the parsed request, ``client'' code (code built on top of the Guile web framework) will not have to check for syntactic validity.  The types already make this information manifest.

This state change on the parsing boundary makes programs more robust, as they themselves are freed from the need to do a number of common error checks, and they can use normal Scheme procedures to handle a request instead of ad-hoc string parsers.

The need for types on the response generation side (in a server) is more subtle, though not less important.  Consider the example of a POST handler, which prints out the text that a user submits from a form. Such a handler might include a procedure like this:

     ;; First, a helper procedure
     (define (para . contents)
       (string-append "<p>" (string-concatenate contents) "</p>"))

     ;; Now the meat of our simple web application
     (define (you-said text)
       (para "You said: " text))

     (display (you-said "Hi!"))
     -| <p>You said: Hi!</p>

This is a perfectly valid implementation, provided that the incoming text does not contain the special HTML characters <, >, or &.  But this provision of a restricted character set is not reflected anywhere in the program itself: we must assume that the programmer understands this, and performs the check elsewhere.

Unfortunately, the short history of the practice of programming does not bear out this assumption.  A cross-site scripting (XSS) vulnerability is just such a common error in which unfiltered user input is allowed into the output.  A user could submit a crafted comment to your web site which results in visitors running malicious Javascript, within the security context of your domain:

     (display (you-said "<script src=\"http://bad.com/nasty.js\" />"))
     -| <p>You said: <script src="http://bad.com/nasty.js" /></p>

The fundamental problem here is that both user data and the program template are represented using strings.  This identity means that types can't help the programmer to make a distinction between these two, so they get confused.

There are a number of possible solutions, but perhaps the best is to treat HTML not as strings, but as native s-expressions: as SXML.  The basic idea is that HTML is either text, represented by a string, or an element, represented as a tagged list.  So foo becomes "foo", and <b>foo</b> becomes (b "foo"). Attributes, if present, go in a tagged list headed by @, like (img (@ (src "http://example.com/foo.png"))).  See sxml simple, for more information.

The good thing about SXML is that HTML elements cannot be confused with text.  Let's make a new definition of para:

     (define (para . contents)
       `(p ,@contents))

     (use-modules (sxml simple))
     (sxml->xml (you-said "Hi!"))
     -| <p>You said: Hi!</p>

     (sxml->xml (you-said "<i>Rats, foiled again!</i>"))
     -| <p>You said: &lt;i&gt;Rats, foiled again!&lt;/i&gt;</p>

So we see in the second example that HTML elements cannot be unwittingly introduced into the output.  However it is now perfectly acceptable to pass SXML to you-said; in fact, that is the big advantage of SXML over everything-as-a-string.

     (sxml->xml (you-said (you-said "<Hi!>")))
     -| <p>You said: <p>You said: &lt;Hi!&gt;</p></p>

The SXML types allow procedures to compose.  The types make manifest which parts are HTML elements, and which are text.  So you needn't worry about escaping user input; the type transition back to a string handles that for you.  XSS vulnerabilities are a thing of the past.

Well.  That's all very nice and opinionated and such, but how do I use the thing?  Read on!

Universal Resource Identifiers
------------------------------

Guile provides a standard data type for Universal Resource Identifiers (URIs), as defined in RFC 3986.

The generic URI syntax is as follows:

     URI := scheme ":" ["//" [userinfo "@"] host [":" port]] path \
            [ "?" query ] [ "#" fragment ]

For example, in the URI, http://www.gnu.org/help/, the scheme is http, the host is www.gnu.org, the path is /help/, and there is no userinfo, port, query, or path.  All URIs have a scheme and a path (though the path might be empty).  Some URIs have a host, and some of those have ports and userinfo.  Any URI might have a query part or a fragment.

Userinfo is something of an abstraction, as some legacy URI schemes allowed userinfo of the form USERNAME:PASSWD.  But since passwords do not belong in URIs, the RFC does not want to condone this practice, so it calls anything before the @ sign userinfo.

Properly speaking, a fragment is not part of a URI.  For example, when a web browser follows a link to http://example.com/#foo, it sends a request for http://example.com/, then looks in the resulting page for the fragment identified foo reference.  A fragment identifies a part of a resource, not the resource itself.  But it is useful to have a fragment field in the URI record itself, so we hope you will forgive the inconsistency.

     (use-modules (web uri))

The following procedures can be found in the (web uri) module. Load it into your Guile, using a form like the above, to have access to them.

 -- Function: build-uri scheme [#:userinfo=#f] [#:host=#f]
     [#:port=#f] [#:path=""] [#:query=#f]
     [#:fragment=#f] [#:validate?=#t]
     Construct a URI object.  SCHEME should be a symbol, and the rest of the fields are either strings or #f.  If VALIDATE? is true, also run some consistency checks to make sure that the constructed URI is valid.

 -- Function: uri? x
 -- Function: uri-scheme uri
 -- Function: uri-userinfo uri
 -- Function: uri-host uri
 -- Function: uri-port uri
 -- Function: uri-path uri
 -- Function: uri-query uri
 -- Function: uri-fragment uri
     A predicate and field accessors for the URI record type.  The URI scheme will be a symbol, and the rest either strings or #f if not present.

 -- Function: string->uri string
     Parse STRING into a URI object.  Return #f if the string could not be parsed.

 -- Function: uri->string uri
     Serialize URI to a string.  If the URI has a port that is the default port for its scheme, the port is not included in the serialization.

 -- Function: declare-default-port! scheme port
     Declare a default port for the given URI scheme.

 -- Function: uri-decode str [#:encoding="utf-8"]
     Percent-decode the given STR, according to ENCODING, which should be the name of a character encoding.

     Note that this function should not generally be applied to a full URI string. For paths, use split-and-decode-uri-path instead. For query strings, split the query on & and = boundaries, and decode the components separately.

     Note also that percent-encoded strings encode bytes, not characters.  There is no guarantee that a given byte sequence is a valid string encoding. Therefore this routine may signal an error if the decoded bytes are not valid for the given encoding. Pass #f for ENCODING if you want decoded bytes as a bytevector directly. See set-port-encoding!, for more information on character encodings.

     Returns a string of the decoded characters, or a bytevector if ENCODING was #f.

Fixme: clarify return type. indicate default values. type of unescaped-chars.

 -- Function: uri-encode str [#:encoding="utf-8"] [#:unescaped-chars]
     Percent-encode any character not in the character set, UNESCAPED-CHARS.

     The default character set includes alphanumerics from ASCII, as well as the special characters -, ., _, and ~.  Any other character will be percent-encoded, by writing out the character to a bytevector within the given ENCODING, then encoding each byte as %HH, where HH is the hexadecimal representation of the byte.

 -- Function: split-and-decode-uri-path path
     Split PATH into its components, and decode each component, removing empty components.

     For example, "/foo/bar%20baz/" decodes to the two-element list, ("foo" "bar baz").

 -- Function: encode-and-join-uri-path parts
     URI-encode each element of PARTS, which should be a list of strings, and join the parts together with / as a delimiter.

     For example, the list ("scrambled eggs" "biscuits&gravy") encodes as "scrambled%20eggs/biscuits%26gravy".

The Hyper-Text Transfer Protocol
--------------------------------

The initial motivation for including web functionality in Guile, rather than rely on an external package, was to establish a standard base on which people can share code.  To that end, we continue the focus on data types by providing a number of low-level parsers and unparsers for elements of the HTTP protocol.

If you are want to skip the low-level details for now and move on to web pages, see Web Server.  Otherwise, load the HTTP module, and read on.

     (use-modules (web http))

The focus of the (web http) module is to parse and unparse standard HTTP headers, representing them to Guile as native data structures.  For example, a Date: header will be represented as a SRFI-19 date record (see SRFI-19), rather than as a string.

Guile tries to follow RFCs fairly strictly---the road to perdition being paved with compatibility hacks---though some allowances are made for not-too-divergent texts.

The first bit is to define a registry of parsers, validators, and unparsers, keyed by header name.  That is the function of the <header-decl> object.

 -- Function: make-header-decl sym name multiple? parser validator writer
 -- Function: header-decl? x
 -- Function: header-decl-sym decl
 -- Function: header-decl-name decl
 -- Function: header-decl-multiple? decl
 -- Function: header-decl-parser decl
 -- Function: header-decl-validator decl
 -- Function: header-decl-writer decl.
     A constructor, predicate, and field accessors for the <header-decl> type. The fields are as follows:

     sym
          The symbol name for this header field, always in lower-case.  For example, "Content-Length" has a symbolic name of content-length.

     name
          The string name of the header, in its preferred capitalization.

     multiple?
          #t iff this header may appear multiple times in a message.

     parser
          A procedure which takes a string and returns a parsed value.

     validator
          A predicate, returning #t iff the value is valid for this header.

     writer
          A writer, which writes a value to the port given in the second argument.

 -- Function: declare-header! sym name [#:multiple?=#f] [#:parser] [#:validator] [#:writer]
     Make a header declaration, as above, and register it by symbol and by name. The PARSER, VALIDATOR, and WRITER arguments are all mandatory.

For example, let's say you are running a web server behind some sort of proxy, and your proxy adds an X-Client-Address header, indicating the IPv4 address of the original client.  You would like for the HTTP request record to parse out this header to a Scheme value, instead of leaving it as a string.  You could register this header with Guile's HTTP stack like this:

     (define (parse-ip str)
       (inet-aton str)
     (define (validate-ip ip)
     (define (write-ip ip port)
       (display (inet-ntoa ip) port))

     (declare-header! 'x-client-address
       "X-Client-Address"
       #:parser    (lambda (str)
                     (inet-aton str))
       #:validator (lambda (ip)
                     (and (integer? ip) (exact? ip) (<= 0 ip 4294967295)))
       #:writer    (lambda (ip port)
                     (display (inet-ntoa ip) port)))

 -- Function: lookup-header-decl name
     Return the HEADER-DECL object registered for the given NAME.

     NAME may be a symbol or a string. Strings are mapped to headers in a case-insensitive fashion.

 -- Function: valid-header? sym val
     Return a true value iff VAL is a valid Scheme value for the header with name SYM.

Now that we have a generic interface for reading and writing headers, we do just that.

 -- Function: read-header port
     Read one HTTP header from PORT. Return two values: the header name and the parsed Scheme value. May raise an exception if the header was known but the value was invalid.

     Returns the end-of-file object for both values if the end of the message body was reached (i.e., a blank line).

 -- Function: parse-header name val
     Parse VAL, a string, with the parser for the header named NAME.

     Return two values, the header name and parsed value. If a parser was found, the header name will be returned as a symbol. If a parser was not found, both the header name and the value are returned as strings.

 -- Function: write-header name val port
     Writes the given header name and value to PORT. If NAME is a symbol, looks up a declared header and uses that writer. Otherwise the value is written using DISPLAY.

 -- Function: read-headers port
     Read the headers of an HTTP message from PORT, returning the headers as an ordered alist.

 -- Function: write-headers headers port
     Write the given header alist to PORT. Doesn't write the final \r\n, as the user might want to add another header.

The (web http) module also has some utility procedures to read and write request and response lines.

 -- Function: parse-http-method str [start] [end]
     Parse an HTTP method from STR. The result is an upper-case symbol, like GET.

 -- Function: parse-http-version str [start] [end]
     Parse an HTTP version from STR, returning it as a major-minor pair. For example, HTTP/1.1 parses as the pair of integers, (1 . 1).

 -- Function: parse-request-uri str [start] [end]
     Parse a URI from an HTTP request line. Note that URIs in requests do not have to have a scheme or host name. The result is a URI object.

 -- Function: read-request-line port
     Read the first line of an HTTP request from PORT, returning three values: the method, the URI, and the version.

 -- Function: write-request-line method uri version port
     Write the first line of an HTTP request to PORT.

 -- Function: read-response-line port
     Read the first line of an HTTP response from PORT, returning three values: the HTTP version, the response code, and the "reason phrase".

 -- Function: write-response-line version code reason-phrase port
     Write the first line of an HTTP response to PORT.

HTTP Headers
------------

In addition to defining the infrastructure to parse headers, the (web http) module defines specific parsers and unparsers for all headers defined in the HTTP/1.1 standard.

For example, if you receive a header named Accept-Language with a value en, es;q=0.8, Guile parses it as follows:

     (parse-header "Accept-Language" "en, es;q=0.8")
     => accept-language
     => ((1000 . "en") (800 . "es"))

There are two results, because parse-header returns two values.  The first value is a symbol, because the accept-language header is known to Guile and has a parser registered.  The format of the value for accept-language headers is defined below, along with all other headers defined in the HTTP standard.  (If the header were not recognized, it and the value would be returned as strings.)

For brevity, the header definitions below are given in the form, TYPE NAME, indicating that values for the header NAME will be of the given TYPE.  A short description of the each header's purpose and an example follow.  For full details on the meanings of all of these headers, see the HTTP 1.1 standard, RFC 2616.

HTTP Header Types
.................

 -- HTTP Header Type: Date
     foo

So for example if you are implementing a

This section describes the parsed format of the various headers.

We cannot describe the function of all of these headers, however, in sufficient detail.  The interested reader would do well to download a copy of RFC 2616 and have it on hand.

example? and examples in each, and brief meaning description.

To begin with, we should make a few definitions:

key-value list
     A key-value list is a list of values.  Each value may be a string, a symbol, or a pair.  Known keys are parsed to symbols; otherwise keys are left as strings.  Keys with values are parsed to pairs, the car of which is the symbol or string key, and the cdr is the parsed value. Parsed values for known keys have key-dependent formats.  Parsed values for unknown keys are strings.

quality
     A number of headers have quality values in them, which are decimal fractions between zero and one indicating a preference for various kinds of responses, which the server may choose to heed.  Given that only three digits are allowed in the fractional part, Guile parses quality values to integers between 0 and 1000 instead of inexact numbers between 0.0 and 1.0.

quality list
     A list of pairs, the car of which is a quality value.

entity tag
     A pair, the car of which is an opaque string, and the cdr of which is true iff the entity tag is a ``strong'' entity tag.

General Headers
...............

 -- HTTP Header: KVList cache-control
     A key-value list of cache-control directives. Known keys are max-age, max-stale, min-fresh, must-revalidate, no-cache, no-store, no-transform, only-if-cached, private, proxy-revalidate, public, and s-maxage.

     If present, parameters to max-age, max-stale, min-fresh, and s-maxage are all parsed as non-negative integers.

     If present, parameters to private and no-cache are parsed as lists of header names, represented as symbols if they are known headers or strings otherwise.

 -- HTTP Header: List of Strings connection
     A list of connection tokens.  A connection token is a string.

 -- HTTP Header: date date
     A SRFI-19 date record.

 -- HTTP Header: Key-Value List pragma
     A key-value list of pragma directives.  no-cache is the only known key.

 -- HTTP Header: tp trailer
     A list of header names.  Known header names are parsed to symbols, otherwise they are left as strings.

 -- HTTP Header: tp transfer-encoding
     A param list of transfer codings.  chunked is the only known key.

 -- HTTP Header: tp upgrade
     A list of strings.

 -- HTTP Header: tp via
     A list of strings.  There may be multiple via headers in ne message.

 -- HTTP Header: tp warning
     A list of warnings.  Each warning is a itself a list of four elements: a code, as an exact integer between 0 and 1000, a host as a string, the warning text as a string, and either #f or a SRFI-19 date.

     There may be multiple warning headers in one message.

Entity Headers
..............

 -- HTTP Header: tp allow
     A list of methods, as strings.  Methods are parsed as strings instead of parse-http-method so as to allow for new methods.

 -- HTTP Header: tp content-encoding
     A list of content codings, as strings.

 -- HTTP Header: tp content-language
     A list of language tags, as strings.

 -- HTTP Header: tp content-length
     An exact, non-negative integer.

 -- HTTP Header: tp content-location
     A URI record.

 -- HTTP Header: tp content-md5
     A string.

 -- HTTP Header: tp content-range
     A list of three elements: the symbol bytes, either the symbol * or a pair of integers, indicating the byte rage, and either * or an integer, for the instance length.

 -- HTTP Header: tp content-type
     A pair, the car of which is the media type as a string, and the cdr is an alist of parameters, with strings as keys and values.

     For example, "text/plain" parses as ("text/plain"), and "text/plain;charset=utf-8" parses as ("text/plain" ("charset" . "utf-8")).

     note charset and encoding

 -- HTTP Header: tp expires
     A SRFI-19 date.

 -- HTTP Header: tp last-modified
     A SRFI-19 date.

Request Headers
...............

 -- HTTP Header: tp accept
     A param list.  Each element in the list indicates one media-range with accept-params.  They only known key is q, whose value is parsed as a quality value.

 -- HTTP Header: tp accept-charset
     A quality-list of charsets, as strings.

     charset and encoding

 -- HTTP Header: tp accept-encoding
     A quality-list of content codings, as strings.

 -- HTTP Header: tp accept-language
     A quality-list of languages, as strings.

 -- HTTP Header: tp authorization
     A string.

 -- HTTP Header: tp expect
     A param list of expectations.  The only known key is 100-continue.

 -- HTTP Header: tp from
     A string.

 -- HTTP Header: tp host
     A pair of the host, as a string, and the port, as an integer. If no port is given, port is #f.

 -- HTTP Header: tp if-match
     Either the symbol *, or a list of entity tags (see above).

 -- HTTP Header: tp if-modified-since
     A SRFI-19 date.

 -- HTTP Header: tp if-none-match
     Either the symbol *, or a list of entity tags (see above).

 -- HTTP Header: tp if-range
     Either an entity tag, or a SRFI-19 date.

 -- HTTP Header: tp if-unmodified-since
     A SRFI-19 date.

 -- HTTP Header: tp max-forwards
     An exact non-negative integer.

 -- HTTP Header: tp proxy-authorization
     A string.

 -- HTTP Header: tp range
     A pair whose car is the symbol bytes, and whose cdr is a list of pairs. Each element of the cdr indicates a range; the car is the first byte position and the cdr is the last byte position, as integers, or #f if not given.

 -- HTTP Header: tp referer
     A URI.

 -- HTTP Header: tp te
     A param list of transfer-codings.  The only known key is trailers.

 -- HTTP Header: tp user-agent
     A string.

Response Headers
................

 -- HTTP Header: tp accept-ranges
     A list of strings.

 -- HTTP Header: tp age
     An exact, non-negative integer.

 -- HTTP Header: tp etag
     An entity tag.

 -- HTTP Header: tp location
     A URI.

 -- HTTP Header: tp proxy-authenticate
     A string.

 -- HTTP Header: tp retry-after
     Either an exact, non-negative integer, or a SRFI-19 date.

 -- HTTP Header: tp server
     A string.

 -- HTTP Header: tp vary
     Either the symbol *, or a list of headers, with known headers parsed to symbols.

 -- HTTP Header: tp www-authenticate
     A string. (FIXME)

HTTP Requests
-------------

     (use-modules (web request))

The request module contains a data type for HTTP requests.  Note that the body is not part of the request, but the port is.  Once you have read a request, you may read the body separately, and likewise for writing requests.

discussion of charsets and bytes and stuff.

 -- Function: build-request [#:method] [#:uri] [#:version] [#:headers] [#:port] [#:meta] [#:validate-headers?]
     Construct an HTTP request object. If VALIDATE-HEADERS? is true, the headers are each run through their respective validators.

 -- Function: request?
 -- Function: request-method
 -- Function: request-uri
 -- Function: request-version
 -- Function: request-headers
 -- Function: request-meta
 -- Function: request-port
     A predicate and field accessors for the request type.  The fields are as follows:

     method
          The HTTP method, for example, GET.

     uri
          The URI as a URI record.

     version
          The HTTP version pair, like (1 . 1).

     headers
          The request headers, as an alist of parsed values.

     meta
          An arbitrary alist of other data, for example information returned in the sockaddr from accept (see Network Sockets and Communication).

     port
          The port on which to read or write a request body, if any.

 -- Function: read-request port [meta]
     Read an HTTP request from PORT, optionally attaching the given metadata, META.

     As a side effect, sets the encoding on PORT to ISO-8859-1 (latin-1), so that reading one character reads one byte. See the discussion of character sets in "HTTP Requests" in the manual, for more information.

Fixme^

 -- Function: write-request r port
     Write the given HTTP request to PORT.

     Return a new request, whose request-port will continue writing on PORT, perhaps using some transfer encoding.

 -- Function: read-request-body/latin-1 r
     Reads the request body from R, as a string.

     Assumes that the request port has ISO-8859-1 encoding, so that the number of characters to read is the same as the request-content-length.  Return #f if there was no request body.

 -- Function: write-request-body/latin-1 r body
     Write BODY, a string encodable in ISO-8859-1, to the port corresponding to the HTTP request R.

 -- Function: read-request-body/bytevector r
     Reads the request body from R, as a bytevector.  Return #f if there was no request body.

 -- Function: write-request-body/bytevector r bv
     Write BODY, a bytevector, to the port corresponding to the HTTP request R.

The various headers that are typically associated with HTTP requests may be accessed with these dedicated accessors.  See HTTP Headers, for more information on the format of parsed headers.

 -- Function: request-accept request [default='()]
 -- Function: request-accept-charset request [default='()]
 -- Function: request-accept-encoding request [default='()]
 -- Function: request-accept-language request [default='()]
 -- Function: request-allow request [default='()]
 -- Function: request-authorization request [default=#f]
 -- Function: request-cache-control request [default='()]
 -- Function: request-connection request [default='()]
 -- Function: request-content-encoding request [default='()]
 -- Function: request-content-language request [default='()]
 -- Function: request-content-length request [default=#f]
 -- Function: request-content-location request [default=#f]
 -- Function: request-content-md5 request [default=#f]
 -- Function: request-content-range request [default=#f]
 -- Function: request-content-type request [default=#f]
 -- Function: request-date request [default=#f]
 -- Function: request-expect request [default='()]
 -- Function: request-expires request [default=#f]
 -- Function: request-from request [default=#f]
 -- Function: request-host request [default=#f]
 -- Function: request-if-match request [default=#f]
 -- Function: request-if-modified-since request [default=#f]
 -- Function: request-if-none-match request [default=#f]
 -- Function: request-if-range request [default=#f]
 -- Function: request-if-unmodified-since request [default=#f]
 -- Function: request-last-modified request [default=#f]
 -- Function: request-max-forwards request [default=#f]
 -- Function: request-pragma request [default='()]
 -- Function: request-proxy-authorization request [default=#f]
 -- Function: request-range request [default=#f]
 -- Function: request-referer request [default=#f]
 -- Function: request-te request [default=#f]
 -- Function: request-trailer request [default='()]
 -- Function: request-transfer-encoding request [default='()]
 -- Function: request-upgrade request [default='()]
 -- Function: request-user-agent request [default=#f]
 -- Function: request-via request [default='()]
 -- Function: request-warning request [default='()]
     Return the given request header, or DEFAULT if none was present.

 -- Function: request-absolute-uri r [default-host] [default-port]
     A helper routine to determine the absolute URI of a request, using the host header and the default host and port.

HTTP Responses
--------------

     (use-modules (web response))

As with requests (see Requests), Guile offers a data type for HTTP responses.  Again, the body is represented separately from the request.

 -- Function: response?
 -- Function: response-version
 -- Function: response-code
 -- Function: response-reason-phrase response
 -- Function: response-headers
 -- Function: response-port
     A predicate and field accessors for the response type.  The fields are as follows:

     version
          The HTTP version pair, like (1 . 1).

     code
          The HTTP response code, like 200.

     reason-phrase
          The reason phrase, or the standard reason phrase for the response's code.

     headers
          The response headers, as an alist of parsed values.

     port
          The port on which to read or write a response body, if any.

 -- Function: read-response port
     Read an HTTP response from PORT, optionally attaching the given metadata, META.

     As a side effect, sets the encoding on PORT to ISO-8859-1 (latin-1), so that reading one character reads one byte. See the discussion of character sets in "HTTP Responses" in the manual, for more information.

 -- Function: build-response [#:version] [#:code] [#:reason-phrase] [#:headers] [#:port]
     Construct an HTTP response object. If VALIDATE-HEADERS? is true, the headers are each run through their respective validators.

FIXME
 -- Function: extend-response r k v . additional
     Extend an HTTP response by setting additional HTTP headers K, V.  Return a new HTTP response.

 -- Function: adapt-response-version response version
     Adapt the given response to a different HTTP version.  Return a new HTTP response.

     The idea is that many applications might just build a response for the default HTTP version, and this method could handle a number of programmatic transformations to respond to older HTTP versions (0.9 and 1.0). But currently this function is a bit heavy-handed, just updating the version field.

 -- Function: write-response r port
     Write the given HTTP response to PORT.

     Return a new response, whose response-port will continue writing on PORT, perhaps using some transfer encoding.

 -- Function: read-response-body/latin-1 r
     Reads the response body from R, as a string.

     Assumes that the response port has ISO-8859-1 encoding, so that the number of characters to read is the same as the response-content-length. Return #f if there was no response body.

 -- Function: write-response-body/latin-1 r body
     Write BODY, a string encodable in ISO-8859-1, to the port corresponding to the HTTP response R.

 -- Function: read-response-body/bytevector r
     Read the response body from R, as a bytevector.  Return #f if there was no response body.

 -- Function: write-response-body/bytevector r bv
     Write BODY, a bytevector, to the port corresponding to the HTTP response R.

As with requests, the various headers that are typically associated with HTTP responses may be accessed with these dedicated accessors. See HTTP Headers, for more information on the format of parsed headers.

 -- Function: response-accept-ranges response [default=#f]
 -- Function: response-age response [default='()]
 -- Function: response-allow response [default='()]
 -- Function: response-cache-control response [default='()]
 -- Function: response-connection response [default='()]
 -- Function: response-content-encoding response [default='()]
 -- Function: response-content-language response [default='()]
 -- Function: response-content-length response [default=#f]
 -- Function: response-content-location response [default=#f]
 -- Function: response-content-md5 response [default=#f]
 -- Function: response-content-range response [default=#f]
 -- Function: response-content-type response [default=#f]
 -- Function: response-date response [default=#f]
 -- Function: response-etag response [default=#f]
 -- Function: response-expires response [default=#f]
 -- Function: response-last-modified response [default=#f]
 -- Function: response-location response [default=#f]
 -- Function: response-pragma response [default='()]
 -- Function: response-proxy-authenticate response [default=#f]
 -- Function: response-retry-after response [default=#f]
 -- Function: response-server response [default=#f]
 -- Function: response-trailer response [default='()]
 -- Function: response-transfer-encoding response [default='()]
 -- Function: response-upgrade response [default='()]
 -- Function: response-vary response [default='()]
 -- Function: response-via response [default='()]
 -- Function: response-warning response [default='()]
 -- Function: response-www-authenticate response [default=#f]
     Return the given request header, or DEFAULT if none was present.

Web Server
----------

(web server) is a generic web server interface, along with a main loop implementation for web servers controlled by Guile.

     (use-modules (web server))

The lowest layer is the <server-impl> object, which defines a set of hooks to open a server, read a request from a client, write a response to a client, and close a server.  These hooks -- open, read, write, and close, respectively -- are bound together in a <server-impl> object.  Procedures in this module take a <server-impl> object, if needed.

A <server-impl> may also be looked up by name.  If you pass the http symbol to run-server, Guile looks for a variable named http in the (web server http) module, which should be bound to a <server-impl> object.  Such a binding is made by instantiation of the define-server-impl syntax.  In this way the run-server loop can automatically load other backends if available.

The life cycle of a server goes as follows:

 1. The open hook is called, to open the server. open takes 0 or more arguments, depending on the backend, and returns an opaque server socket object, or signals an error.

 2. The read hook is called, to read a request from a new client. The read hook takes one argument, the server socket.  It should return three values: an opaque client socket, the request, and the request body. The request should be a <request> object, from (web request).  The body should be a string or a bytevector, or #f if there is no body.

     If the read failed, the read hook may return #f for the client socket, request, and body.

 3. A user-provided handler procedure is called, with the request and body as its arguments.  The handler should return two values: the response, as a <response> record from (web response), and the response body as a string, bytevector, or #f if not present.  We also allow the response to be simply an alist of headers, in which case a default response object is constructed with those headers.

 4. The write hook is called with three arguments: the client socket, the response, and the body.  The write hook returns no values.

 5. At this point the request handling is complete. For a loop, we loop back and try to read a new request.

 6. If the user interrupts the loop, the close hook is called on the server socket.

A user may define a server implementation with the following form:

 -- Function: define-server-impl name open read write close
     Make a <server-impl> object with the hooks OPEN, READ, WRITE, and CLOSE, and bind it to the symbol NAME in the current module.

 -- Function: lookup-server-impl impl
     Look up a server implementation. If IMPL is a server implementation already, it is returned directly. If it is a symbol, the binding named IMPL in the (web server IMPL) module is looked up. Otherwise an error is signaled.

     Currently a server implementation is a somewhat opaque type, useful only for passing to other procedures in this module, like read-client.

The (web server) module defines a number of routines that use <server-impl> objects to implement parts of a web server.  Given that we don't expose the accessors for the various fields of a <server-impl>, indeed these routines are the only procedures with any access to the impl objects.

 -- Function: open-server impl open-params
     Open a server for the given implementation.  Return one value, the new server object. The implementation's open procedure is applied to OPEN-PARAMS, which should be a list.

 -- Function: read-client impl server
     Read a new client from SERVER, by applying the implementation's read procedure to the server.  If successful, return three values: an object corresponding to the client, a request object, and the request body. If any exception occurs, return #f for all three values.

 -- Function: handle-request handler request body state
     Handle a given request, returning the response and body.

     The response and response body are produced by calling the given HANDLER with REQUEST and BODY as arguments.

     The elements of STATE are also passed to HANDLER as arguments, and may be returned as additional values. The new STATE, collected from the HANDLER's return values, is then returned as a list. The idea is that a server loop receives a handler from the user, along with whatever state values the user is interested in, allowing the user's handler to explicitly manage its state.

 -- Function: sanitize-response request response body
     "Sanitize" the given response and body, making them appropriate for the given request.

     As a convenience to web handler authors, RESPONSE may be given as an alist of headers, in which case it is used to construct a default response. Ensures that the response version corresponds to the request version. If BODY is a string, encodes the string to a bytevector, in an encoding appropriate for RESPONSE. Adds a content-length and content-type header, as necessary.

     If BODY is a procedure, it is called with a port as an argument, and the output collected as a bytevector. In the future we might try to instead use a compressing, chunk-encoded port, and call this procedure later, in the write-client procedure. Authors are advised not to rely on the procedure being called at any particular time.

 -- Function: write-client impl server client response body
     Write an HTTP response and body to CLIENT. If the server and client support persistent connections, it is the implementation's responsibility to keep track of the client thereafter, presumably by attaching it to the SERVER argument somehow.

 -- Function: close-server impl server
     Release resources allocated by a previous invocation of open-server.

Given the procedures above, it is a small matter to make a web server:

 -- Function: serve-one-client handler impl server state
     Read one request from SERVER, call HANDLER on the request and body, and write the response to the client.  Return the new state produced by the handler procedure.

 -- Function: run-server handler [impl] [open-params] . state
     Run Guile's built-in web server.

     HANDLER should be a procedure that takes two or more arguments, the HTTP request and request body, and returns two or more values, the response and response body.

     For example, here is a simple "Hello, World!" server:

           (define (handler request body)
             (values '((content-type . ("text/plain")))
                     "Hello, World!"))
           (run-server handler)

     The response and body will be run through sanitize-response before sending back to the client.

     Additional arguments to HANDLER are taken from STATE. Additional return values are accumulated into a new STATE, which will be used for subsequent requests. In this way a handler can explicitly manage its state.

     FIXME: elide?

     The default server implementation is http, which accepts OPEN-PARAMS like (#:port 8081), among others. See "Web Server" in the manual, for more information.

Web Examples
------------

Well, enough about the tedious internals.  Let's make a web application!

Hello, World!
.............

The first program we have to write, of course, is ``Hello, World!''. This means that we have to implement a web handler that does what we want.

Now we define a handler, a function of two arguments and two return values:

     (define (handler request request-body)
       (values RESPONSE RESPONSE-BODY))

In this first example, we take advantage of a short-cut, returning an alist of headers instead of a proper response object. The response body is our payload:

     (define (hello-world-handler request request-body)
       (values '((content-type . ("text/plain")))
               "Hello World!"))

Now let's test it, by running a server with this handler. Load up the web server module if you haven't yet done so, and run a server with this handler:

     (use-modules (web server))
     (run-server hello-world-handler)

By default, the web server listens for requests on localhost:8080.  Visit that address in your web browser to test.  If you see the string, Hello World!, sweet!

Inspecting the Request
......................

The Hello World program above is a general greeter, responding to all URIs.  To make a more exclusive greeter, we need to inspect the request object, and conditionally produce different results.  So let's load up the request, response, and URI modules, and do just that.

     (use-modules (web server)) ; you probably did this already
     (use-modules (web request)
                  (web response)
                  (web uri))

     (define (request-path-components request)
       (split-and-decode-uri-path (uri-path (request-uri request))))

     (define (hello-hacker-handler request body)
       (if (equal? (request-path-components request)
                   '("hacker"))
           (values '((content-type . ("text/plain")))
                   "Hello hacker!")
           (not-found request)))

     (run-server hello-hacker-handler)

Here we see that we have defined a helper to return the components of the URI path as a list of strings, and used that to check for a request to /hacker/. Then the success case is just as before -- visit http://localhost:8080/hacker/ in your browser to check.

You should always match against URI path components as decoded by split-and-decode-uri-path. The above example will work for /hacker/, //hacker///, and /h%61ck%65r.

But we forgot to define not-found!  If you are pasting these examples into a REPL, accessing any other URI in your web browser will drop your Guile console into the debugger:

     <unnamed port>:38:7: In procedure module-lookup:
     <unnamed port>:38:7: Unbound variable: not-found

     Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
     scheme@(guile-user) [1]>

So let's define the function, right there in the debugger.  As you probably know, we'll want to return a 404 response.

     ;; Paste this in your REPL
     (define (not-found request)
       (values (build-response #:code 404)
               (string-append "Resource not found: "
                              (unparse-uri (request-uri request)))))

     ;; Now paste this to let the web server keep going:
     ,continue

Now if you access http://localhost/foo/, you get this error message.  (Note that some popular web browsers won't show server-generated 404 messages, showing their own instead, unless the 404 message body is long enough.)

Higher-Level Interfaces
.......................

The web handler interface is a common baseline that all kinds of Guile web applications can use.  You will usually want to build something on top of it, however, especially when producing HTML.  Here is a simple example that builds up HTML output using SXML (see sxml simple).

First, load up the modules:

     (use-modules (web server)
                  (web request)
                  (web response)
                  (sxml simple))

Now we define a simple templating function that takes a list of HTML body elements, as SXML, and puts them in our super template:

     (define (templatize title body)
       `(html (head (title ,title))
              (body ,@body)))

For example, the simplest Hello HTML can be produced like this:

     (sxml->xml (templatize "Hello!" '((b "Hi!"))))
     -|
     <html><head><title>Hello!</title></head><body><b>Hi!</b></body></html>

Much better to work with Scheme data types than to work with HTML as strings. Now we define a little response helper:

     (define* (respond #:optional body #:key
                       (status 200)
                       (title "Hello hello!")
                       (doctype "<!DOCTYPE html>\n")
                       (content-type-params '(("charset" . "utf-8")))
                       (content-type "text/html")
                       (extra-headers '())
                       (sxml (and body (templatize title body))))
       (values (build-response
                #:code status
                #:headers `((content-type
                             . (,content-type ,@content-type-params))
                            ,@extra-headers))
               (lambda (port)
                 (if sxml
                     (begin
                       (if doctype (display doctype port))
                       (sxml->xml sxml port))))))

Here we see the power of keyword arguments with default initializers. By the time the arguments are fully parsed, the sxml local variable will hold the templated SXML, ready for sending out to the client.

Also, instead of returning the body as a string, respond gives a procedure, which will be called by the web server to write out the response to the client.

Now, a simple example using this responder, which lays out the incoming headers in an HTML table.

     (define (debug-page request body)
       (respond
        `((h1 "hello world!")
          (table
           (tr (th "header") (th "value"))
           ,@(map (lambda (pair)
                    `(tr (td (tt ,(with-output-to-string
                                    (lambda () (display (car pair))))))
                         (td (tt ,(with-output-to-string
                                    (lambda ()
                                      (write (cdr pair))))))))
                  (request-headers request))))))

     (run-server debug-page)

Now if you visit any local address in your web browser, we actually see some HTML, finally.

Conclusion
..........

Well, this is about as far as Guile's built-in web support goes, for now.  There are many ways to make a web application, but hopefully by standardizing the most fundamental data types, users will be able to choose the approach that suits them best, while also being able to switch between implementations of the server.  This is a relatively new part of Guile, so if you have feedback, let us know, and we can take it into account.  Happy hacking on the web!