* libguile/read.c (scm_read_opts): Default "positions" to #t. The
compiler was already turning it on anyway, and this allows
primitive-load without --auto-compile to also propagate source
information through the expander, for better errors and to let macros
know their source.
* module/language/scheme/spec.scm: No need to enable positions here
now.
* libguile/private-options.h (SCM_HUNGRY_EOL_ESCAPES_P): New private
option.
* libguile/read.c: Define SCM_HUNGRY_EOL_ESCAPES_P, defaulting to #f.
(skip_intraline_whitespace): New helper.
(scm_read_string): If SCM_HUNGRY_EOL_ESCAPES_P,
skip_intraline_whitespace after an escaped EOL.
* test-suite/tests/reader.test ("read-options"): Add test.
* libguile/bytevectors.c:
* libguile/eval.c:
* libguile/goops.c:
* libguile/i18n.c:
* libguile/load.c:
* libguile/memoize.c:
* libguile/modules.c:
* libguile/ports.c:
* libguile/print.c:
* libguile/procs.c:
* libguile/programs.c:
* libguile/read.c:
* libguile/script.c:
* libguile/srfi-14.c:
* libguile/stacks.c:
* libguile/strings.c:
* libguile/throw.c:
* libguile/vm.c: Use scm_from_latin1_symboln to make symbols from string
literals, because they aren't in the user's locale -- they are in
ASCII, and we can optimize this case.
* libguile/vm-i-loader.c: Also use scm_from_latin1_symboln when loading
narrow symbols.
* libguile/bytevectors.c:
* libguile/goops.c:
* libguile/instructions.c:
* libguile/numbers.c:
* libguile/random.c:
* libguile/read.c:
* libguile/vm-i-scheme.c: Fix a number of assumptions that a long could
hold an inum. This is not the case on platforms whose void* is larger
than their long.
* libguile/numbers.c (scm_i_inum2big): New helper, only implemented for
sizeof(void*) == sizeof(long); produces a compile error on other
platforms. Basically gmp doesn't have a nice interface for converting
between mpz values and intmax_t.
* libguile/debug.c:
* libguile/eval.c:
* libguile/frames.c:
* libguile/objcodes.c:
* libguile/print.c:
* libguile/programs.c:
* libguile/read.c:
* libguile/struct.c:
* libguile/vm.c: Fix a number of instances in which we assumed we could
fit a pointer into a long.
This allows customizing the reader behavior for a dynamic extent more easily.
* libguile/read.c (scm_read_hash_procedures): Renamed to
`scm_i_read_hash_procedures'.
(scm_i_read_hash_procedures_ref, scm_i_read_hash_procedures_set_x):
New (internal) accessor functions for the fluid.
(scm_read_hash_extend, scm_get_hash_procedure): Use these accessor
functions.
(scm_init_read): Create the fluid, named `%read-hash-procedures' instead of
the previous plain list `read-hash-procedures'.
* test-suite/tests/reader.test: Adapt the "R6RS/SRFI-30 block comment
syntax overridden" test to make use of the fluid.
* module/ice-9/deprecated.scm (read-hash-procedures):
New identifier macro -- backward-compatibility shim.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* libguile/private-options.h (SCM_ELISP_VECTORS_P, SCM_ESCAPED_PARENS_P):
* libguile/read.c (scm_read_opts): Remove unused elisp-vectors option,
and the elisp-strings option (which allowed \( and \) escapes in
strings).
(scm_read_string): Remove the elisp-strings case.
* doc/ref/api-options.texi (Reader options): Update, and update wording
of the case-insensitive bit.
R6RS character hex escapes do not conflict with legacy Guile octal
character escapes, so they can be enabled by default.
* libguile/read.c (scm_read_character): modified
* test-suite/tests/reader.test: modify character escape tests
* doc/ref/api-data.texi: modified
* doc/ref/api-options.texi: modified
Note especially that the variable 'i' has two different uses in this
function, and they get confused.
* libguile/read.c (scm_i_scan_for_encoding): cleanup
* libguile/read.c (scm_read_shebang): New function;
(scm_read_sharp): Call scm_read_shebang on '!', which delegates to
scm_read_scsh_block_comment as necessary.
* test-suite/tests/reader.test ("R6RS lexeme comment", "partial R6RS
lexeme comment"): New tests.
* libguile/read.c (read_token): now takes a C buffer instead of a SCM.
string. All callers changed.
(read_complete_token): now takes C buffers, not SCM strings. No longer
does port position updates or encoding processing. All callers changed.
(scm_read_number, scm_read_mixed_case_symbol, scm_read_number_and_radix)
(scm_read_character): Do port updates and string processing no longer
done by read_complete_token. Some reordering for optimization.
* libguile/private-options.h:
* libguile/read.c (scm_read_opts, SCM_SQUARE_BRACKETS_P): Add an option
for treating [ and ] as parentheses, on by default. Note that this
makes them delimiters also, so [ and ] cannot appear in a symbol name,
with this read option on.
(scm_read_sexp): If we start with [, we end with ].
(scm_read_expression): Add case for [.
When the reader option 'r6rs-hex-escapes is enabled, the \uNNNN and
\UNNNNNN string escape sequences should be disabled.
* libguile/read.c (scm_read_string): added checks for SCM_R6RS_ESCAPES_P
This adds a reader option 'r6rs-hex-escapes that modifies the
behavior of numeric escapes in characters and strings. When enabled,
variable-length character hex escapes (#\xNNN) are allowed and become
the default output format for numerically-escaped characters. Also,
string hex escapes switch to a semicolon terminated hex escape (\xNNNN;).
* libguile/print.c (PRINT_CHAR_ESCAPE): new macro
(iprin1): use new macro PRINT_CHAR_ESCAPE
* libguile/private-options.h (SCM_R6RS_ESCAPES_P): new #define
* libguile/read.c (scm_read_opts): add new option r6rs-hex-escapes
(SCM_READ_HEX_ESCAPE): modify to take a terminator parameter
(scm_read_string): parse R6RS hex string escapes
(scm_read_character): parse R6RS hex character escapes
* test-suite/tests/chars.test (with-read-options): new procedure
(R6RS hex escapes): new tests
* test-suite/tests/strings.test (with-read-options): new procedure
(R6RS hex escapes): new tests
R6RS suggests that '\b' should be a string escape for the backspace
character.
* libguile/read.c (scm_read_string): parse backspace escape
* test-suite/tests/strings.test (R6RS backslash escapes): new test
(Guile extensions backslash escapes): remove R6RS escapes from test.
* doc/ref/api-data.texi (Strings): document new string escape
The commit "don't take string-write mutex in read.c:read_token", from
8b0d7b9d94, had a number of bugs. Not sure
how I missed these before.
* libguile/read.c (read_token): Remove a couple of bogus
scm_i_string_stop_writing () calls, now that we no longer take the
string-write mutex.
(read_complete_token): read_token really needs a fresh buffer, which
was not the case when we are reading long tokens and thus hit the
overflow case. Fixes fractions.test.
* libguile/read.c (read_token): Don't take the string-write mutex when
reading a token into a buffer, because it's assumed that the buffer is
fresh (not seen by other threads), and a soft port can call a
procedure that needs the string-write mutex.
They made Sun C 5.8 emit a warning such as:
line 71: warning: dead part of constant expression is nonconstant
* libguile/print.c (scm_print_opts): Don't use `SCM_UNPACK ()' here.
* libguile/read.c (scm_read_opts): Likewise.
* doc/ref/api-evaluation.texi (Character Encoding of Source Files):
Don't suggest `latin1' as a good encoding name since Emacs cannot deal
with it.
* libguile/read.c (scm_file_encoding): Fix "Emacs" spelling.
* libguile/read.c (scm_scan_for_encoding): Rename to ...
(scm_i_scan_for_encoding): ... this; update callers. Use
`scm_gc_strndup ()' instead of `scm_malloc ()'.
* libguile/read.h: Update accordingly.
* libguile/load.c (scm_primitive_load): Don't call free(3) on the value
returned by `scm_i_scan_for_encoding ()'.
Since combining characters, such as accents, modify the appearance of the
previous letter, it looks awkward in its character literal form (#\name)
since it modified the backslash. This instead prints the combining
character on a small circle.
* libguile/chars.h (SCM_CODEPOINT_DOTTED_CIRCLE): new #define
* libguile/print.c (iprint1): print combining characters on dotted circles
* libguile/read.c (scm_read_character): parse the combination of combining
characters and dotted circles
* libguile/gc_os_dep.c (GC_linux_stack_base) [LINUX_STACKBOTTOM]: cast
input of ctype functions to int
* libguile/inet_aton.c (inet_aton): cast input of ctype functions to int
* libguile/read.c (scm_scan_for_encoding): cast input of isalnum to int
* libguile/win32-socket.c (scm_i_socket_uncomment): cast input of isspace
to int
Ports are given two additional properties: a character encoding and
a conversion failure strategy. These properties have getters and setters.
The new properties are used to convert any locale text to/from the
internal representation of strings.
If unspecified, ports use a default value. The default value of these
properties is held in a fluid. The default character encoding can be
modified by calling setlocale.
ISO-8859-1 is treated specially. Since it is a native encoding of
strings, it can be processed more quickly. Source code is assumed to be
ISO-8859-1 unless otherwise specified. The encoding of a source code
file can be given as 'coding: XXXXX' in a magic comment at the top of a
file.
The C functions that deal with encoding often use a null pointer
as shorthand for the native Latin-1 encoding, for efficiency's sake.
* test-suite/tests/encoding-iso88591.test: new tests
* test-suite/tests/encoding-iso88597.test: new tests
* test-suite/tests/encoding-utf8.test: new tests
* test-suite/tests/encoding-escapes.test: new tests
* test-suite/tests/numbers.test: declare 'binary' encoding
* test-suite/tests/ports.test: declare 'binary' encoding
* test-suite/tests/r6rs-ports.test: declare 'binary' encoding
* module/system/base/compile.scm (compile-file): use source-code
file's self-declared encoding when compiling files
* libguile/strports.c: store string ports in locale encoding
(scm_strport_to_locale_u8vector, scm_call_with_output_locale_u8vector)
(scm_open_input_locale_u8vector, scm_get_output_locale_u8vector):
new functions
* libguile/strings.h: new declaration for scm_i_string_contains_char
* libguile/strings.c (scm_i_string_contains_char): new function
(scm_from_stringn, scm_to_stringn): use NULL for Latin-1
(scm_from_locale_stringn, scm_to_locale_stringn): respect character
encoding of input and output ports
* libguile/read.h: declaration for scm_scan_for_encoding
* libguile/read.c:
(read_token): now takes scheme string instead of C string/length
(read_complete_token): new function
(scm_read_sexp, scm_read_number, scm_read_mixed_case_symbol)
(scm_read_number_and_radix, scm_read_quote, scm_read_semicolon_comment)
(scm_read_srfi4_vector, scm_read_bytevector, scm_read_guile_bit_vector)
(scm_read_scsh_block_comment, scm_read_commented_expression)
(scm_read_extended_symbol, scm_read_sharp_extension, scm_read_shart)
(scm_read_expression): use scm_t_wchar for char type, use read_complete_token
(scm_scan_for_encoding): new function to find a file's character encoding
(scm_file_encoding): new function to find a port's character encoding
* libguile/rdelim.c: don't unpack strings
* libguile/print.h: declaration for modified function
scm_i_charprint
* libguile/print.c: use locale when printing characters and
strings
(scm_i_charprint): input parameter is now scm_t_wchar
(scm_simple_format): don't unpack strings
* libguile/posix.h: new declaration for scm_setbinary.
* libguile/posix.c (scm_setlocale): set default and stdio port
encodings based on the locale's character encoding
(scm_setbinary): new function
* libguile/ports.h (scm_t_port): add encoding and failed
conversion handler to port type. Declarations for new or modified
functions scm_getc, scm_unget_byte, scm_ungetc,
scm_i_get_port_encoding, scm_i_set_port_encoding_x,
scm_port_encoding, scm_set_port_encoding_x,
scm_i_get_conversion_strategy, scm_i_set_conversion_strategy_x,
scm_port_conversion_strategy, scm_set_port_conversion_strategy_x.
* libguile/ports.c: assign the current ports to zero on startup so
we can see if they've been set.
(scm_current_input_port, scm_current_output_port,
scm_current_error_port): return #f if the port is not yet
initialized
(scm_new_port_table_entry): set up a new port's encoding and
illegal sequence handler based on the thread's current defaults
(scm_i_remove_port): free port encoding name when port is removed
(scm_i_mode_bits_n): now takes a scheme string instead of a c
string and length. All callers changed.
(SCM_MBCHAR_BUF_SIZE): new const
(scm_getc): new function, since the scm_getc in inline.h is now
scm_get_byte_or_eof. This pulls one codepoint from a port.
(scm_lfwrite_substr, scm_lfwrite_str): now uses port's encoding
(scm_unget_byte): new function, incorportaing the low-level functionality
of scm_ungetc
(scm_ungetc): uses scm_unget_byte
* libguile/numbers.h (scm_t_wchar): compilation order problem with
scm_t_wchar being use in functions in multiple headers. Forward
declare scm_t_wchar.
* libguile/load.c (scm_primitive_load): scan for file encoding at
top of file and use it to set the load port's encoding
* libguile/inline.h (scm_get_byte_or_eof): new function
incorporating most of the functionality of scm_getc.
* libguile/fports.c (fport_fill_input): now returns scm_t_wchar
* libguile/chars.h (scm_t_wchar): avoid compilation order problem
with declaration of scm_t_wchar
* libguile/strings.c (unistring_escapes_to_guile_escapes): cast
tolower's parameter to int
* libguile/read.c (CHAR_DOWNCASE): cast tolower's parameter to int
This adds full Unicode strings as a datatype, and it adds some
minimal functionality. The terminal and port encoding is assumed
to be ISO-8859-1. Non-ISO-8859-1 characters are written or
input as string character escapes.
The string character escapes now have 3 forms: \xXX \uXXXX and
\UXXXXXX, for unprintable characters that have 2, 4 or 6 hex digits.
The process for writing to strings has been modified. There is now a
function scm_i_string_start_writing that does the copy-on-write
conversion if necessary.
To compile strings that may be wide, the VM storage of strings and
string-likes has changed.
Most string-using functions have not yet been updated and may break
when used with wide strings.
* module/language/assembly/compile-bytecode.scm (write-bytecode):
use variable width string bytecode format
* module/language/assembly.scm (byte-length): use variable width
bytecode format
* libguile/vm-i-loader.c (load-string, load-symbol):
(load-keyword, define): use variable-width bytecode format
* libguile/vm-engine.h (FETCH_WIDTH): new macro
* libguile/strings.h: new declarations
* libguile/strings.c (make_wide_stringbuf): new function
(widen_stringbuf): new function
(scm_i_make_wide_string): new function
(scm_i_is_narrow_string): new function
(scm_i_string_wide_chars): new function
(scm_i_string_start_writing): new function
(scm_i_string_ref): new function
(scm_i_string_set_x): new function
(scm_i_is_narrow_symbol): new function
(scm_i_symbol_wide_chars, scm_i_symbol_ref): new function
(scm_string_width): new function
(unistring_escapes_to_guile_escapes): new function
(scm_to_stringn): new function
(scm_i_stringbuf_free): modify for wide strings
(scm_i_substring_copy): modify for wide strings
(scm_i_string_chars, scm_string_append): modify for wide strings
(scm_i_make_symbol, scm_to_locale_stringn): modify for wide strings
(scm_string_dump, scm_symbol_dump, scm_to_locale_stringbuf):
(scm_string, scm_i_deprecated_string_chars): modify for wide strings
(scm_from_locale_string, scm_from_locale_stringn): add null test
* libguile/srfi-13.c: add calls for scm_i_string_start_writing for
each call of scm_i_string_stop_writing
(scm_string_for_each): modify for wide strings
* libguile/socket.c: add calls for scm_i_string_start_writing for each
call of scm_i_string_stop_writing
* libguile/rw.c: add calls for scm_i_string_start_writing for each
call of scm_i_string_stop_writing
* libguile/read.c (scm_read_string): allow reading of wide strings
* libguile/print.h: add declaration for scm_charprint
* libguile/print.c (iprin1): print wide strings and add new string
escapes
(scm_charprint): new function
* libguile/ports.h: new declarations for scm_lfwrite_substr and
scm_lfwrite_str
* libguile/ports.c (update_port_lf): new function
(scm_lfwrite): use update_port_lf
(scm_lfwrite_substr): new function
(scm_lfwrite_str): new function
* test-suite/tests/asm-to-bytecode.test ("compiler"): add string
width byte to sting-like asm tests
The global variables scm_charnames and scm_charnums are replaced with
the accessor functions scm_i_charname and scm_i_charname_to_num.
Also, the incomplete and broken EBCDIC support is removed.
* libguile/print.c (iprin1): use new func scm_i_charname
* libguile/read.c (scm_read_character): use new func
scm_i_charname_to_num
* libguile/chars.c (scm_i_charname): new function
(scm_i_charname_to_char): new function
(scm_charnames, scm_charnums): removed
* libguile/chars.h: new declarations
