Fixes <https://bugs.gnu.org/41204>.
Reported by Caleb Ristvedt <caleb.ristvedt@cune.org>.
Until now, Guile would leak the file descriptors of all the file ports
not explicitly closed.
* libguile/fports.c (scm_i_fdes_to_port): Initialize fp->revealed.
* test-suite/tests/ports.test ("initial revealed count")
("non-revealed port is closed"): New tests.
("close-port & revealed port"): Check (port-revealed port).
("revealed port fdes not closed"): Likewise.
* libguile/fports.c (scm_i_fdes_is_valid): New internal helper.
(scm_i_fdes_to_port): Use new helper.
* libguile/fports.h: Declare new helper.
* libguile/init.c (scm_standard_stream_to_port): Refactor to use
scm_i_fdes_is_valid.
This is a followup to commit 91ba73b397.
* libguile/fports.c (fport_seek): Use 'lseek' instead of
'lseek_or_lseek64', and use 'scm_t_off' uniformly. That's the type used
in the function signature, and there's no benefit to using a wider type
internally. Remove the overflow check, which is no longer needed.
* libguile/bytevectors.c (INTEGER_ACCESSOR_PROLOGUE)
(scm_bytevector_copy_x, bytevector_large_set): Rewrite checks to reliably
detect overflows.
(make_bytevector): Constrain the bytevector length to avoid later
overflows during allocation.
(make_bytevector_from_buffer): Fix indentation.
(scm_bytevector_length): Use 'scm_from_size_t' to convert a 'size_t',
not 'scm_from_uint'.
* libguile/fports.c (fport_seek): Check for overflow before the implicit
conversion of the return value.
* libguile/guardians.c (guardian_print): Use 'scm_from_ulong' to convert
an 'unsigned long', not 'scm_from_uint'.
* libguile/ports.c (scm_unread_string): Change a variable to type 'size_t'.
(scm_seek, scm_truncate_file): Use 'scm_t_off' instead of
'off_t_or_off64_t' to avoid implicit type conversions that could
overflow, because 'ptob->seek' and 'ptob->truncate' use 'scm_t_off'.
* libguile/r6rs-ports.c (bytevector_input_port_seek)
(custom_binary_port_seek, bytevector_output_port_seek): Rewrite offset
calculations to reliably detect overflows. Use 'scm_from_off_t' to
convert a 'scm_t_off', not 'scm_from_long' nor 'scm_from_int'.
(scm_get_bytevector_n_x, scm_get_bytevector_all, scm_unget_bytevector)
(bytevector_output_port_write): Rewrite checks to reliably detect
overflows. Use 'size_t' where appropriate.
(bytevector_output_port_buffer_grow): Rewrite size calculations to
reliably detect overflows. Minor change in the calculation of the new
size: now it is max(min_size, 2*current_size), whereas previously it
would multiply current_size by the smallest power of 2 needed to surpass
min_size.
* libguile/strings.c (make_stringbuf): Constrain the stringbuf length to
avoid later overflows during allocation.
(scm_string_append): Change overflow check to use INT_ADD_OVERFLOW.
* libguile/strports.c (string_port_write): Rewrite size calculations to
reliably detect overflows.
(string_port_seek): Rewrite offset calculations to reliably detect
overflows. Use 'scm_from_off_t' to convert a 'scm_t_off', not
'scm_from_long'.
(string_port_truncate): Use 'scm_from_off_t' to convert a 'scm_t_off',
not 'scm_from_off_t_or_off64_t'.
* libguile/vectors.c (scm_c_make_vector): Change a variable to type
'size_t'.
As the FSF advises, 'There is no legal significance to using the
three-character sequence “(C)”, but it does no harm.' It does take up
space though! For that reason, we remove it here from our C files.
* libguile/bytevectors.h: Include uniform.h, for use in the macros.
* libguile/extensions.h: Include libpath.h, for the
SCM_EFFECTIVE_VERSION, which is almost always used with these
routines.
* libguile/frames.h:
* libguile/instructions.h:
* libguile/intrinsics.h:
* libguile/loader.h:
* libguile/programs.h:
* libguile/vm.h: Include <libguile/__scm.h> instead of <libguile.h>.
Cuts a circular include, but also precipitates a lot of maintenance in
the .c files.
* libguile/*.c: Update C files to add needed all needed includes that
before were getting automatically pulled in by the indirect inclusion
of libguile.h.
Reported at <https://bugs.gnu.org/28784>.
Discussed at
<https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00003.html>.
* libguile/fports.c (revealed_ports, revealed_lock): Remove.
(scm_revealed_count): Just return 'SCM_REVEALED (port)'.
(scm_set_port_revealed_x, scm_adjust_port_revealed_x): Remove
REVEALED_PORTS manipulation.
(fport_close): Do nothing when SCM_REVEALED (port) > 0.
* libguile/fports.h (scm_t_fport): Adjust comment; make 'revealed'
unsigned.
* libguile/ports.c (do_close): Call 'close_port' instead of
'scm_close_port'.
(scm_close_port): Rename to...
(close_port): ... this. Add 'explicit' parameter. Clear 'revealed'
field when PORT is a file port and EXPLICIT is true.
(scm_close_port): Call 'close_port'.
* test-suite/tests/ports.test ("close-port & revealed port")
("revealed port fdes not closed"): New tests.
* libguile/__scm.h (SCM_TICK): Always define as scm_async_tick().
* libguile/error.c (scm_syserror, scm_syserror_msg):
* libguile/fports.c (fport_read, fport_write):
* libguile/_scm.h (SCM_SYSCALL): Replace SCM_ASYNC_TICK with
scm_async_tick ().
(SCM_ASYNC_TICK, SCM_ASYNC_TICK_WITH_CODE)
(SCM_ASYNC_TICK_WITH_GUARD_CODE): Remove internal definitions. We
inline into vm-engine.c, the only place where it matters.
* libguile/async.h:
* libguile/async.c (scm_async_tick, scm_i_setup_sleep):
(scm_i_reset_sleep, scm_system_async_mark_for_thread):
* libguile/threads.h (struct scm_thread_wake_data):
* libguile/threads.h (scm_i_thread):
* libguile/threads.c (block_self, guilify_self_1, scm_std_select):
Rewrite to use sequentially-consistent atomic references.
* libguile/atomics-internal.h (scm_atomic_set_pointer):
(scm_atomic_ref_pointer): New definitions.
* libguile/finalizers.c (queue_finalizer_async): We can allocate, so
just use scm_system_async_mark_for_thread instead of the set-cdr!
shenanigans.
* libguile/scmsigs.c (take_signal):
* libguile/gc.c (queue_after_gc_hook): Adapt to new asyncs mechanism.
Can't allocate but we're just manipulating the current thread when no
other threads are running so we should be good.
* libguile/vm-engine.c (VM_HANDLE_INTERRUPTS): Inline the async_tick
business.
* m4/mkstemp.m4: Remove.
* lib/mkstemp.c: Remove.
* lib/mkostemp.c: New file.
* m4/mkostemp.m4: New file.
* lib/Makefile.am:
* m4/gnulib-cache.m4:
* m4/gnulib-comp.m4: Remove mkstemp module, replace with mkostemp.
* libguile/fports.h:
* libguile/fports.c (scm_i_mode_to_open_flags): Factor out helper to
parse mode string to open flags.
(scm_open_file_with_encoding): Use the new helper.
* libguile/filesys.c:
(scm_i_mkstemp): Adapt to take optional second argument, being a mode
string. Use mkostemp.
(scm_mkstemp): Backwards compatible shim that calls scm_i_mkstemp.
* doc/ref/posix.texi:
* NEWS: Update.
* module/system/base/compile.scm (call-with-output-file/atomic): Pass
"wb" as mode, to cause O_BINARY to be added on MinGW.
This removes a limitation on the number of port types, simplifies the
API, and removes a central point of coordination.
* libguile/ports-internal.h (struct scm_t_port_type): Rename from
scm_t_ptob_descriptor, now that it's private. Add GOOPS class
fields.
(struct scm_t_port): Rename from struct scm_port, especially
considering that deprecated.h redefines scm_port using the
preprocessor :(.
* libguile/ports.h: Add definitions of SCM_PORT and SCM_PORT_TYPE,
though the scm_t_port and scm_t_port_type types are incomplete.
(SCM_TC2PTOBNUM, SCM_PTOBNUM, SCM_PTOBNAME): Remove, as there are no
more typecodes for port types.
(scm_c_num_port_types, scm_c_port_type_ref, scm_c_port_type_add_x):
Remove.
(scm_make_port_type): Return a scm_t_port_type*. All methods adapted
to take a scm_t_port_type* instead of a ptobnum.
(scm_c_make_port_with_encoding, scm_c_make_port): Take a port type
pointer instead of a tag.
(scm_new_port_table_entry): Remove; not useful.
* libguile/ports.c: Remove things related to the port kind table. Adapt
uses of SCM_PORT_DESCRIPTOR / scm_t_ptob_descriptor to use
SCM_PORT_TYPE and scm_t_port_type.
* libguile/deprecated.c:
* libguile/deprecated.h:
* libguile/filesys.c:
* libguile/fports.c:
* libguile/fports.h:
* libguile/print.c:
* libguile/r6rs-ports.c:
* libguile/strports.c:
* libguile/strports.h:
* libguile/tags.h:
* libguile/vports.c:
* test-suite/standalone/test-scm-c-read.c: Adapt to change.
* libguile/goops.c (scm_class_of, make_port_classes)
(scm_make_port_classes, create_port_classes): Adapt to store the
classes in the ptob.
* libguile/ports-internal.h (enum scm_port_encoding_mode): Remove unused
enum.
(scm_t_port_internal, scm_t_port): Make encoding and
conversion_strategy private. Instead of scm_t_port_internal containing
scm_t_port, now that all members are private, we can store the user's
"stream" in a word in the port object itself and make the whole of
scm_t_port private. The next commit will remove scm_t_port_internal.
(SCM_PTAB_ENTRY, SCM_PORT_DESCRIPTOR): Make private.
* libguile/ports.c (scm_c_make_port_with_encoding): Adapt to new port
layout.
(scm_port_print): Use SCM_PTAB_ENTRY when printing.
* libguile/ports.h: Remove scm_t_port definition.
* libguile/ioext.c (get_matching_port): Simplify.
* libguile/fports.c (scm_i_evict_port): Simplify.
* doc/ref/api-io.texi (I/O Extensions): Update documentation on
implementing port types. Document get_natural_buffer_sizes. Document
the new random_access_p.
* libguile/fports.c (scm_i_fdes_to_port, fport_random_access_p):
(scm_make_fptob): Instead of frobbing rw_random manually, implement a
random_access_p function.
* libguile/ports.c (default_random_access_p)
(scm_set_port_random_access_p): New functions.
scm_make_port_type, scm_c_make_port_with_encoding): Arrange for
random_access_p to work.
* module/ice-9/ports.scm: New file.
* am/bootstrap.am (SOURCES): Add ice-9/ports.scm.
* libguile/fports.c (scm_init_ice_9_fports): New function.
(scm_init_fports): Arrange for scm_init_ice_9_fports to be called via
load-extension, and load snarfed things there. Move open-file
definition early, to allow ports to bootstrap.
* libguile/ioext.c (scm_init_ice_9_ioext): New function.
(scm_init_ioext): Similarly, register scm_init_ice_9_ioext as an
extension.
* libguile/ports.c (scm_set_current_input_port)
(scm_set_current_output_port, scm_set_current_error_port): Don't
define Scheme bindings; do that in Scheme.
* libguile/ports.c (scm_i_set_default_port_encoding):
(scm_i_default_port_encoding, scm_i_default_port_conversion_handler):
(scm_i_set_default_port_conversion_handler): Since we now init
encoding early, remove the "init" flags on these encoding/strategy
vars.
(scm_init_ice_9_ports): New function.
(scm_init_ports): Register scm_init_ice_9_ports extension, and define
some bindings needed by the bootstrap.
* module/Makefile.am (SOURCES): Add ice-9/ports.scm.
* module/ice-9/boot-9.scm: Remove code that's not on the boot path,
moving it to ice-9/ports.scm. At the end, load (ice-9 ports).
* module/ice-9/psyntax.scm (include): Use close-port instead of
close-input-port.
* module/ice-9/psyntax-pp.scm (include): Regenerate.
This will allow better Scheme integration for ports.
* libguile/ports.h (scm_t_port_buffer): Change "holder" member to be a
bytevector defined to have "buf" as its starting point.
(scm_t_ptob_descriptor): Change read and write functions to take
bytevectors as arguments and to return the number of octets read or
written.
(scm_make_port_type): Adapt accordingly.
(scm_c_read_bytes, scm_c_write_bytes): New functions that take
bytevectors.
* libguile/ports.c (scm_make_port_type): Adapt to read/write function
prototype change.
(scm_c_make_port_buffer): Arrange to populate the "bytevector" field.
(scm_i_read_bytes_unlocked): New function.
(scm_i_read_unlocked): Use scm_i_read_bytes_unlocked.
(scm_c_read_bytes_unlocked): New function.
(scm_c_read_unlocked): Update comment, and always go through the
buffer.
(scm_c_read_bytes): New function.
(scm_flush_unlocked): Use scm_i_write_unlocked instead of the port's
write function.
(scm_i_write_bytes_unlocked): New function.
(scm_i_write_unlocked): Use scm_i_write_bytes_unlocked.
(scm_c_write_bytes_unlocked): New function.
(scm_c_write_unlocked): Always write through the buffer.
(scm_c_write_bytes): New function.
(scm_truncate_file): Remove unused variable.
(void_port_read, void_port_write): Adapt to read/write prototype
change.
* libguile/fports.c (fport_read, fport_write):
* libguile/r6rs-ports.c (bytevector_input_port_read)
(custom_binary_input_port_read, bytevector_output_port_write)
(custom_binary_output_port_write, transcoded_port_write)
(transcoded_port_read): Adapt to read/write prototype
change.
(scm_get_bytevector_n, scm_get_bytevector_n_x)
(scm_get_bytevector_all): Use scm_c_read_bytes.
(scm_put_bytevector): Use scm_c_write_bytes.
* libguile/strports.c (string_port_read, string_port_write):
* libguile/vports.c (soft_port_write, soft_port_read): Adapt to
read/write prototype change.
* test-suite/standalone/test-scm-c-read.c (custom_port_read): Fix for
read API change.
* libguile/ports.h (struct scm_t_port_buffer): New data type.
(struct scm_t_port): Refactor to use port buffers instead of
implementation-managed read and write pointers. Add "read_buffering"
member.
(SCM_INITIAL_PUTBACK_BUF_SIZE, SCM_READ_BUFFER_EMPTY_P): Remove.
(scm_t_ptob_descriptor): Rename "fill_input" function to "read", and
take a port buffer, returning void. Likewise "write" takes a port
buffer and returns void. Remove "end_input"; instead if there is
buffered input and rw_random is true, then there must be a seek
function, so just seek back if needed. Remove "flush"; instead all
calls to the "write" function implicitly include a "flush", since the
buffering happens in the generic port code now. Remove "setvbuf", but
add "get_natural_buffer_sizes"; instead the generic port code can
buffer any port.
(scm_make_port_type): Adapt to read and write prototype changes.
(scm_set_port_flush, scm_set_port_end_input, scm_set_port_setvbuf):
Remove.
(scm_slow_get_byte_or_eof_unlocked)
(scm_slow_get_peek_or_eof_unlocked): Remove; the slow path is to call
scm_fill_input.
(scm_set_port_get_natural_buffer_sizes): New function.
(scm_c_make_port_buffer): New internal function.
(scm_port_non_buffer): Remove. This was a function for
implementations that is no longer needed. Instead open with BUF0 or
use (setvbuf port 'none).
(scm_fill_input, scm_fill_input_unlocked): Return the filled port
buffer.
(scm_get_byte_or_eof_unlocked, scm_peek_byte_or_eof_unlocked): Adapt
to changes in buffering and EOF management.
* libguile/ports.c: Adapt to port interface changes.
(initialize_port_buffers): New function, using the port mode flags to
set up appropriate initial buffering for all ports.
(scm_c_make_port_with_encoding): Create port buffers here instead of
delegating to implementations.
(scm_close_port): Flush the port if needed instead of delegating to
the implementation.
* libguile/filesys.c (set_element): Adapt to buffering changes.
* libguile/fports.c (fport_get_natural_buffer_sizes): New function,
replacing scm_fport_buffer_add.
(fport_write, fport_read): Update to let the generic ports code do the
buffering.
(fport_flush, fport_end_input): Remove.
(fport_close): Don't flush in a dynwind; that's the core ports' job.
(scm_make_fptob): Adapt.
* libguile/ioext.c (scm_redirect_port): Adapt to buffering changes.
* libguile/poll.c (scm_primitive_poll): Adapt to buffering changes.
* libguile/ports-internal.h (struct scm_port_internal): Remove
pending_eof flag; this is now set on the read buffer.
* libguile/r6rs-ports.c (struct bytevector_input_port): New type. The
new buffering arrangement means that there's now an intermediate
buffer between the bytevector and the user of the port; this could
lead to a perf degradation, but on the other hand there are some other
speedups enabled by the buffering refactor, so probably the memcpy
cost is dwarfed by the cost of the other parts of the ports
machinery.
(make_bytevector_input_port, bytevector_input_port_read):
(bytevector_input_port_seek, initialize_bytevector_input_ports): Adapt
to new buffering arrangement.
(struct custom_binary_port): Remove read buffer, as Guile handles that
now.
(custom_binary_input_port_setvbuf): Remove; now handled by Guile.
(make_custom_binary_input_port, custom_binary_input_port_read)
(initialize_custom_binary_input_ports): Adapt.
(scm_get_bytevector_some): Adapt to new EOF management.
(scm_t_bytevector_output_port_buffer): Hold on to the underlying port,
so we can flush it if it's open.
(make_bytevector_output_port, bytevector_output_port_write):
(bytevector_output_port_seek): Adapt.
(bytevector_output_port_procedure): Flush the port as appropriate, so
that we get all the bytes.
(make_custom_binary_output_port, custom_binary_output_port_write):
Adapt.
(make_transcoded_port): Don't muck with buffering.
(transcoded_port_write): Simply forward the write to the underlying
port.
(transcoded_port_read): Likewise.
(transcoded_port_close): No need to flush.
(initialize_transcoded_ports): Adapt.
* libguile/read.c (scm_i_scan_for_encoding): Adapt to buffering
changes.
* libguile/rw.c (scm_write_string_partial): Adapt to buffering changes.
* libguile/strports.c: Adapt to the fact that we don't manage the
buffer. Probably room for speed improvements here...
* libguile/vports.c (soft_port_get_natural_buffer_sizes): New function.
Adapt the rest of the file for the new buffering regime.
* test-suite/tests/r6rs-ports.test ("8.2.10 Output ports"): Custom
binary output ports need to be flushed before you can rely on the
write! procedure having been called. Add necessary flush-port
invocations.
("8.2.6 Input and output ports"): Transcoded ports now have an
internal buffer by default. This test checks that the characters are
transcoded one at a time, so to do that, call setvbuf on the
transcoded port to remove the buffer.
* test-suite/tests/web-client.test (run-with-http-transcript): Fix for
different flushing regime on soft ports. (The vestigial flush
procedure is now called after each write, which is not what the test
was expecting.)
* test-suite/standalone/test-scm-c-read.c: Update for changes to the C
interface for defining port types.
* doc/ref/api-io.texi (Ports): Update to discuss buffering in a generic
way, and to remove a hand-wavey paragraph describing string ports as
"interesting and powerful".
(Reading, Writing): Remove placeholder comments. Document
`scm_lfwrite'.
(Buffering): New section.
(File Ports): Link to buffering.
(I/O Extensions): Join subnodes into parent and describe new API,
including buffering API.
* doc/ref/posix.texi (Ports and File Descriptors): Link to buffering.
Remove unread-char etc, as they are documented elsewhere.
(Pipes, Network Sockets and Communication): Link to buffering.
* libguile/ports.h (scm_t_ptob_descriptor): The port close function now
returns void.
(scm_set_port_close): Adapt prototype.
* libguile/ports.c (scm_close_port): Always return true if we managed to
call the close function. There's no other sensible result; exceptions
are handled, well, exceptionally.
* libguile/fports.c (fport_close)
* libguile/r6rs-ports.c (custom_binary_port_close, transcoded_port_close):
* libguile/vports.c (soft_port_close): Adapt.
* doc/ref/api-io.texi (Port Implementation): Update.
* libguile/ports.c (scm_seek): Flush before seeking on a buffered port.
* libguile/fports.c (fport_seek):
* libguile/strports.c (st_seek): Remove code to flush buffers.
* test-suite/tests/ports.test: Update test expectations that the
putback buffer is flushed on a seek. Previously there was a special
case for SEEK_CUR with an offset of 0 to avoid flushing buffers, but
that's an arbitrary choice that differs from all other combinations of
OFFSET and WHENCE.
* libguile/fports.c (fport_flush, fport_end_input): Move rw_active
handling to ports.c.
* libguile/ioext.c (scm_redirect_port): Use scm_flush_unlocked instead
of calling the flush function directly.
* libguile/ports.c (scm_c_make_port_with_encoding): Ports default to
"rw_random" mode when they have a seek function.
(scm_c_read_unlocked, scm_i_unget_bytes_unlocked)
(scm_slow_get_byte_or_eof_unlocked)
(scm_slow_peek_byte_or_eof_unlocked): Flush write buffer and set
rw_active always in the same way, and only if rw_random is true.
(scm_end_input_unlocked, scm_flush_unlocked): Clear rw_active here
unconditionally.
(scm_c_write_unlocked): Flush read buffer and set rw_active always in
the same way, but only if rw_random is true.
(scm_c_write, scm_lfwrite): Whitespace fixes.
(scm_lfwrite_substr): Don't flush read buffer; lower-level code will
do this.
(scm_truncate_file): Use scm_flush_unlocked instead of calling the
flush function directly.
* libguile/r6rs-ports.c (transcoded_port_flush): Don't muck with
rw_active.
* libguile/read.c (scm_i_scan_for_encoding): Flush write buffer if
needed in same way as ports.c.
* libguile/strports.c (st_end_input): Don't muck with rw_active.
(scm_mkstrport): rw_random defaults to 1 now.
* libguile/ports.h (scm_t_port_type_flags): Replace
SCM_PORT_TYPE_HAS_FLUSH with SCM_PORT_TYPE_NEEDS_CLOSE_ON_GC.
(scm_t_ptob_descriptor): Remove free function.
* libguile/ports.c (scm_set_port_needs_close_on_gc): New function.
(scm_set_port_flush): Don't set flags.
(scm_c_make_port_with_encoding, scm_close_port): Use the new flag to
determine when to add a finalizer and also when to include the port in
the weak set.
(scm_set_port_free): Remove.
(do_close, finalize_port): Close port instead of calling free
function.
* libguile/r6rs-ports.c (initialize_transcoded_ports):
* libguile/vports.c (scm_make_sfptob):
* libguile/fports.c (scm_make_fptob): Mark these ports as needing close
on GC.
* libguile/fports.c (fport_free): Remove.
* NEWS: Update.
* doc/ref/api-io.texi (Port Implementation): Update.
* libguile/fports.c (scm_setvbuf): Initialize 'drained'. GCC 4.9 raises
a "may be used uninitialized warning" for 'drained', because it can't
tell that 'drained' is initialized anytime NDRAINED > 0.
