1
Fork 0
mirror of https://https.git.savannah.gnu.org/git/guix.git/ synced 2025-07-10 16:50:43 +02:00
Commit graph

151 commits

Author SHA1 Message Date
Congcong Kuo
e3bd9a65cc
daemon: Remove ‘defined’ in macro definition.
Closes #6.

* nix/libstore/build.cc (CHROOT_ENABLED, CLONE_ENABLED): Wrap in #ifdefs.

Change-Id: I217e46fc2cac579a199fcd7c28ef5a8155a12750
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2025-05-25 17:47:03 +02:00
Congcong Kuo
5f3518ca83
daemon: Replace ‘random_shuffle’ with ‘shuffle’.
‘std::random_shuffle’ was removed in C++14.

* nix/libstore/gc.cc (LocalStore::collectGarbage): Use ‘std::random’ and
‘std::shuffle’.

Change-Id: If91ed3ec3596a419ae7c87d7ce677e0970853e9f
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2025-05-12 10:40:02 +02:00
Ludovic Courtès
0d3bc50b0c
daemon: Use the guest GID in /etc/group.
Partly fixes <https://issues.guix.gnu.org/77862>.

Fixes a bug whereby, when running guix-daemon unprivileged, /etc/group
would contain the wrong GID for the “nixbld” group.  This inconsistency
would lead to failures in the Coreutils test suite, for instance.

* nix/libstore/build.cc (DerivationGoal::startBuilder): Use ‘guestGID’
when writing /etc/group.
* tests/store.scm ("/etc/passwd and /etc/group"): New test.

Reported-by: keinflue <keinflue@posteo.net>
Change-Id: I739bc96c4c935fd9015a45e2bfe5b3e3f90554a9
2025-04-25 20:25:54 +02:00
Ludovic Courtès
ff5181e27e
daemon: Do not make chroot root directory read-only.
Fixes <https://issues.guix.gnu.org/77570>.

Commit 40f69b586a made chroot root
directory read-only; as a consequence, build processes attempting to
write to the root directory would now get EROFS instead of EACCES.

It turns out that a number of test suites (Go, Ruby, SCons, Shepherd)
would fail because of this observable difference.

To restore previous behavior in build environments while still
preventing build processes from exposing their root directory to outside
processes, this patch (1) keeps the root writable but #o555 by default,
thereby restoring the EACCES behavior, and (2) ensures that the parent
of the chroot root directory is itself user-accessible only.

* nix/libstore/build.cc (class DerivationGoal)[chrootRootTop]: New
field.
(DerivationGoal::startBuilder): Initialize ‘chrootRootTop’ and make it
‘AutoDelete’.  Replace ‘mount’ call that made the root directory
read-only by a mere ‘chmod_’ call.
* tests/store.scm ("build root cannot be made world-readable"): Remove.
("writing to build root leads to EACCES"): New test.

Reported-by: Ada Stevenson <adanskana@gmail.com>
Reported-by: keinflue <keinflue@posteo.net>
Suggested-by: Reepca Russelstein <reepca@russelstein.xyz>
Change-Id: I5912e8b3b293f8242a010cfc79255fc981314445
2025-04-11 12:18:01 +02:00
Ludovic Courtès
bdd7b9a45d
daemon: Move comments where they belong.
* nix/libstore/build.cc (DerivationGoal::startBuilder): Shuffle
comments for clarity.

Change-Id: I6557c103ade4a3ab046354548ea193c68f8c9c05
2025-03-26 17:57:44 +01:00
Ludovic Courtès
0163c732a1
daemon: Drop Linux ambient capabilities before executing builder.
* config-daemon.ac: Check for <sys/prctl.h>.
* nix/libstore/build.cc (DerivationGoal::runChild): When ‘useChroot’ is
true, call ‘prctl’ to drop all ambient capabilities.

Change-Id: If34637fc508e5fb6d278167f5df7802fc595284f
2025-03-26 17:57:44 +01:00
Ludovic Courtès
a3d6f5ae70
daemon: Create /var/guix/profiles/per-user unconditionally.
* nix/libstore/local-store.cc (LocalStore::LocalStore): Create
‘perUserDir’ unconditionally.

Change-Id: I5188320f9630a81d16f79212d0fffabd55d94abe
2025-03-26 17:57:44 +01:00
Ludovic Courtès
ae18b3d9e6
daemon: Allow running as non-root with unprivileged user namespaces.
Many thanks to Reepca Russelstein for their review and guidance on these
changes.

* nix/libstore/build.cc (guestUID, guestGID): New variables.
(DerivationGoal)[readiness]: New field.
(initializeUserNamespace): New function.
(DerivationGoal::runChild): When ‘readiness.readSide’ is positive, read
from it.
(DerivationGoal::startBuilder): Call ‘chown’
only when ‘buildUser.enabled()’ is true.  Pass CLONE_NEWUSER to ‘clone’
when ‘buildUser.enabled()’ is false or not running as root.  Retry
‘clone’ without CLONE_NEWUSER upon EPERM.
(DerivationGoal::registerOutputs): Make ‘actualPath’ writable before
‘rename’.
(DerivationGoal::deleteTmpDir): Catch ‘SysError’ around ‘_chown’ call.
* nix/libstore/local-store.cc (LocalStore::createUser): Do nothing if
‘dirs’ already exists.  Warn instead of failing when failing to chown
‘dir’.
* guix/substitutes.scm (%narinfo-cache-directory): Check for
‘_NIX_OPTIONS’ rather than getuid() == 0 to determine the cache
location.
* doc/guix.texi (Build Environment Setup): Reorganize a bit.  Add
section headings “Daemon Running as Root” and “The Isolated Build
Environment”.  Add “Daemon Running Without Privileges” subsection.
Remove paragraph about ‘--disable-chroot’.
(Invoking guix-daemon): Warn against ‘--disable-chroot’ and explain why.
* tests/derivations.scm ("builder is outside the store"): New test.

Reviewed-by: Reepca Russelstein <reepca@russelstein.xyz>
2025-03-26 17:57:43 +01:00
Ludovic Courtès
40f69b586a
daemon: Remount root directory as read-only.
* nix/libstore/build.cc (DerivationGoal::runChild): Bind-mount the store
and /tmp under ‘chrootRootDir’ to themselves as read-write.
Remount / as read-only.

Change-Id: I79565094c8ec8448401897c720aad75304fd1948
2025-03-26 17:57:43 +01:00
Ludovic Courtès
93474f9288
daemon: Remount inputs as read-only.
* nix/libstore/build.cc (DerivationGoal::runChild): Remount ‘target’ as
read-only.

Reported-by: Reepca Russelstein <reepca@russelstein.xyz>
Change-Id: Ib7201bcf4363be566f205d23d17fe2f55d3ad666
2025-03-26 17:57:43 +01:00
Ludovic Courtès
550ca89744
daemon: Bind-mount all the inputs, not just directories.
* nix/libstore/build.cc (DerivationGoal::startBuilder): Add all of
‘inputPaths’ to ‘dirsInChroot’ instead of hard-linking regular files.
Special-case symlinks.
(DerivationGoal)[regularInputPaths]: Remove.

Reported-by: Reepca Russelstein <reepca@russelstein.xyz>
Change-Id: I070987f92d73f187f7826a975bee9ee309d67f56
2025-03-26 17:57:43 +01:00
Ludovic Courtès
5c0b93b244
daemon: Bind-mount /etc/nsswitch.conf & co. only if it exists.
Those files may be missing in some contexts, for instance within the
build environment.

* nix/libstore/build.cc (DerivationGoal::runChild): Add /etc/resolv.conf
and related files to ‘ss’ only if they exist.

Change-Id: Ie19664a86c8101a1dc82cf39ad4b7abb10f8250a
2025-03-26 17:57:43 +01:00
Reepca Russelstein
78da695178
daemon: Explicitly unlock output path in the has-become-valid case.
Fixes <https://issues.guix.gnu.org/31785>.

Similar to <https://github.com/NixOS/nix/issues/178>, fixed in
<29cde917fe>.

We can't rely on Goal deletion to release our locks in a timely manner.  In
the case in which multiple guix-daemon processes simultaneously try producing
an output path path1, the one that gets there first (P1) will get the lock,
and the second one (P2) will continue trying to acquire the lock until it is
released.  Once it has acquired the lock, it checks to see whether the path
has already become valid in the meantime, and if so it reports success to
those Goals waiting on its completion and finishes.  Unfortunately, it fails
to release the locks it holds first, so those stay held until that Goal gets
deleted.

Suppose we have the following store path dependency graph:

          path4
      /     |     \
   path1   path2   path3

P2 is now sitting on path1's lock, and will continue to do so until path4 is
completed.  Suppose there is also a P3, and it has been blocked while P1
builds path2.  Now P3 is sitting on path2's lock, and can't acquire path1's
lock to determine that it has been completed.  Likewise P2 is sitting on
path1's lock, and now can't acquire path2's lock to determine that it has been
completed.  Finally, P3 completes path3 while P2 is blocked.

Now:

- P1 knows that path1 and path2 are complete, and holds no locks, but can't
  determine that path3 is complete
- P2 knows that path1 and path3 are complete, and holds locks on path1 and
  path3, but can't determine that path2 is complete
- P3 knows that path2 and path3 are complete, and holds a lock on path2, but
  can't determine that path1 is complete

And none of these locks will be released until path4 is complete.  Thus, we
have a deadlock.

To resolve this, we should explicitly release these locks as soon as they
should be released.

* nix/libstore/build.cc
  (DerivationGoal::tryToBuild, SubstitutionGoal::tryToRun):
  Explicitly release locks in the has-become-valid case.
* tests/store-deadlock.scm: New file.
* Makefile.am (SCM_TESTS): Add it.

Change-Id: Ie510f84828892315fe6776c830db33d0f70bcef8
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2024-12-30 00:51:57 +01:00
Brennan Vincent
cf10268a77
daemon: Improve error message in ‘checkStoreName’.
* nix/libstore/store-api.cc (checkStoreName): Clarify message when NAME
starts with a dot.

Change-Id: I045a663bc6cd9844677c65b38a31d3941cf212b5
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2024-11-17 23:15:49 +01:00
Reepca Russelstein
5ab3c4c1e4
daemon: Sanitize successful build outputs prior to exposing them.
There is currently a window of time between when the build outputs are exposed
and when their metadata is canonicalized.

* nix/libstore/build.cc (DerivationGoal::registerOutputs): wait until after
  metadata canonicalization to move successful build outputs to the store.

Change-Id: Ia995136f3f965eaf7b0e1d92af964b816f3fb276
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2024-10-21 00:09:24 +02:00
Reepca Russelstein
558224140d
daemon: Sanitize failed build outputs prior to exposing them.
The only thing keeping a rogue builder and a local user from collaborating to
usurp control over the builder's user during the build is the fact that
whatever files the builder may produce are not accessible to any other users
yet.  If we're going to make them accessible, we should probably do some
sanity checking to ensure that sort of collaborating can't happen.

Currently this isn't happening when failed build outputs are moved from the
chroot as an aid to debugging.

* nix/libstore/build.cc (secureFilePerms): new function.
  (DerivationGoal::buildDone): use it.

Change-Id: I9dce1e3d8813b31cabd87a0e3219bf9830d8be96
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2024-10-21 00:09:10 +02:00
Ludovic Courtès
ff1251de0b
daemon: Address shortcoming in previous security fix for CVE-2024-27297.
This is a followup to 8f4ffb3fae.

Commit 8f4ffb3fae fell short in two
ways: (1) it didn’t have any effet for fixed-output derivations
performed in a chroot, which is the case for all of them except those
using “builtin:download” and “builtin:git-download”, and (2) it did not
preserve ownership when copying, leading to “suspicious ownership or
permission […] rejecting this build output” errors.

* nix/libstore/build.cc (DerivationGoal::buildDone): Account for
‘chrootRootDir’ when copying ‘drv.outputs’.
* nix/libutil/util.cc (copyFileRecursively): Add ‘fchown’ and ‘fchownat’
calls to preserve file ownership; this is necessary for chrooted
fixed-output derivation builds.
* nix/libutil/util.hh: Update comment.

Change-Id: Ib59f040e98fed59d1af81d724b874b592cbef156
2024-03-12 14:07:28 +01:00
Ludovic Courtès
8f4ffb3fae
daemon: Protect against FD escape when building fixed-output derivations (CVE-2024-27297).
This fixes a security issue (CVE-2024-27297) whereby a fixed-output
derivation build process could open a writable file descriptor to its
output, send it to some outside process for instance over an abstract
AF_UNIX socket, which would then allow said process to modify the file
in the store after it has been marked as “valid”.

Vulnerability discovered by puck <https://github.com/puckipedia>.

Nix security advisory:
https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37

Nix fix:
244f3eee0b

* nix/libutil/util.cc (readDirectory): Add variants that take a DIR* and
a file descriptor.  Rewrite the ‘Path’ variant accordingly.
(copyFile, copyFileRecursively): New functions.
* nix/libutil/util.hh (copyFileRecursively): New declaration.
* nix/libstore/build.cc (DerivationGoal::buildDone): When ‘fixedOutput’
is true, call ‘copyFileRecursively’ followed by ‘rename’ on each output.

Change-Id: I7952d41093eed26e123e38c14a4c1424be1ce1c4

Reported-by: Picnoir <picnoir@alternativebit.fr>, Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Change-Id: Idb5f2757f35af86b032a9851cecb19b70227bd88
2024-03-11 22:12:34 +01:00
Ludovic Courtès
d5724111ea
daemon: Change default ‘timeout’ and ‘max-silent-time’ values.
Having a timeout seems generally preferable as it makes sure build slots
are not kept busy for no good reason (few package builds, if any, are
expected to exceed these values).

* nix/libstore/globals.cc (Settings::Settings): Change ‘maxSilentTime’
and ‘buildTimeout’.
* gnu/services/base.scm (<guix-configuration>)[max-silent-time]
[timeout]: Change default values.
* doc/guix.texi (Invoking guix-daemon, Base Services): Adjust
accordingly.

Change-Id: I25c50893f3f7fcca451b8f093d9d4d1a15fa58d8
2024-01-05 17:27:21 +01:00
Tobias Geerinckx-Rice
892ec53b7e
daemon: Fix my own whitespace errors.
This follows up on commit d993ed43b2 and
was entirely predictable.  The noise, sorry for it.

Change-Id: I8ddb8cfe33db3864949f6589cc091616a90ebc5b
2023-12-17 01:00:00 +01:00
Tobias Geerinckx-Rice
d993ed43b2
daemon: Sacrifice builders on OOM.
* nix/libstore/build.cc (DerivationGoal::runChild):
Maximise our OOM score adjustment.

Change-Id: I418c763b499ca16e1ffe3c6033319112b9744f51
2023-12-10 01:00:00 +01:00
Ludovic Courtès
1e47148f46
daemon: Implement ‘substitute-urls’ RPC.
* nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump.
(WorkerOp): Add ‘wopSubstituteURLs’.
* nix/nix-daemon/nix-daemon.cc (performOp): Implement it.
* guix/store.scm (%protocol-version): Bump.
(operation-id): Add ‘substitute-urls’.
(substitute-urls): New procedure.
* tests/store.scm ("substitute-urls, default")
("substitute-urls, client-specified URLs")
("substitute-urls, disabled"): New tests.

Change-Id: I2c0119500c3a1eecfa5ebf32463ffb0f173161de
2023-12-11 23:18:53 +01:00
Ludovic Courtès
d83d4488da
daemon: Simplify “empty status” substitute error message.
* nix/libstore/build.cc (SubstitutionGoal::finished): Don’t show what
the empty status looks like.

Change-Id: Ie898432aeb047aff3d59024de6ed6d18f68903c4
2023-12-04 22:26:36 +01:00
Ludovic Courtès
95f2123135
daemon: Add “git-download” built-in builder.
The new builder makes it possible to break cycles that occurs when the
fixed-output derivation for the source of a dependency of ‘git’ would
itself depend on ‘git’.

* guix/scripts/perform-download.scm (perform-git-download): New
procedure.
(perform-download): Move fixed-output derivation check to…
(guix-perform-download): … here.  Invoke ‘perform-download’ or
‘perform-git-download’ depending on what ‘derivation-builder’ returns.
* nix/libstore/builtins.cc (builtins): Add “git-download”.
* tests/derivations.scm ("built-in-builders"): Update.
("'git-download' built-in builder")
("'git-download' built-in builder, invalid hash")
("'git-download' built-in builder, invalid commit")
("'git-download' built-in builder, not found"): New tests.
2023-09-26 17:36:58 +02:00
Sören Tempel
cd46993275
daemon: Fix build with GCC 13.
The sqlite.hh file uses fixed-width integer types from stdint.h. As
such, it needs to include <cstdint>. Without this include, the file
doesn't compile successfully with GCC13.

See: https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes

* nix/libstore/sqlite.hh: include <cstdint>

Signed-off-by: Sören Tempel <soeren@soeren-tempel.net>
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2023-09-09 18:47:35 +02:00
Ludovic Courtès
2d4d26769d
daemon: Make "opening file" error messages distinguishable.
* nix/libstore/build.cc (DerivationGoal::openLogFile): Customize
"opening file" error message.
* nix/libutil/hash.cc (hashFile): Likewise.
* nix/libutil/util.cc (readFile, writeFile): Likewise.
2022-12-18 01:16:47 +01:00
Ludovic Courtès
eec920ba93
Revert "nix: Guard against removing temporary roots of living processes."
This reverts commit bb0beaecde.  This
change broke a number of tests that assumed the previous behavior.
2022-10-17 09:37:27 +02:00
Ludovic Courtès
bb0beaecde
nix: Guard against removing temporary roots of living processes.
Fixes <https://issues.guix.gnu.org/25018>.

* nix/libstore/gc.cc (readTempRoots): Add a check to guard against removing
the temporary roots of a living process.

Signed-off-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
2022-10-07 16:54:20 -04:00
Maxime Devos
4b32f87e9a
daemon: Remove unused function findOutput.
* nix/libstore/misc.cc (findOutput): Remove it.
* nix/libstore/misc.hh (findOutput): Likewise.

Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
2022-09-11 16:43:30 +02:00
Maxime Devos
93032586df
daemon: Remove unused function exportPaths.
* nix/libstore/store-api.cc (exportPaths): Remove it.
* nix/libstore/store-api.hh (exportPaths): Likewise.

Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
2022-09-11 16:43:30 +02:00
Maxime Devos
1c5f5d6d8c
daemon: Remove unused function openStore.
* nix/libstore/store-api.cc (openStore): Remove it.
* nix/libstore/store-api.hh (openStore): Likewise.

Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
2022-09-11 16:43:30 +02:00
Tobias Geerinckx-Rice
0ace58b99c
daemon: Quote consistently within a string.
* nix/libstore/build.cc (DerivationGoal::registerOutput): ‘’ → `'.
2022-06-05 02:00:00 +02:00
Tobias Geerinckx-Rice
82b06436b4
daemon: Clarify ‘--check’ error when outputs are missing.
Drop the confusing ‘invalid’ jargon and display a hint like we do
for ‘--fallback’.

* nix/libstore/build.cc (DerivationGoal::outputsSubstituted): Rewrite error message.
2022-05-29 02:00:00 +02:00
yarl-baudig@mailoo.org
32916e04f7
daemon: runChild() is forbidden to talk during environment set up
DerivationGoal::startBuilder() is waiting for an empty line as a check that
the environment set up is fine.

Fixes <https://issues.guix.gnu.org/55324>.

* nix/libstore/build.cc (DerivationGoal::runChild): Remove 'debug'
statement corresponding to bind mounts.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2022-05-18 22:29:55 +02:00
Ludovic Courtès
575e52ac2b
daemon: Always default to gzip for log compression.
* nix/libstore/globals.cc (Settings::Settings): Have 'logCompression'
default to COMPRESSION_GZIP unconditionally.
* gnu/services/base.scm (<guix-configuration>)[log-compression]: Default
to 'gzip.
* doc/guix.texi (Invoking guix-daemon, Base Services): Adjust accordingly.
2022-01-18 18:46:51 +01:00
Tobias Geerinckx-Rice
1b1f557d88
daemon: Print the line whence we expect an integer.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Include the malformed substituter stream line in the error message.
2021-11-27 00:49:53 +01:00
Ludovic Courtès
f9b1bb916c
daemon: Read substitute nar size as 'unsigned long long'.
Fixes <https://issues.guix.gnu.org/46212>.
Reported by Christopher Baines <mail@cbaines.net>.

Previously, the nar size returned by 'guix substitute' would be read as
an 'int'; thus, values above 2^31 - 1 would be read and then stored as
negative integers in the database.

Regression introduced in 9dfa20a22a.

* nix/libstore/build.cc (SubstitutionGoal::finished): Use templatized
'string2Int' instead of 'std::atoi' to get an 'unsigned long long',
which is the type of 'hash.second'.
* tests/store.scm ("substitute and large size"): New test.
2021-11-25 00:17:21 +01:00
Ludovic Courtès
1eb40a6dc4
daemon: Read unsigned nar size and download size from substituter.
Fixes <https://issues.guix.gnu.org/51983>.
Reported by Tobias Geerinckx-Rice <me@tobias.gr>.

* nix/libstore/local-store.cc (LocalStore::querySubstitutablePathInfos):
Expect 'unsigned long long' for 'downloadSize' and 'narSize'.
* tests/store.scm ("substitute query and large size"): New test.
2021-11-25 00:17:21 +01:00
Ludovic Courtès
472a0e82a5
daemon: Do not deduplicate files smaller than 8 KiB.
Files smaller than 8 KiB typically represent ~70% of the entries in
/gnu/store/.links but only contribute to ~4% of the space savings
afforded by deduplication.

Not considering these files for deduplication speeds up file insertion
in the store and, more importantly, leaves 'removeUnusedLinks' with
fewer entries to traverse, thereby speeding it up proportionally.

Partly fixes <https://issues.guix.gnu.org/24937>.

* config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK
definition.
* guix/store/deduplication.scm (%deduplication-minimum-size): New
variable.
(deduplicate)[loop]: Do not recurse when FILE's size is below
%DEDUPLICATION-MINIMUM-SIZE.
(dump-port): New procedure.
(dump-file/deduplicate)[hash]: Turn into...
[dump-and-compute-hash]: ... this thunk.
Call 'deduplicate' only when SIZE is greater than
%DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where
st.st_size < deduplicationMinSize.
* nix/libstore/local-store.hh (deduplicationMinSize): New declaration.
* nix/libstore/optimise-store.cc (deduplicationMinSize): New variable.
(LocalStore::optimisePath_): Return when PATH is a symlink or smaller
than 'deduplicationMinSize'.
* tests/derivations.scm ("identical files are deduplicated"): Produce
files bigger than %DEDUPLICATION-MINIMUM-SIZE.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
Likewise.
* tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"):
New test.
("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than
%DEDUPLICATION-MINIMUM-SIZE.
* tests/store.scm ("substitute, deduplication"): Likewise.
2021-11-16 14:34:28 +01:00
Ludovic Courtès
2d73086262
daemon: 'guix substitute' replies on FD 4.
This avoids the situation where error messages would unintentionally go
to stderr and be wrongfully interpreted as a reply by the daemon.

Fixes <https://bugs.gnu.org/46362>.
This is a followup to ee3226e9d5.

* guix/scripts/substitute.scm (display-narinfo-data): Add 'port'
parameter and honor it.
(process-query): Likewise.
(process-substitution): Likewise.
(%error-to-file-descriptor-4?, with-redirected-error-port): Remove.
(%reply-file-descriptor): New variable.
(guix-substitute): Remove use of 'with-redirected-error-port'.  Define
'reply-port' and pass it to 'process-query' and 'process-substitution'.
* nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap
'builderOut' and 'fromAgent'.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Likewise.
* tests/substitute.scm <top level>: Set '%reply-file-descriptor'
rather than '%error-to-file-descriptor-4?'.
2021-04-09 17:46:38 +02:00
Ludovic Courtès
ec7fb66994
daemon: Prevent privilege escalation with '--keep-failed' [security].
Fixes <https://bugs.gnu.org/47229>.
Reported by Nathan Nye of WhiteBeam Security.

* nix/libstore/build.cc (DerivationGoal::startBuilder): When 'useChroot'
is true, add "/top" to 'tmpDir'.
(DerivationGoal::deleteTmpDir): Adjust accordingly.  When
'settings.keepFailed' is true, chown in two steps: first the "/top"
sub-directory, and then rename "/top" to its parent.
2021-03-18 12:18:56 +01:00
Ludovic Courtès
c7c7f068c1
daemon: Delegate deduplication to 'guix substitute'.
This removes the main source of latency between subsequent downloads.

* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a
"deduplicate" key to ENV.
(SubstitutionGoal::finished): Remove call to 'optimisePath'.
* guix/scripts/substitute.scm (process-substitution)[destination-in-store?]
[dump-file/deduplicate*]: New variables.
Pass #:dump-file to 'restore-file'.
* guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New
variable.
Pass #:deduplicate? to 'process-substitution'.
* guix/serialization.scm (dump-file): Export and augment 'dump-file'.
2020-12-19 23:25:01 +01:00
Ludovic Courtès
15cf28fbb4
daemon: Do not reset timestamps and permissions on substituted items.
'guix substitute' now takes care of it via 'restore-file'.

* nix/libstore/build.cc (SubstitutionGoal::finished): Remove call to
'canonicalisePathMetaData'.
2020-12-19 23:25:01 +01:00
Ludovic Courtès
9dfa20a22a
daemon: Let 'guix substitute' perform hash checks.
This way, the hash of the store item can be computed as it is restored,
thereby avoiding an additional file tree traversal ('hashPath' call)
later on in the daemon.  Consequently, it should reduce latency between
subsequent substitute downloads.

This is a followup to 5ff521452b.

* guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New
procedure.
(process-substitution): Wrap INPUT into a hash input port, 'hashed', and
read from it.  Compare the actual and expected hashes, and print a
"hash-mismatch" status line when they differ.  When they match, print
not just "success" but also the nar hash and size.
* nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]:
Remove.
(SubstitutionGoal::finished): Tokenize 'status'.  Parse it and handle
"success" and "hash-mismatch" accordingly.  Call 'hashPath' only when
the returned hash is not SHA256.
(SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr'
handling.
* tests/substitute.scm ("substitute, invalid hash"): Rename to...
("substitute, invalid narinfo hash"): ... this.
("substitute, invalid hash"): New test.
2020-12-19 23:25:00 +01:00
Ludovic Courtès
bfe4cdf88e
daemon: Raise an error if substituter doesn't send the expected hash.
It was already impossible in practice for 'expectedHashStr' to be empty
if 'status' == "success".

* nix/libstore/build.cc (SubstitutionGoal::finished): Throw 'SubstError'
when 'expectedHashStr' is empty.
2020-12-08 22:30:08 +01:00
Ludovic Courtès
5ff521452b
substitute: Cache and reuse connections while substituting.
That way, when fetching a series of substitutes from the same server(s),
the connection is reused instead of being closed/opened for each
substitutes, which saves on network round trips and TLS handshakes.

* guix/http-client.scm (http-fetch): Add #:keep-alive? and honor it.
* guix/progress.scm (progress-report-port): Add #:close? parameter and
honor it.
* guix/scripts/substitute.scm (at-most): Return the tail as a second
value.
(fetch): Add #:port and #:keep-alive? and honor them.
(%max-cached-connections): New variable.
(open-connection-for-uri/cached, call-with-cached-connection): New
procedures.
(with-cached-connection): New macro.
(process-substitution): Wrap 'fetch' call in 'with-cached-connection'.
Pass #:close? to 'progress-report-port'.
2020-12-08 22:30:08 +01:00
Ludovic Courtès
711df9ef3c
daemon: Run 'guix substitute --substitute' as an agent.
This avoids spawning one substitute process per substitution.

* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'.  Run "guix substitute --substitute" as an 'Agent'.  Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments.  Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
2020-12-08 22:30:08 +01:00
Ludovic Courtès
a618a8c620
daemon: Factorize substituter agent spawning.
* nix/libstore/local-store.hh (class LocalStore)[substituter]: New
method.
[runningSubstituter]: Turn into a shared_ptr.
* nix/libstore/local-store.cc (LocalStore::querySubstitutablePaths):
Call 'substituter' instead of using inline code.
(LocalStore::querySubstitutablePathInfos): Likewise.
(LocalStore::substituter): New method.
2020-12-08 22:30:08 +01:00
Ludovic Courtès
79c6614f58
daemon: Use 'Agent' to spawn 'guix substitute --query'.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed.  Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'.  Set 'guix-warning-port'
too.  No longer exit when 'substitute-urls' returns the empty list.  No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
2020-12-08 22:30:08 +01:00
Ludovic Courtès
56fc14577e
daemon: Remove unneeded forward declaration.
This is a followup to ee9dff34f9.

* nix/libstore/build.cc: Remove 'struct Agent' forward declaration.
2020-12-01 00:10:48 +01:00