diff --git a/third_party/nixpkgs/doc/builders/fetchers.chapter.md b/third_party/nixpkgs/doc/builders/fetchers.chapter.md
index 30d0653448..b1b00106b6 100644
--- a/third_party/nixpkgs/doc/builders/fetchers.chapter.md
+++ b/third_party/nixpkgs/doc/builders/fetchers.chapter.md
@@ -1,8 +1,16 @@
# Fetchers {#chap-pkgs-fetchers}
-When using Nix, you will frequently need to download source code and other files from the internet. Nixpkgs comes with a few helper functions that allow you to fetch fixed-output derivations in a structured way.
+When using Nix, you will frequently need to download source code and other files from the internet. For this purpose, Nix provides the [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) feature and Nixpkgs provides various functions that implement the actual fetching from various protocols and services.
-The two fetcher primitives are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `sha256`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `sha256`. This hash will be used by Nix to identify your source. A typical usage of fetchurl is provided below.
+## Caveats
+
+Because fixed output derivations are _identified_ by their hash, a common mistake is to update a fetcher's URL or a version parameter, without updating the hash. **This will cause the old contents to be used.** So remember to always invalidate the hash argument.
+
+For those who develop and maintain fetcheres, a similar problem arises with changes to the implementation of a fetcher. These may cause a fixed output derivation to fail, but won't normally be caught by tests because the supposed output is already in the store or cache. For the purpose of testing, you can use a trick that is embodied by the [`invalidateFetcherByDrvHash`](#sec-pkgs-invalidateFetcherByDrvHash) function. It uses the derivation `name` to create a unique output path per fetcher implementation, defeating the caching precisely where it would be harmful.
+
+## `fetchurl` and `fetchzip` {#fetchurl}
+
+Two basic fetchers are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `sha256`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `sha256`. This hash will be used by Nix to identify your source. A typical usage of fetchurl is provided below.
```nix
{ stdenv, fetchurl }:
@@ -20,7 +28,7 @@ The main difference between `fetchurl` and `fetchzip` is in how they store the c
`fetchpatch` works very similarly to `fetchurl` with the same arguments expected. It expects patch files as a source and performs normalization on them before computing the checksum. For example it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
-Other fetcher functions allow you to add source code directly from a VCS such as subversion or git. These are mostly straightforward nambes based on the name of the command used with the VCS system. Because they give you a working repository, they act most like `fetchzip`.
+Most other fetchers return a directory rather than a single file.
## `fetchsvn` {#fetchsvn}
diff --git a/third_party/nixpkgs/doc/builders/special.xml b/third_party/nixpkgs/doc/builders/special.xml
index 8902ce5c81..2f84599cdd 100644
--- a/third_party/nixpkgs/doc/builders/special.xml
+++ b/third_party/nixpkgs/doc/builders/special.xml
@@ -7,4 +7,5 @@
+
diff --git a/third_party/nixpkgs/doc/builders/special/invalidateFetcherByDrvHash.section.md b/third_party/nixpkgs/doc/builders/special/invalidateFetcherByDrvHash.section.md
new file mode 100644
index 0000000000..7c2f03a64b
--- /dev/null
+++ b/third_party/nixpkgs/doc/builders/special/invalidateFetcherByDrvHash.section.md
@@ -0,0 +1,31 @@
+
+## `invalidateFetcherByDrvHash` {#sec-pkgs-invalidateFetcherByDrvHash}
+
+Use the derivation hash to invalidate the output via name, for testing.
+
+Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
+
+Normally, fixed output derivations can and should be cached by their output
+hash only, but for testing we want to re-fetch everytime the fetcher changes.
+
+Changes to the fetcher become apparent in the drvPath, which is a hash of
+how to fetch, rather than a fixed store path.
+By inserting this hash into the name, we can make sure to re-run the fetcher
+every time the fetcher changes.
+
+This relies on the assumption that Nix isn't clever enough to reuse its
+database of local store contents to optimize fetching.
+
+You might notice that the "salted" name derives from the normal invocation,
+not the final derivation. `invalidateFetcherByDrvHash` has to invoke the fetcher
+function twice: once to get a derivation hash, and again to produce the final
+fixed output derivation.
+
+Example:
+
+ tests.fetchgit = invalidateFetcherByDrvHash fetchgit {
+ name = "nix-source";
+ url = "https://github.com/NixOS/nix";
+ rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
+ sha256 = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
+ };
diff --git a/third_party/nixpkgs/doc/languages-frameworks/coq.section.md b/third_party/nixpkgs/doc/languages-frameworks/coq.section.md
index 39b60d83ac..a5155aedaf 100644
--- a/third_party/nixpkgs/doc/languages-frameworks/coq.section.md
+++ b/third_party/nixpkgs/doc/languages-frameworks/coq.section.md
@@ -28,12 +28,12 @@ The recommended way of defining a derivation for a Coq library, is to use the `c
* `domain` (optional, defaults to `"github.com"`), domains including the strings `"github"` or `"gitlab"` in their names are automatically supported, otherwise, one must change the `fetcher` argument to support them (cf `pkgs/development/coq-modules/heq/default.nix` for an example),
* `releaseRev` (optional, defaults to `(v: v)`), provides a default mapping from release names to revision hashes/branch names/tags,
* `displayVersion` (optional), provides a way to alter the computation of `name` from `pname`, by explaining how to display version numbers,
-* `namePrefix` (optional), provides a way to alter the computation of `name` from `pname`, by explaining which dependencies must occur in `name`,
+* `namePrefix` (optional, defaults to `[ "coq" ]`), provides a way to alter the computation of `name` from `pname`, by explaining which dependencies must occur in `name`,
* `extraBuildInputs` (optional), by default `buildInputs` just contains `coq`, this allows to add more build inputs,
* `mlPlugin` (optional, defaults to `false`). Some extensions (plugins) might require OCaml and sometimes other OCaml packages. Standard dependencies can be added by setting the current option to `true`. For a finer grain control, the `coq.ocamlPackages` attribute can be used in `extraBuildInputs` to depend on the same package set Coq was built against.
* `useDune2ifVersion` (optional, default to `(x: false)` uses Dune2 to build the package if the provided predicate evaluates to true on the version, e.g. `useDune2if = versions.isGe "1.1"` will use dune if the version of the package is greater or equal to `"1.1"`,
* `useDune2` (optional, defaults to `false`) uses Dune2 to build the package if set to true, the presence of this attribute overrides the behavior of the previous one.
-* `opam-name` (optional, defaults to `coq-` followed by the value of `pname`), name of the Dune package to build.
+* `opam-name` (optional, defaults to concatenating with a dash separator the components of `namePrefix` and `pname`), name of the Dune package to build.
* `enableParallelBuilding` (optional, defaults to `true`), since it is activated by default, we provide a way to disable it.
* `extraInstallFlags` (optional), allows to extend `installFlags` which initializes the variable `COQMF_COQLIB` so as to install in the proper subdirectory. Indeed Coq libraries should be installed in `$(out)/lib/coq/${coq.coq-version}/user-contrib/`. Such directories are automatically added to the `$COQPATH` environment variable by the hook defined in the Coq derivation.
* `setCOQBIN` (optional, defaults to `true`), by default, the environment variable `$COQBIN` is set to the current Coq's binary, but one can disable this behavior by setting it to `false`,
diff --git a/third_party/nixpkgs/doc/languages-frameworks/index.xml b/third_party/nixpkgs/doc/languages-frameworks/index.xml
index 49cdf94a44..29a1e08969 100644
--- a/third_party/nixpkgs/doc/languages-frameworks/index.xml
+++ b/third_party/nixpkgs/doc/languages-frameworks/index.xml
@@ -24,6 +24,7 @@
+
diff --git a/third_party/nixpkgs/doc/languages-frameworks/octave.section.md b/third_party/nixpkgs/doc/languages-frameworks/octave.section.md
new file mode 100644
index 0000000000..ff872f4a75
--- /dev/null
+++ b/third_party/nixpkgs/doc/languages-frameworks/octave.section.md
@@ -0,0 +1,100 @@
+# Octave {#sec-octave}
+
+## Introduction {#ssec-octave-introduction}
+
+Octave is a modular scientific programming language and environment.
+A majority of the packages supported by Octave from their [website](https://octave.sourceforge.io/packages.php) are packaged in nixpkgs.
+
+## Structure {#ssec-octave-structure}
+
+All Octave add-on packages are available in two ways:
+1. Under the top-level `Octave` attribute, `octave.pkgs`.
+2. As a top-level attribute, `octavePackages`.
+
+## Packaging Octave Packages {#ssec-octave-packaging}
+
+Nixpkgs provides a function `buildOctavePackage`, a generic package builder function for any Octave package that complies with the Octave's current packaging format.
+
+All Octave packages are defined in [pkgs/top-level/octave-packages.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/octave-packages.nix) rather than `pkgs/all-packages.nix`.
+Each package is defined in their own file in the [pkgs/development/octave-modules](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/octave-modules) directory.
+Octave packages are made available through `all-packages.nix` through both the attribute `octavePackages` and `octave.pkgs`.
+You can test building an Octave package as follows:
+
+```ShellSession
+$ nix-build -A octavePackages.symbolic
+```
+
+When building Octave packages with `nix-build`, the `buildOctavePackage` function adds `octave-octaveVersion` to; the start of the package's name attribute.
+
+This can be required when installing the package using `nix-env`:
+
+```ShellSession
+$ nix-env -i octave-6.2.0-symbolic
+```
+
+Although, you can also install it using the attribute name:
+
+```ShellSession
+$ nix-env -i -A octavePackages.symbolic
+```
+
+You can build Octave with packages by using the `withPackages` passed-through function.
+
+```ShellSession
+$ nix-shell -p 'octave.withPackages (ps: with ps; [ symbolic ])'
+```
+
+This will also work in a `shell.nix` file.
+
+```nix
+{ pkgs ? import { }}:
+
+pkgs.mkShell {
+ nativeBuildInputs = with pkgs; [
+ (octave.withPackages (opkgs: with opkgs; [ symbolic ]))
+ ];
+}
+```
+
+### `buildOctavePackage` Steps {#sssec-buildOctavePackage-steps}
+
+The `buildOctavePackage` does several things to make sure things work properly.
+
+1. Sets the environment variable `OCTAVE_HISTFILE` to `/dev/null` during package compilation so that the commands run through the Octave interpreter directly are not logged.
+2. Skips the configuration step, because the packages are stored as gzipped tarballs, which Octave itself handles directly.
+3. Change the hierarchy of the tarball so that only a single directory is at the top-most level of the tarball.
+4. Use Octave itself to run the `pkg build` command, which unzips the tarball, extracts the necessary files written in Octave, and compiles any code written in C++ or Fortran, and places the fully compiled artifact in `$out`.
+
+`buildOctavePackage` is built on top of `stdenv` in a standard way, allowing most things to be customized.
+
+### Handling Dependencies {#sssec-octave-handling-dependencies}
+
+In Octave packages, there are four sets of dependencies that can be specified:
+
+`nativeBuildInputs`
+: Just like other packages, `nativeBuildInputs` is intended for architecture-dependent build-time-only dependencies.
+
+`buildInputs`
+: Like other packages, `buildInputs` is intended for architecture-independent build-time-only dependencies.
+
+`propagatedBuildInputs`
+: Similar to other packages, `propagatedBuildInputs` is intended for packages that are required for both building and running of the package.
+See [Symbolic](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/octave-modules/symbolic/default.nix) for how this works and why it is needed.
+
+`requiredOctavePackages`
+: This is a special dependency that ensures the specified Octave packages are dependent on others, and are made available simultaneously when loading them in Octave.
+
+### Installing Octave Packages {#sssec-installing-octave-packages}
+
+By default, the `buildOctavePackage` function does _not_ install the requested package into Octave for use.
+The function will only build the requested package.
+This is due to Octave maintaining an text-based database about which packages are installed where.
+To this end, when all the requested packages have been built, the Octave package and all its add-on packages are put together into an environment, similar to Python.
+
+1. First, all the Octave binaries are wrapped with the environment variable `OCTAVE_SITE_INITFILE` set to a file in `$out`, which is required for Octave to be able to find the non-standard package database location.
+2. Because of the way `buildEnv` works, all tarballs that are present (which should be all Octave packages to install) should be removed.
+3. The path down to the default install location of Octave packages is recreated so that Nix-operated Octave can install the packages.
+4. Install the packages into the `$out` environment while writing package entries to the database file.
+This database file is unique for each different (according to Nix) environment invocation.
+5. Rewrite the Octave-wide startup file to read from the list of packages installed in that particular environment.
+6. Wrap any programs that are required by the Octave packages so that they work with all the paths defined within the environment.
diff --git a/third_party/nixpkgs/doc/languages-frameworks/rust.section.md b/third_party/nixpkgs/doc/languages-frameworks/rust.section.md
index 9de9a06465..b2f045b11b 100644
--- a/third_party/nixpkgs/doc/languages-frameworks/rust.section.md
+++ b/third_party/nixpkgs/doc/languages-frameworks/rust.section.md
@@ -20,7 +20,7 @@ or use Mozilla's [Rust nightlies overlay](#using-the-rust-nightlies-overlay).
Rust applications are packaged by using the `buildRustPackage` helper from `rustPlatform`:
```nix
-{ lib, rustPlatform }:
+{ lib, fetchFromGitHub, rustPlatform }:
rustPlatform.buildRustPackage rec {
pname = "ripgrep";
@@ -116,22 +116,44 @@ is updated after every change to `Cargo.lock`. Therefore,
a `Cargo.lock` file using the `cargoLock` argument. For example:
```nix
-rustPlatform.buildRustPackage rec {
+rustPlatform.buildRustPackage {
pname = "myproject";
version = "1.0.0";
cargoLock = {
lockFile = ./Cargo.lock;
- }
+ };
# ...
}
```
This will retrieve the dependencies using fixed-output derivations from
-the specified lockfile. Note that setting `cargoLock.lockFile` doesn't
-add a `Cargo.lock` to your `src`, and a `Cargo.lock` is still required
-to build a rust package. A simple fix is to use:
+the specified lockfile.
+
+One caveat is that `Cargo.lock` cannot be patched in the `patchPhase`
+because it runs after the dependencies have already been fetched. If
+you need to patch or generate the lockfile you can alternatively set
+`cargoLock.lockFileContents` to a string of its contents:
+
+```nix
+rustPlatform.buildRustPackage {
+ pname = "myproject";
+ version = "1.0.0";
+
+ cargoLock = let
+ fixupLockFile = path: f (builtins.readFile path);
+ in {
+ lockFileContents = fixupLockFile ./Cargo.lock;
+ };
+
+ # ...
+}
+```
+
+Note that setting `cargoLock.lockFile` or `cargoLock.lockFileContents`
+doesn't add a `Cargo.lock` to your `src`, and a `Cargo.lock` is still
+required to build a rust package. A simple fix is to use:
```nix
postPatch = ''
diff --git a/third_party/nixpkgs/doc/stdenv/multiple-output.chapter.md b/third_party/nixpkgs/doc/stdenv/multiple-output.chapter.md
index d04f83302a..62bf543e51 100644
--- a/third_party/nixpkgs/doc/stdenv/multiple-output.chapter.md
+++ b/third_party/nixpkgs/doc/stdenv/multiple-output.chapter.md
@@ -79,7 +79,7 @@ A commonly adopted convention in `nixpkgs` is that executables provided by the p
The `glibc` package is a deliberate single exception to the “binaries first” convention. The `glibc` has `libs` as its first output allowing the libraries provided by `glibc` to be referenced directly (e.g. `${stdenv.glibc}/lib/ld-linux-x86-64.so.2`). The executables provided by `glibc` can be accessed via its `bin` attribute (e.g. `${stdenv.glibc.bin}/bin/ldd`).
-The reason for why `glibc` deviates from the convention is because referencing a library provided by `glibc` is a very common operation among Nix packages. For instance, third-party executables packaged by Nix are typically patched and relinked with the relevant version of `glibc` libraries from Nix packages (please see the documentation on [patchelf](https://github.com/NixOS/patchelf/blob/master/README) for more details).
+The reason for why `glibc` deviates from the convention is because referencing a library provided by `glibc` is a very common operation among Nix packages. For instance, third-party executables packaged by Nix are typically patched and relinked with the relevant version of `glibc` libraries from Nix packages (please see the documentation on [patchelf](https://github.com/NixOS/patchelf) for more details).
### File type groups {#multiple-output-file-type-groups}
diff --git a/third_party/nixpkgs/maintainers/maintainer-list.nix b/third_party/nixpkgs/maintainers/maintainer-list.nix
index be13f14bc5..393be185b0 100644
--- a/third_party/nixpkgs/maintainers/maintainer-list.nix
+++ b/third_party/nixpkgs/maintainers/maintainer-list.nix
@@ -1853,6 +1853,12 @@
githubId = 1762540;
name = "Changlin Li";
};
+ chanley = {
+ email = "charlieshanley@gmail.com";
+ github = "charlieshanley";
+ githubId = 8228888;
+ name = "Charlie Hanley";
+ };
CharlesHD = {
email = "charleshdespointes@gmail.com";
github = "CharlesHD";
@@ -4441,6 +4447,12 @@
fingerprint = "D618 7A03 A40A 3D56 62F5 4B46 03EF BF83 9A5F DC15";
}];
};
+ hleboulanger = {
+ email = "hleboulanger@protonmail.com";
+ name = "Harold Leboulanger";
+ github = "thbkrhsw";
+ githubId = 33122;
+ };
hlolli = {
email = "hlolli@gmail.com";
github = "hlolli";
@@ -4565,6 +4577,16 @@
githubId = 2789926;
name = "Imran Hossain";
};
+ iagoq = {
+ email = "18238046+iagocq@users.noreply.github.com";
+ github = "iagocq";
+ githubId = 18238046;
+ name = "Iago Manoel Brito";
+ keys = [{
+ longkeyid = "rsa4096/0x35D39F9A9A1BC8DA";
+ fingerprint = "DF90 9D58 BEE4 E73A 1B8C 5AF3 35D3 9F9A 9A1B C8DA";
+ }];
+ };
iammrinal0 = {
email = "nixpkgs@mrinalpurohit.in";
github = "iammrinal0";
@@ -9182,6 +9204,12 @@
githubId = 546296;
name = "Eric Ren";
};
+ renesat = {
+ name = "Ivan Smolyakov";
+ email = "smol.ivan97@gmail.com";
+ github = "renesat";
+ githubId = 11363539;
+ };
renzo = {
email = "renzocarbonara@gmail.com";
github = "k0001";
@@ -9846,12 +9874,6 @@
githubId = 11613056;
name = "Scott Dier";
};
- sdll = {
- email = "sasha.delly@gmail.com";
- github = "sdll";
- githubId = 17913919;
- name = "Sasha Illarionov";
- };
SeanZicari = {
email = "sean.zicari@gmail.com";
github = "SeanZicari";
diff --git a/third_party/nixpkgs/maintainers/scripts/haskell/dependencies.nix b/third_party/nixpkgs/maintainers/scripts/haskell/dependencies.nix
new file mode 100644
index 0000000000..f0620902c0
--- /dev/null
+++ b/third_party/nixpkgs/maintainers/scripts/haskell/dependencies.nix
@@ -0,0 +1,10 @@
+# Nix script to calculate the Haskell dependencies of every haskellPackage. Used by ./hydra-report.hs.
+let
+ pkgs = import ../../.. {};
+ inherit (pkgs) lib;
+ getDeps = _: pkg: {
+ deps = builtins.filter (x: !isNull x) (map (x: x.pname or null) (pkg.propagatedBuildInputs or []));
+ broken = (pkg.meta.hydraPlatforms or [null]) == [];
+ };
+in
+ lib.mapAttrs getDeps pkgs.haskellPackages
diff --git a/third_party/nixpkgs/maintainers/scripts/haskell/hydra-report.hs b/third_party/nixpkgs/maintainers/scripts/haskell/hydra-report.hs
index fd6430d43c..0d22a67036 100755
--- a/third_party/nixpkgs/maintainers/scripts/haskell/hydra-report.hs
+++ b/third_party/nixpkgs/maintainers/scripts/haskell/hydra-report.hs
@@ -26,6 +26,8 @@ Because step 1) is quite expensive and takes roughly ~5 minutes the result is ca
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TupleSections #-}
{-# OPTIONS_GHC -Wall #-}
+{-# LANGUAGE ViewPatterns #-}
+{-# LANGUAGE TupleSections #-}
import Control.Monad (forM_, (<=<))
import Control.Monad.Trans (MonadIO (liftIO))
@@ -41,7 +43,7 @@ import Data.List.NonEmpty (NonEmpty, nonEmpty)
import qualified Data.List.NonEmpty as NonEmpty
import Data.Map.Strict (Map)
import qualified Data.Map.Strict as Map
-import Data.Maybe (fromMaybe, mapMaybe)
+import Data.Maybe (fromMaybe, mapMaybe, isNothing)
import Data.Monoid (Sum (Sum, getSum))
import Data.Sequence (Seq)
import qualified Data.Sequence as Seq
@@ -70,6 +72,12 @@ import System.Directory (XdgDirectory (XdgCache), getXdgDirectory)
import System.Environment (getArgs)
import System.Process (readProcess)
import Prelude hiding (id)
+import Data.List (sortOn)
+import Control.Concurrent.Async (concurrently)
+import Control.Exception (evaluate)
+import qualified Data.IntMap.Strict as IntMap
+import qualified Data.IntSet as IntSet
+import Data.Bifunctor (second)
newtype JobsetEvals = JobsetEvals
{ evals :: Seq Eval
@@ -134,20 +142,17 @@ hydraEvalCommand = "hydra-eval-jobs"
hydraEvalParams :: [String]
hydraEvalParams = ["-I", ".", "pkgs/top-level/release-haskell.nix"]
-handlesCommand :: FilePath
-handlesCommand = "nix-instantiate"
+nixExprCommand :: FilePath
+nixExprCommand = "nix-instantiate"
-handlesParams :: [String]
-handlesParams = ["--eval", "--strict", "--json", "-"]
-
-handlesExpression :: String
-handlesExpression = "with import ./. {}; with lib; zipAttrsWith (_: builtins.head) (mapAttrsToList (_: v: if v ? github then { \"${v.email}\" = v.github; } else {}) (import maintainers/maintainer-list.nix))"
+nixExprParams :: [String]
+nixExprParams = ["--eval", "--strict", "--json"]
-- | This newtype is used to parse a Hydra job output from @hydra-eval-jobs@.
-- The only field we are interested in is @maintainers@, which is why this
-- is just a newtype.
--
--- Note that there are occassionally jobs that don't have a maintainers
+-- Note that there are occasionally jobs that don't have a maintainers
-- field, which is why this has to be @Maybe Text@.
newtype Maintainers = Maintainers { maintainers :: Maybe Text }
deriving stock (Generic, Show)
@@ -195,13 +200,49 @@ type EmailToGitHubHandles = Map Text Text
-- @@
type MaintainerMap = Map Text (NonEmpty Text)
--- | Generate a mapping of Hydra job names to maintainer GitHub handles.
+-- | Information about a package which lists its dependencies and whether the
+-- package is marked broken.
+data DepInfo = DepInfo {
+ deps :: Set Text,
+ broken :: Bool
+}
+ deriving stock (Generic, Show)
+ deriving anyclass (FromJSON, ToJSON)
+
+-- | Map from package names to their DepInfo. This is the data we get out of a
+-- nix call.
+type DependencyMap = Map Text DepInfo
+
+-- | Map from package names to its broken state, number of reverse dependencies (fst) and
+-- unbroken reverse dependencies (snd).
+type ReverseDependencyMap = Map Text (Int, Int)
+
+-- | Calculate the (unbroken) reverse dependencies of a package by transitively
+-- going through all packages if it’s a dependency of them.
+calculateReverseDependencies :: DependencyMap -> ReverseDependencyMap
+calculateReverseDependencies depMap = Map.fromDistinctAscList $ zip keys (zip (rdepMap False) (rdepMap True))
+ where
+ -- This code tries to efficiently invert the dependency map and calculate
+ -- it’s transitive closure by internally identifying every pkg with it’s index
+ -- in the package list and then using memoization.
+ keys = Map.keys depMap
+ pkgToIndexMap = Map.fromDistinctAscList (zip keys [0..])
+ intDeps = zip [0..] $ (\DepInfo{broken,deps} -> (broken,mapMaybe (`Map.lookup` pkgToIndexMap) $ Set.toList deps)) <$> Map.elems depMap
+ rdepMap onlyUnbroken = IntSet.size <$> resultList
+ where
+ resultList = go <$> [0..]
+ oneStepMap = IntMap.fromListWith IntSet.union $ (\(key,(_,deps)) -> (,IntSet.singleton key) <$> deps) <=< filter (\(_, (broken,_)) -> not (broken && onlyUnbroken)) $ intDeps
+ go pkg = IntSet.unions (oneStep:((resultList !!) <$> IntSet.toList oneStep))
+ where oneStep = IntMap.findWithDefault mempty pkg oneStepMap
+
+-- | Generate a mapping of Hydra job names to maintainer GitHub handles. Calls
+-- hydra-eval-jobs and the nix script ./maintainer-handles.nix.
getMaintainerMap :: IO MaintainerMap
getMaintainerMap = do
hydraJobs :: HydraJobs <-
- readJSONProcess hydraEvalCommand hydraEvalParams "" "Failed to decode hydra-eval-jobs output: "
+ readJSONProcess hydraEvalCommand hydraEvalParams "Failed to decode hydra-eval-jobs output: "
handlesMap :: EmailToGitHubHandles <-
- readJSONProcess handlesCommand handlesParams handlesExpression "Failed to decode nix output for lookup of github handles: "
+ readJSONProcess nixExprCommand ("maintainers/scripts/haskell/maintainer-handles.nix":nixExprParams) "Failed to decode nix output for lookup of github handles: "
pure $ Map.mapMaybe (splitMaintainersToGitHubHandles handlesMap) hydraJobs
where
-- Split a comma-spearated string of Maintainers into a NonEmpty list of
@@ -211,6 +252,12 @@ getMaintainerMap = do
splitMaintainersToGitHubHandles handlesMap (Maintainers maint) =
nonEmpty . mapMaybe (`Map.lookup` handlesMap) . Text.splitOn ", " $ fromMaybe "" maint
+-- | Get the a map of all dependencies of every package by calling the nix
+-- script ./dependencies.nix.
+getDependencyMap :: IO DependencyMap
+getDependencyMap =
+ readJSONProcess nixExprCommand ("maintainers/scripts/haskell/dependencies.nix":nixExprParams) "Failed to decode nix output for lookup of dependencies: "
+
-- | Run a process that produces JSON on stdout and and decode the JSON to a
-- data type.
--
@@ -219,11 +266,10 @@ readJSONProcess
:: FromJSON a
=> FilePath -- ^ Filename of executable.
-> [String] -- ^ Arguments
- -> String -- ^ stdin to pass to the process
-> String -- ^ String to prefix to JSON-decode error.
-> IO a
-readJSONProcess exe args input err = do
- output <- readProcess exe args input
+readJSONProcess exe args err = do
+ output <- readProcess exe args ""
let eitherDecodedOutput = eitherDecodeStrict' . encodeUtf8 . Text.pack $ output
case eitherDecodedOutput of
Left decodeErr -> error $ err <> decodeErr <> "\nRaw: '" <> take 1000 output <> "'"
@@ -264,7 +310,13 @@ platformIcon (Platform x) = case x of
data BuildResult = BuildResult {state :: BuildState, id :: Int} deriving (Show, Eq, Ord)
newtype Platform = Platform {platform :: Text} deriving (Show, Eq, Ord)
newtype Table row col a = Table (Map (row, col) a)
-type StatusSummary = Map Text (Table Text Platform BuildResult, Set Text)
+data SummaryEntry = SummaryEntry {
+ summaryBuilds :: Table Text Platform BuildResult,
+ summaryMaintainers :: Set Text,
+ summaryReverseDeps :: Int,
+ summaryUnbrokenReverseDeps :: Int
+}
+type StatusSummary = Map Text SummaryEntry
instance (Ord row, Ord col, Semigroup a) => Semigroup (Table row col a) where
Table l <> Table r = Table (Map.unionWith (<>) l r)
@@ -275,11 +327,11 @@ instance Functor (Table row col) where
instance Foldable (Table row col) where
foldMap f (Table a) = foldMap f a
-buildSummary :: MaintainerMap -> Seq Build -> StatusSummary
-buildSummary maintainerMap = foldl (Map.unionWith unionSummary) Map.empty . fmap toSummary
+buildSummary :: MaintainerMap -> ReverseDependencyMap -> Seq Build -> StatusSummary
+buildSummary maintainerMap reverseDependencyMap = foldl (Map.unionWith unionSummary) Map.empty . fmap toSummary
where
- unionSummary (Table l, l') (Table r, r') = (Table $ Map.union l r, l' <> r')
- toSummary Build{finished, buildstatus, job, id, system} = Map.singleton name (Table (Map.singleton (set, Platform system) (BuildResult state id)), maintainers)
+ unionSummary (SummaryEntry (Table lb) lm lr lu) (SummaryEntry (Table rb) rm rr ru) = SummaryEntry (Table $ Map.union lb rb) (lm <> rm) (max lr rr) (max lu ru)
+ toSummary Build{finished, buildstatus, job, id, system} = Map.singleton name (SummaryEntry (Table (Map.singleton (set, Platform system) (BuildResult state id))) maintainers reverseDeps unbrokenReverseDeps)
where
state :: BuildState
state = case (finished, buildstatus) of
@@ -297,6 +349,7 @@ buildSummary maintainerMap = foldl (Map.unionWith unionSummary) Map.empty . fmap
name = maybe packageName NonEmpty.last splitted
set = maybe "" (Text.intercalate "." . NonEmpty.init) splitted
maintainers = maybe mempty (Set.fromList . toList) (Map.lookup job maintainerMap)
+ (reverseDeps, unbrokenReverseDeps) = Map.findWithDefault (0,0) name reverseDependencyMap
readBuildReports :: IO (Eval, UTCTime, Seq Build)
readBuildReports = do
@@ -339,25 +392,29 @@ makeSearchLink evalId linkLabel query = "[" <> linkLabel <> "](" <> "https://hyd
statusToNumSummary :: StatusSummary -> NumSummary
statusToNumSummary = fmap getSum . foldMap (fmap Sum . jobTotals)
-jobTotals :: (Table Text Platform BuildResult, a) -> Table Platform BuildState Int
-jobTotals (Table mapping, _) = getSum <$> Table (Map.foldMapWithKey (\(_, platform) (BuildResult buildstate _) -> Map.singleton (platform, buildstate) (Sum 1)) mapping)
+jobTotals :: SummaryEntry -> Table Platform BuildState Int
+jobTotals (summaryBuilds -> Table mapping) = getSum <$> Table (Map.foldMapWithKey (\(_, platform) (BuildResult buildstate _) -> Map.singleton (platform, buildstate) (Sum 1)) mapping)
details :: Text -> [Text] -> [Text]
details summary content = ["" <> summary <> "
", ""] <> content <> ["