Allow with custom cabal path

.github/workflows/ci.yaml

@@ -43,6 +43,11 @@ jobs:
 
       - name: Functional Tests
         id: functional
+        # We want to run these tests even if the unit tests fail, because
+        # it is useful to know if e.g. the unit tests fail due to one
+        # stackage endpoint failing, but the functional tests pass due to
+        # a backup working.
+        if: always()
         shell: bash
         run: NO_CLEANUP=1 cabal test functional
 

README.md

@@ -61,23 +61,50 @@ The procedure is as follows:
 
 ### The clc-stackage exe
 
-Previously, this project was just a single (massive) cabal file that had to be manually updated. Usage was fairly simple: `cabal build clc-stackage --keep-going` to build the project, `--keep-going` so that as many packages as possible are built.
+`clc-stackage` is an executable that will:
 
-This has been updated so that `clc-stackage` is now an executable that will automatically generate the desired cabal file based on the results of querying stackage directly. This streamlines updates, provides a more flexible build process, and potentially has prettier output (with `--batch` arg):
+1. Download the stackage snapshot from the stackage server.
+2. Divide the snapshot into groups (determined by `--batch` argument).
+3. For each group, generate a cabal file and attempt to build it.
 
-![demo](example_output.png)
+#### Querying stackage
 
-In particular, the `clc-stackage` exe allows for splitting the entire package set into subset groups of size `N` with the `--batch N` option. Each group is then built sequentially. Not only can this be useful for situations where building the entire package set in one go is infeasible, but it also provides a "cache" functionality, that allows us to interrupt the program at any point (e.g. `CTRL-C`), and pick up where we left off. For example:
+By default, `clc-stackage` queries https://www.stackage.org/ for snapshot information. In situations where this is not desirable (e.g. the server is not working, or we want to test a custom snapshot), the snapshot can be overridden:
 
+```sh
+$ ./bin/clc-stackage --snapshot-path=path/to/snapshot
 ```
+
+This snapshot should be formatted similar to the `cabal.config` endpoint on the stackage server (e.g. https://www.stackage.org/nightly/cabal.config). That is, package lines should be formatted `<pkgs> ==<vers>`:
+
+```
+abstract-deque ==0.3
+abstract-deque-tests ==0.3
+abstract-par ==0.3.3
+AC-Angle ==1.0
+acc ==0.2.0.3
+...
+```
+
+The stackage config itself is valid, so trailing commas and other extraneous lines are allowed (and ignored).
+
+#### Investigating failures
+
+By default (`--write-logs save-failures`), the build logs are saved to the `./output/logs/` directory, with `./output/logs/current-build/` streaming the current build logs.
+
+#### Group batching
+
+The `clc-stackage` exe allows for splitting the entire package set into subset groups of size `N` with the `--batch N` option. Each group is then built sequentially. Not only can this be useful for situations where building the entire package set in one go is infeasible, but it also provides a "cache" functionality, that allows us to interrupt the program at any point (e.g. `CTRL-C`), and pick up where we left off. For example:
+
+```sh
 $ ./bin/clc-stackage --batch 100
 ```
 
 This will split the entire downloaded package set into groups of size 100. Each time a group finishes (success or failure), stdout/err will be updated, and then the next group will start. If the group failed to build and we have `--write-logs save-failures` (the default), then the logs and error output will be in `./output/logs/<pkg>/`, where `<pkg>` is the name of the first package in the group.
 
 See `./bin/clc-stackage --help` for more info.
 
-#### Optimal performance
+##### Optimal performance
 
 On the one hand, splitting the entire package set into `--batch` groups makes the output easier to understand and offers a nice workflow for interrupting/restarting the build. On the other hand, there is a question of what the best value of `N` is for `--batch N`, with respect to performance.
 

app/Main.hs

@@ -2,26 +2,15 @@ module Main (main) where
 
 import CLC.Stackage.Runner qualified as Runner
 import CLC.Stackage.Utils.Logging qualified as Logging
-import Data.Text qualified as T
-import Data.Time.LocalTime qualified as Local
 import System.Console.Terminal.Size qualified as TermSize
-import System.IO (hPutStrLn, stderr)
 
 main :: IO ()
 main = do
   mWidth <- (fmap . fmap) TermSize.width TermSize.size
 
+  let hLogger = Logging.mkDefaultLogger
   case mWidth of
-    Just w -> Runner.run $ mkLogger w
+    Just w -> Runner.run $ hLogger {Logging.terminalWidth = w}
     Nothing -> do
-      let hLogger = mkLogger 80
-      Logging.putTimeInfoStr hLogger False "Failed detecting terminal width"
+      Logging.putTimeInfoStr hLogger "Failed detecting terminal width"
       Runner.run hLogger
-  where
-    mkLogger w =
-      Logging.MkHandle
-        { Logging.getLocalTime = Local.zonedTimeToLocalTime <$> Local.getZonedTime,
-          Logging.logStrErrLn = hPutStrLn stderr . T.unpack,
-          Logging.logStrLn = putStrLn . T.unpack,
-          Logging.terminalWidth = w
-        }

cabal.project

@@ -17,7 +17,6 @@ program-options
     -Wprepositive-qualified-module
     -Wredundant-constraints
     -Wunused-binds
-    -Wunused-packages
     -Wunused-type-patterns
     -Wno-unticked-promoted-constructors
 

clc-stackage.cabal

@@ -24,7 +24,6 @@ common common-lang
   if os(windows)
     cpp-options: -DWINDOWS
 
-  build-depends:      base >=4.16.0.0 && <4.22
   default-language:   GHC2021
 
 library utils
@@ -35,15 +34,19 @@ library utils
     CLC.Stackage.Utils.JSON
     CLC.Stackage.Utils.Logging
     CLC.Stackage.Utils.OS
+    CLC.Stackage.Utils.Package
     CLC.Stackage.Utils.Paths
 
   build-depends:
     , aeson            >=2.0       && <2.3
     , aeson-pretty     ^>=0.8.9
+    , base             >=4.16.0.0  && <4.22
     , bytestring       >=0.10.12.0 && <0.13
+    , deepseq          >=1.4.6.0   && <1.6
     , directory        ^>=1.3.5.0
     , file-io          ^>=0.1.0.0
-    , filepath         >=1.4.2.1   && <1.6
+    , filepath         >=1.5.0.0   && <1.6
+    , os-string        ^>=2.0.0
     , pretty-terminal  ^>=0.1.0.0
     , text             >=1.2.3.2   && <2.2
     , time             >=1.9.3     && <1.15
@@ -55,13 +58,16 @@ library parser
   exposed-modules:
     CLC.Stackage.Parser
     CLC.Stackage.Parser.API
-    CLC.Stackage.Parser.Data.Response
-    CLC.Stackage.Parser.Query
+    CLC.Stackage.Parser.API.CabalConfig
+    CLC.Stackage.Parser.API.Common
+    CLC.Stackage.Parser.API.JSON
 
   build-depends:
     , aeson
+    , base
     , bytestring
     , containers       >=0.6.3.1 && <0.9
+    , deepseq
     , filepath
     , http-client      >=0.5.9   && <0.8
     , http-client-tls  ^>=0.3
@@ -70,19 +76,19 @@ library parser
     , utils
 
   hs-source-dirs:  src/parser
+  ghc-options:     -Wunused-packages
 
 library builder
   import:          common-lang
   exposed-modules:
     CLC.Stackage.Builder
     CLC.Stackage.Builder.Batch
     CLC.Stackage.Builder.Env
-    CLC.Stackage.Builder.Package
     CLC.Stackage.Builder.Process
     CLC.Stackage.Builder.Writer
 
   build-depends:
-    , aeson
+    , base
     , containers
     , directory
     , filepath
@@ -91,6 +97,7 @@ library builder
     , utils
 
   hs-source-dirs:  src/builder
+  ghc-options:     -Wunused-packages
 
 library runner
   import:          common-lang
@@ -102,6 +109,7 @@ library runner
 
   build-depends:
     , aeson
+    , base
     , builder
     , containers
     , directory
@@ -114,56 +122,60 @@ library runner
     , utils
 
   hs-source-dirs:  src/runner
+  ghc-options:     -Wunused-packages
 
 executable clc-stackage
   import:         common-lang
   main-is:        Main.hs
   build-depends:
+    , base
     , runner
     , terminal-size  ^>=0.3.4
-    , text
-    , time
     , utils
 
   hs-source-dirs: ./app
-  ghc-options:    -threaded -with-rtsopts=-N
+  ghc-options:    -threaded -with-rtsopts=-N -Wunused-packages
 
 library test-utils
   import:          common-lang
-  exposed-modules:
-    Test.Utils
-
+  exposed-modules: Test.Utils
   build-depends:
     , base
     , tasty         >=1.1.0.3  && <1.6
     , tasty-golden  ^>=2.3.1.1
 
   hs-source-dirs:  test/utils
+  ghc-options:     -Wunused-packages
 
 test-suite unit
   import:         common-lang
   type:           exitcode-stdio-1.0
   main-is:        Main.hs
   other-modules:
+    Unit.CLC.Stackage.Parser.API
     Unit.CLC.Stackage.Runner.Env
     Unit.CLC.Stackage.Runner.Report
+    Unit.CLC.Stackage.Utils.Package
     Unit.Prelude
 
   build-depends:
     , base
     , builder
     , containers
+    , deepseq
     , filepath
+    , http-client-tls
+    , parser
     , runner
     , tasty
     , tasty-golden
-    , tasty-hunit   >=0.9      && <0.11
+    , tasty-hunit      >=0.9 && <0.11
     , test-utils
     , time
     , utils
 
   hs-source-dirs: test/unit
-  ghc-options:    -threaded -with-rtsopts=-N
+  ghc-options:    -threaded -with-rtsopts=-N -Wunused-packages
 
 test-suite functional
   import:         common-lang
@@ -177,11 +189,15 @@ test-suite functional
     , env-guard     ^>=0.2
     , filepath
     , runner
-    , test-utils
     , tasty
     , tasty-golden
+    , test-utils
     , text
     , time
     , utils
 
   hs-source-dirs: test/functional
+
+-- For some reason -Wunused-packages is complaining about clc-stackage
+-- being an unnecessary dep for the functional test suite...hence it is
+-- removed from cabal.project and added manually to other targets.

dev.md

@@ -96,9 +96,11 @@ The executable that actually runs. This is a very thin wrapper over `runner`, wh
     1. `ghc-version` in [.github/workflows/ci.yaml](.github/workflows/ci.yaml).
     2. [README.md](README.md).
 
-5. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
+5. Update functional tests as needed i.e. exact package versions in `*golden` and `test/functional/snapshot.txt`.
 
-6. Optional: Update nix inputs (`nix flake update`).
+6. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
+
+7. Optional: Update nix inputs (`nix flake update`).
 
 ## Testing
 

src/builder/CLC/Stackage/Builder/Batch.hs

@@ -10,7 +10,7 @@ import CLC.Stackage.Builder.Env
         packagesToBuild
       ),
   )
-import CLC.Stackage.Builder.Package (Package)
+import CLC.Stackage.Utils.Package (Package)
 import Data.Bifunctor (Bifunctor (first))
 import Data.List qualified as L
 import Data.List.NonEmpty (NonEmpty ((:|)), (<|))

src/builder/CLC/Stackage/Builder/Env.hs

@@ -1,54 +1,16 @@
 -- | Provides the environment for building.
 module CLC.Stackage.Builder.Env
   ( BuildEnv (..),
-    CabalVerbosity (..),
-    cabalVerbosityToArg,
-    Jobs (..),
-    jobsToArg,
     Progress (..),
     WriteLogs (..),
   )
 where
 
-import CLC.Stackage.Builder.Package (Package)
 import CLC.Stackage.Utils.Logging qualified as Logging
+import CLC.Stackage.Utils.Package (Package)
 import Data.IORef (IORef)
 import Data.List.NonEmpty (NonEmpty)
 import Data.Set (Set)
-import Data.Word (Word8)
-
--- | Cabal's --verbose flag
-data CabalVerbosity
-  = -- | V0
-    CabalVerbosity0
-  | -- | V1
-    CabalVerbosity1
-  | -- | V2
-    CabalVerbosity2
-  | -- | V3
-    CabalVerbosity3
-  deriving stock (Eq, Show)
-
-cabalVerbosityToArg :: CabalVerbosity -> String
-cabalVerbosityToArg CabalVerbosity0 = "--verbose=0"
-cabalVerbosityToArg CabalVerbosity1 = "--verbose=1"
-cabalVerbosityToArg CabalVerbosity2 = "--verbose=2"
-cabalVerbosityToArg CabalVerbosity3 = "--verbose=3"
-
--- | Number of build jobs.
-data Jobs
-  = -- | Literal number of jobs.
-    JobsN Word8
-  | -- | String "$ncpus"
-    JobsNCpus
-  | -- | Job semaphore. Requires GHC 9.8 and Cabal 3.12
-    JobsSemaphore
-  deriving stock (Eq, Show)
-
-jobsToArg :: Jobs -> String
-jobsToArg (JobsN n) = "--jobs=" ++ show n
-jobsToArg JobsNCpus = "--jobs=$ncpus"
-jobsToArg JobsSemaphore = "--semaphore"
 
 data Progress = MkProgress
   { -- | Dependencies that built successfully.
@@ -74,8 +36,8 @@ data BuildEnv = MkBuildEnv
     batch :: Maybe Int,
     -- | Build arguments for cabal.
     buildArgs :: [String],
-    -- | If true, colors logs.
-    colorLogs :: Bool,
+    -- | Optional path to cabal executable.
+    cabalPath :: FilePath,
     -- | If true, the first group that fails to completely build stops
     -- clc-stackage. Defaults to false.
     groupFailFast :: Bool,