Add stackage cabal.config fallback

When the default stackage json endpoint fails, fallback to trying
the <snapshot>/cabal.config endpoint. The latter seems more reliable.

This also allows us to easily add a --snapshot-path CLI arg, giving
users the ability to pass in a snapshot file manually.

Author: tbidne

.github/workflows/ci.yaml

@@ -43,6 +43,11 @@ jobs:
 
       - name: Functional Tests
         id: functional
+        # We want to run these tests even if the unit tests fail, because
+        # it is useful to know if e.g. the unit tests fail due to one
+        # stackage endpoint failing, but the functional tests pass due to
+        # a backup working.
+        if: always()
         shell: bash
         run: NO_CLEANUP=1 cabal test functional
 

README.md

@@ -61,23 +61,50 @@ The procedure is as follows:
 
 ### The clc-stackage exe
 
-Previously, this project was just a single (massive) cabal file that had to be manually updated. Usage was fairly simple: `cabal build clc-stackage --keep-going` to build the project, `--keep-going` so that as many packages as possible are built.
+`clc-stackage` is an executable that will:
 
-This has been updated so that `clc-stackage` is now an executable that will automatically generate the desired cabal file based on the results of querying stackage directly. This streamlines updates, provides a more flexible build process, and potentially has prettier output (with `--batch` arg):
+1. Download the stackage snapshot from the stackage server.
+2. Divide the snapshot into groups (determined by `--batch` argument).
+3. For each group, generate a cabal file and attempt to build it.
 
-![demo](example_output.png)
+#### Querying stackage
 
-In particular, the `clc-stackage` exe allows for splitting the entire package set into subset groups of size `N` with the `--batch N` option. Each group is then built sequentially. Not only can this be useful for situations where building the entire package set in one go is infeasible, but it also provides a "cache" functionality, that allows us to interrupt the program at any point (e.g. `CTRL-C`), and pick up where we left off. For example:
+By default, `clc-stackage` queries https://www.stackage.org/ for snapshot information. In situations where this is not desirable (e.g. the server is not working, or we want to test a custom snapshot), the snapshot can be overridden:
 
+```sh
+$ ./bin/clc-stackage --snapshot-path=path/to/snapshot
 ```
+
+This snapshot should be formatted similar to the `cabal.config` endpoint on the stackage server (e.g. https://www.stackage.org/nightly/cabal.config). That is, package lines should be formatted `<pkgs> ==<vers>`:
+
+```
+abstract-deque ==0.3
+abstract-deque-tests ==0.3
+abstract-par ==0.3.3
+AC-Angle ==1.0
+acc ==0.2.0.3
+...
+```
+
+The stackage config itself is valid, so trailing commas and other extraneous lines are allowed (and ignored).
+
+#### Investigating failures
+
+By default (`--write-logs save-failures`), the build logs are saved to the `./output/logs/` directory, with `./output/logs/current-build/` streaming the current build logs.
+
+#### Group batching
+
+The `clc-stackage` exe allows for splitting the entire package set into subset groups of size `N` with the `--batch N` option. Each group is then built sequentially. Not only can this be useful for situations where building the entire package set in one go is infeasible, but it also provides a "cache" functionality, that allows us to interrupt the program at any point (e.g. `CTRL-C`), and pick up where we left off. For example:
+
+```sh
 $ ./bin/clc-stackage --batch 100
 ```
 
 This will split the entire downloaded package set into groups of size 100. Each time a group finishes (success or failure), stdout/err will be updated, and then the next group will start. If the group failed to build and we have `--write-logs save-failures` (the default), then the logs and error output will be in `./output/logs/<pkg>/`, where `<pkg>` is the name of the first package in the group.
 
 See `./bin/clc-stackage --help` for more info.
 
-#### Optimal performance
+##### Optimal performance
 
 On the one hand, splitting the entire package set into `--batch` groups makes the output easier to understand and offers a nice workflow for interrupting/restarting the build. On the other hand, there is a question of what the best value of `N` is for `--batch N`, with respect to performance.
 

app/Main.hs

@@ -2,26 +2,15 @@ module Main (main) where
 
 import CLC.Stackage.Runner qualified as Runner
 import CLC.Stackage.Utils.Logging qualified as Logging
-import Data.Text qualified as T
-import Data.Time.LocalTime qualified as Local
 import System.Console.Terminal.Size qualified as TermSize
-import System.IO (hPutStrLn, stderr)
 
 main :: IO ()
 main = do
   mWidth <- (fmap . fmap) TermSize.width TermSize.size
 
+  let hLogger = Logging.mkDefaultLogger
   case mWidth of
-    Just w -> Runner.run $ mkLogger w
+    Just w -> Runner.run $ hLogger {Logging.terminalWidth = w}
     Nothing -> do
-      let hLogger = mkLogger 80
-      Logging.putTimeInfoStr hLogger False "Failed detecting terminal width"
+      Logging.putTimeInfoStr hLogger "Failed detecting terminal width"
       Runner.run hLogger
-  where
-    mkLogger w =
-      Logging.MkHandle
-        { Logging.getLocalTime = Local.zonedTimeToLocalTime <$> Local.getZonedTime,
-          Logging.logStrErrLn = hPutStrLn stderr . T.unpack,
-          Logging.logStrLn = putStrLn . T.unpack,
-          Logging.terminalWidth = w
-        }

clc-stackage.cabal

@@ -55,13 +55,15 @@ library parser
   exposed-modules:
     CLC.Stackage.Parser
     CLC.Stackage.Parser.API
-    CLC.Stackage.Parser.Data.Response
-    CLC.Stackage.Parser.Query
+    CLC.Stackage.Parser.API.CabalConfig
+    CLC.Stackage.Parser.API.Common
+    CLC.Stackage.Parser.API.JSON
 
   build-depends:
     , aeson
     , bytestring
     , containers       >=0.6.3.1 && <0.9
+    , deepseq          >=1.4.6.0 && <1.6
     , filepath
     , http-client      >=0.5.9   && <0.8
     , http-client-tls  ^>=0.3
@@ -121,8 +123,6 @@ executable clc-stackage
   build-depends:
     , runner
     , terminal-size  ^>=0.3.4
-    , text
-    , time
     , utils
 
   hs-source-dirs: ./app
@@ -143,6 +143,7 @@ test-suite unit
   type:           exitcode-stdio-1.0
   main-is:        Main.hs
   other-modules:
+    Unit.CLC.Stackage.Parser.API
     Unit.CLC.Stackage.Runner.Env
     Unit.CLC.Stackage.Runner.Report
     Unit.Prelude
@@ -151,11 +152,14 @@ test-suite unit
     , base
     , builder
     , containers
+    , deepseq
     , filepath
+    , http-client-tls
+    , parser
     , runner
     , tasty
     , tasty-golden
-    , tasty-hunit   >=0.9 && <0.11
+    , tasty-hunit      >=0.9 && <0.11
     , test-utils
     , time
     , utils

dev.md

@@ -96,9 +96,11 @@ The executable that actually runs. This is a very thin wrapper over `runner`, wh
     1. `ghc-version` in [.github/workflows/ci.yaml](.github/workflows/ci.yaml).
     2. [README.md](README.md).
 
-5. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
+5. Update functional tests as needed i.e. exact package versions in `*golden` and `test/functional/snapshot.txt`.
 
-6. Optional: Update nix inputs (`nix flake update`).
+6. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
+
+7. Optional: Update nix inputs (`nix flake update`).
 
 ## Testing
 

example_output.png

@@ -38,8 +38,6 @@ data BuildEnv = MkBuildEnv
     buildArgs :: [String],
     -- | Optional path to cabal executable.
     cabalPath :: FilePath,
-    -- | If true, colors logs.
-    colorLogs :: Bool,
     -- | If true, the first group that fails to completely build stops
     -- clc-stackage. Defaults to false.
     groupFailFast :: Bool,

src/builder/CLC/Stackage/Builder/Env.hs

@@ -9,7 +9,6 @@ import CLC.Stackage.Builder.Batch (PackageGroup (unPackageGroup))
 import CLC.Stackage.Builder.Env
   ( BuildEnv
       ( cabalPath,
-        colorLogs,
         groupFailFast,
         hLogger,
         progress,
@@ -101,11 +100,11 @@ buildProject env idx pkgs = do
     ExitSuccess -> do
       -- save results
       modifyIORef' env.progress.successesRef addPackages
-      Logging.putTimeSuccessStr env.hLogger env.colorLogs msg
+      Logging.putTimeSuccessStr env.hLogger msg
     ExitFailure _ -> do
       -- save results
       modifyIORef' env.progress.failuresRef addPackages
-      Logging.putTimeErrStr env.hLogger env.colorLogs msg
+      Logging.putTimeErrStr env.hLogger msg
 
       -- throw error if fail fast
       when env.groupFailFast $ throwIO exitCode

src/builder/CLC/Stackage/Builder/Process.hs

@@ -1,4 +1,3 @@
-{-# LANGUAGE CPP #-}
 {-# LANGUAGE QuasiQuotes #-}
 
 module CLC.Stackage.Parser
@@ -11,28 +10,32 @@ module CLC.Stackage.Parser
   )
 where
 
-import CLC.Stackage.Parser.Data.Response
+import CLC.Stackage.Parser.API
   ( PackageResponse (name, version),
     StackageResponse (packages),
   )
-import CLC.Stackage.Parser.Query qualified as Query
+import CLC.Stackage.Parser.API qualified as API
+import CLC.Stackage.Parser.API.CabalConfig qualified as CabalConfig
 import CLC.Stackage.Utils.IO qualified as IO
 import CLC.Stackage.Utils.JSON qualified as JSON
+import CLC.Stackage.Utils.Logging qualified as Logging
 import CLC.Stackage.Utils.OS (Os (Linux, Osx, Windows))
 import CLC.Stackage.Utils.OS qualified as OS
+import Control.Monad (when)
 import Data.Aeson (FromJSON, ToJSON)
 import Data.Foldable (for_)
 import Data.Set (Set)
 import Data.Set qualified as Set
 import Data.Text (Text)
 import Data.Text qualified as T
 import GHC.Generics (Generic)
-import System.OsPath (osp)
+import System.OsPath (OsPath, osp)
 
 -- | Retrieves the list of packages, based on
 -- 'CLC.Stackage.Parser.API.stackageUrl'.
-getPackageList :: IO [PackageResponse]
-getPackageList = getPackageListByOs OS.currentOs
+getPackageList :: Logging.Handle -> Maybe OsPath -> IO [PackageResponse]
+getPackageList hLogger msnapshotPath =
+  getPackageListByOs hLogger msnapshotPath OS.currentOs
 
 -- | Prints the package list to a file.
 printPackageList :: Bool -> Maybe Os -> IO ()
@@ -52,20 +55,36 @@ printPackageList incVers mOs = do
 
 -- | Retrieves the package list formatted to text.
 getPackageListByOsFmt :: Bool -> Os -> IO [Text]
-getPackageListByOsFmt incVers = (fmap . fmap) toText . getPackageListByOs
+getPackageListByOsFmt incVers =
+  (fmap . fmap) toText
+    . getPackageListByOs Logging.mkDefaultLogger Nothing
   where
     toText r =
       if incVers
         then r.name <> "-" <> r.version
         else r.name
 
 -- | Helper in case we want to see what the package set for a given OS is.
-getPackageListByOs :: Os -> IO [PackageResponse]
-getPackageListByOs os = do
+getPackageListByOs :: Logging.Handle -> Maybe OsPath -> Os -> IO [PackageResponse]
+getPackageListByOs hLogger msnapshotPath os = do
   excludedPkgs <- getExcludedPkgs os
   let filterExcluded = flip Set.notMember excludedPkgs . (.name)
 
-  response <- Query.getStackage
+  response <- case msnapshotPath of
+    Nothing -> API.getStackage hLogger
+    Just snapshotPath ->
+      CabalConfig.parseCabalConfig
+        <$> IO.readFileUtf8 snapshotPath
+
+  let numPackages = length response.packages
+  when (numPackages < 2000) $ do
+    let msg =
+          mconcat
+            [ "Only found ",
+              T.pack $ show numPackages,
+              " packages. Is that right?"
+            ]
+    Logging.putTimeWarnStr hLogger msg
 
   let packages = filter filterExcluded response.packages
 

src/parser/CLC/Stackage/Parser.hs

@@ -1,35 +1,54 @@
 -- | REST API for stackage.org.
 module CLC.Stackage.Parser.API
-  ( withResponse,
+  ( -- * Querying stackage
+    StackageResponse (..),
+    PackageResponse (..),
+    getStackage,
+
+    -- ** Exceptions
+    StackageException (..),
+    ExceptionReason (..),
+
+    -- * Misc
     stackageSnapshot,
   )
 where
 
-import Network.HTTP.Client (BodyReader, Request, Response)
-import Network.HTTP.Client qualified as HttpClient
+import CLC.Stackage.Parser.API.CabalConfig qualified as CabalConfig
+import CLC.Stackage.Parser.API.Common
+  ( ExceptionReason
+      ( ReasonDecodeJson,
+        ReasonDecodeUtf8,
+        ReasonReadBody,
+        ReasonStatus
+      ),
+    PackageResponse (name, version),
+    StackageException (MkStackageException),
+    StackageResponse (MkStackageResponse, packages),
+  )
+import CLC.Stackage.Parser.API.JSON qualified as JSON
+import CLC.Stackage.Utils.Exception qualified as Ex
+import CLC.Stackage.Utils.Logging qualified as Logging
+import Control.Exception (Exception (displayException))
+import Data.Text qualified as T
 import Network.HTTP.Client.TLS qualified as TLS
 
--- | Hits the stackage endpoint, invoking the callback on the result.
-withResponse :: (Response BodyReader -> IO a) -> IO a
-withResponse onResponse = do
+-- | Returns the 'StackageResponse' corresponding to the given snapshot.
+getStackage :: Logging.Handle -> IO StackageResponse
+getStackage hLogger = do
   manager <- TLS.newTlsManager
-  req <- getRequest
-  HttpClient.withResponse req manager onResponse
+  Ex.tryAny (JSON.getStackage manager stackageSnapshot) >>= \case
+    Right r1 -> pure $ r1
+    Left jsonEx -> do
+      let msg =
+            mconcat
+              [ "Json endpoint failed. Trying cabal config next: ",
+                T.pack $ displayException jsonEx
+              ]
 
-getRequest :: IO Request
-getRequest = updateReq <$> mkReq
-  where
-    mkReq = HttpClient.parseRequest stackageUrl
-    updateReq r =
-      r
-        { HttpClient.requestHeaders =
-            [ ("Accept", "application/json;charset=utf-8,application/json")
-            ]
-        }
+      Logging.putTimeWarnStr hLogger msg
 
--- | Url for the stackage snapshot.
-stackageUrl :: String
-stackageUrl = "https://stackage.org/" <> stackageSnapshot
+      CabalConfig.getStackage manager stackageSnapshot
 
 -- | Stackage snapshot. Note that picking a "good" snapshot is something of
 -- an art i.e. not all valid snapshots return json output at the