Feature request: configuration file with sensible defaults like basedir, /etc or ~/.sitecrawler

[hanoii]

I don't think generating default for the crawler through a global configuration file is possible, right?

[janreges]

Hi @hanoii.

Adding a configuration file option is a good idea. It can save the need to define a lot of CLI parameters. It would help me if you could identify specific variables that should be set optionally from the configuration file.

Alternatively, we can also introduce the possibility of setting any parameters that are currently in the CLI parameters from the configuration file, but we would just come up with some form of notation that will work both ways.

Theoretically, a definition could also work, such as instead of --max-reqs-per-sec 10 in the CLI params, I would simply specify max-reqs-per-sec = 10 in the configuration file. If an item allows multiple definitions, it would be possible to add more lines with the same key in the configuration file.

Do you agree with this solution?

[hanoii]

I would do it in something more elaborate than a config file... something like yaml file where you can also have arrays (for multiple entries). And yes, I would continue supporting both command line arguments and config file overrides. And I would do the usual system-wide, user-wide, and --config-file override