Merge pull request #13 from advanced-security/trap-caching

Massive improvement to workflow

Author: GeekMasher

README.md

@@ -1,10 +1,30 @@
-# gh-codeql-scan
+<!-- markdownlint-disable -->
+<div align="center">
 
-GitHub CLI CodeQL Scan Extension to help abstract CodeQL away from users.
+<h1>gh-codeql-scan</h1>
+
+[![GitHub](https://img.shields.io/badge/github-%23121011.svg?style=for-the-badge&logo=github&logoColor=white)][github]
+[![GitHub Issues](https://img.shields.io/github/issues/advanced-security/gh-codeql-scan?style=for-the-badge)][github-issues]
+[![GitHub Stars](https://img.shields.io/github/stars/advanced-security/gh-codeql-scan?style=for-the-badge)][github]
+[![License](https://img.shields.io/github/license/advanced-security/gh-codeql-scan?style=for-the-badge)][license]
+
+</div>
+<!-- markdownlint-restore -->
+
+[GitHub CLI CodeQL Scan Extension][github] to help abstract [CodeQL][codeql] away from users.
+
+<details>
+<summary>Motivation</summary>
+
+This project was created to make the lives of users that use CodeQL simpiler.
+CodeQL outside of GitHub Actions can be complicated but this projects aim is to make it as simple as possible.
+
+</details>
 
 ## Requirements
 
 - [GitHub CLI](https://cli.github.com/)
+- [CodeQL GH Extension][gh-codeql] (optional)
 
 ## Install and Setup
 
@@ -17,9 +37,59 @@ gh extensions install advanced-security/gh-codeql-scan
 gh codeql-scan --help
 ```
 
+<details>
+<summary>CLI Help</summary>
+
+<pre>
+GitHub CodeQL Scan tool
+
+gh codeql-scan {MODE} {ARGS}
+
+# Modes
+
+gh codeql-scan              # default: "scan"
+gh codeql-scan init         # initialise the scan 
+gh codeql-scan analyze      # run the analysis
+gh codeql-scan upload       # upload present SARIF files
+gh codeql-scan scan         # full end-to-end scan 
+
+# Arguments
+
+> All arguments can be set with enviroment variables
+
+-h|--help               # Print help
+--debug                 # Enable debugging
+    
+-r=*|--repo=*           # GitHub Respository (OWNER/NAME)
+-i=*|--instance=*       # GitHub Instance (github.com or Enterprise Server)
+
+-l=*|--language=*       # Set language to scan
+--auto-detect           # Auto-detect languages
+
+-s=*|--suite=*          # Query Suite to use
+-d=*|--databases=*      # Location of the databases to store
+-b=*|--binary=*         # Path to the CodeQL Binary
+-w=*|--workspace=*      # Workspace for the souce code
+
+-c=*|--command=*        # Set the build comment (compiled languages)
+-m=*|--mode=*           # Build mode (autobuild | none)
+--buildless             # Enable buildless / build mode none
+
+--view-in-vscode        # Auto-open the results in VSCode
+
+--disable-tracing       # Disable Build Tracing
+--disable-trap-caching  # Disable Trap file caching
+--disable-upload        # Disable Uploading SARIF to GitHub
+--disable-banner        # Disable printing banner
+</pre>
+
+</details>
+
+### Alias / Stub
+
 A couple of tips and tricks:
 
-```
+```bash
 # Create an alias to make things even easier
 alias codeql-scan="gh codeql-scan"
 ```
@@ -33,46 +103,79 @@ The main use of the script is to automatically run CodeQL in a number of modes.
 gh codeql-scan
 ```
 
+#### Initialise with language
+
+Automatically detect languages or manually set the language to create an initial CodeQL database.
+
 ```bash
-# `init` mode: Create only the Codeql database
 gh codeql-scan init --auto-detect
 # or manually set language
 gh codeql-scan init -l=java 
 ```
 
+#### Scan without build
+
+This will scan your code in build mode `none`.
+
+```bash
+gh codeql-scan -m="none"
+# or simply
+gh codeql-scan --buildless
+```
+
+#### Scan with Build Command
+
+Pass in the build command for a compiled language and it will be run along with CodeQL.
+
 ```bash
-# Compiled languages -  pass in build command
 gh codeql-scan -c "mvn build ..."
 ```
 
+#### Indirect build tracing
+
+For Compiled languages, complicated build process using indirect build tracing
+
 ```bash
-# Compiled languages - complicated build process using indirect build tracing
 gh codeql-scan init
 echo "password=$password" > settings.xml
 mvn build --random-custom=flags
 gh codeql-scan analyze
 ```
 
+#### Running analysis
+
+Run query-suites on an existing database (auto-detects databases)
+
 ```bash
-# `analyze` mode: Run query-suites on an existing database (auto-detects databases)
 gh codeql-scan analyze
 ```
 
+#### Uploading results to GitHub
+
+The `upload` mode will upload all SARIF files for you to a repository
+
 ```bash
-# `upload` mode: Upload all SARIF files
 gh codeql-scan upload
 ```
 
-## License 
-
-This project is licensed under the terms of the MIT open source license. Please refer to [MIT](./LICENSE.md) for the full terms.
-
 ## Maintainers 
 
 - @GeekMasher
 
-## Support
+## Support / Maintainance
+
+Support is via [GitHub Issues][github-issues]
+
+## License 
+
+This project is licensed under the terms of the MIT open source license.
+Please refer to [MIT][license] for the full terms.
+
+<!-- Resources -->
 
-Support is via [GitHub Issues](https://github.com/advanced-security/gh-codeql-scan/issues)
+[license]: ./LICENSE
+[github]: https://github.com/advanced-security/gh-codeql-scan
+[github-issues]: https://github.com/advanced-security/gh-codeql-scan/issues
+[codeql]: https://codeql.github.com/
+[gh-codeql]: https://github.com/github/gh-codeql
 
-## Acknowledgement

bin/codeql-analyze

@@ -19,10 +19,6 @@ for i in "$@"; do
         CODEQL_ADD_SNIPPETS=""
         shift
         ;;
-    --disable-uploading)
-        GITHUB_UPLOAD=0
-        shift
-        ;;
     -s=*|--suite=*)
         export CODEQL_SUITE="${i#*=}"
         shift
@@ -36,7 +32,7 @@ for i in "$@"; do
   esac
 done
 
-if [ -z $CODEQL_DATABASE_PATHS ]; then
+if [[ -z $CODEQL_DATABASE_PATHS ]]; then
     error "No CodeQL Databases found"
     exit 1
 fi
@@ -55,14 +51,14 @@ for CODEQL_DATABASE in $CODEQL_DATABASE_PATHS ; do
 
     # For tracing
     debug "Finalizing CodeQL Database"
-    $CODEQL_BINARY database finalize $CODEQL_DATABASE 2>/dev/null | true
+    $CODEQL_BINARY database finalize --finalize-dataset $CODEQL_DATABASE 2>/dev/null | true
 
     # The --sarif-category must be set in case of multiple databases
     $CODEQL_BINARY database analyze \
         --format="sarif-latest" \
         --sarif-category="codeql-scan:${CODEQL_LANGUAGE}" \
         --output=$CODEQL_SARIF \
-        -j=0 \
+        -j=0 -M="$(memory)" \
         $CODEQL_ADD_HELP $CODEQL_ADD_SNIPPETS \
         ${CODEQL_DATABASE} ${CODEQL_SUITE}
 

bin/codeql-init

@@ -2,31 +2,34 @@
 set -e
 source $EXTENSION_LOCATION/bin/codeql-utils
 
-for i in "$@"; do
-  case $i in
+while [[ "$#" -gt 0 ]]; do
+  case $1 in
     --auto-detect)
         GITHUB_AUTO_DETECT="1"
-        shift
         ;;
     --disable-tracing)
         export CODEQL_TRACING=0
-        shift
         ;;
      -l=*|--language=*)
-        export CODEQL_LANGUAGES="${i#*=}"
-        shift
+        export CODEQL_LANGUAGES="${1#*=}"
         ;;
-    -c="*"|--command="*")
-        export CODEQL_BUILD_COMMAND="${i#*=}"
+    -c=*|--command=*)
+        export CODEQL_BUILD_COMMAND="${1#*=}"
         export CODEQL_TRACING="1"    # enable tracing
-        shift
+        ;;
+    -m=*|--mode=*)
+        export CODEQL_BUILD_MODE="${1#*=}"
+        ;;
+    --disable-trap-caching)
+        export CODEQL_TRAP_CACHING=0
         ;;
     *)
         ;;
   esac
+  shift
 done
 
- 
+
 if [ ${GITHUB_UPLOAD} == 0 ] && [ -z ${GITHUB_REPOSITORY+x} ]; then
     # If the repo isn't set 
     export GITHUB_REPOSITORY="$(basename $PWD)"
@@ -62,6 +65,9 @@ if [ -z $CODEQL_LANGUAGES ] ; then
     exit 1
 fi
 
+MEMORY=$(memory)
+info "Memory :: $MEMORY (MB)"
+
 # comma seperated list (`java,javascript`)
 for CODEQL_LANGUAGE in ${CODEQL_LANGUAGES//,/ } ; do
     debug "CodeQL Language :: $CODEQL_LANGUAGE"
@@ -79,9 +85,14 @@ for CODEQL_LANGUAGE in ${CODEQL_LANGUAGES//,/ } ; do
     debug "CodeQL Binary :: $CODEQL_BINARY"
     debug "Tracing :: $CODEQL_TRACING"
 
-    CODEQL_CREATE="$CODEQL_BINARY database create -j=0 -M=16000 --language=$CODEQL_LANGUAGE"
-    
-    if [ "$CODEQL_TRACING" = "1" ] && [[ "$CODEQL_LANGUAGE" =~ ^(cpp|csharp|java|go)$ ]]; then
+    CODEQL_CREATE="$CODEQL_BINARY"
+
+    if [ "$CODEQL_BUILD_MODE" = "none" ] && [[ "$CODEQL_LANGUAGES" =~ ^(csharp|java)$ ]]; then
+        info "CodeQL build mode is set to none"
+
+        CODEQL_CREATE="$CODEQL_CREATE database init --language=$CODEQL_LANGUAGE --build-mode=none"
+
+    elif [ "$CODEQL_TRACING" = "1" ] && [[ "$CODEQL_LANGUAGE" =~ ^(cpp|csharp|java|go)$ ]]; then
         debug "Swap to a init command and add --begin-tracing"
 
         # check if the OS is Mac
@@ -91,13 +102,40 @@ for CODEQL_LANGUAGE in ${CODEQL_LANGUAGES//,/ } ; do
         fi
 
         # https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#using-indirect-build-tracing
-        CODEQL_CREATE="$CODEQL_BINARY database init --begin-tracing --language=$CODEQL_LANGUAGE --source-root=$WORKSPACE $CODEQL_DATABASE"
+        CODEQL_CREATE="$CODEQL_CREATE database init --begin-tracing --language=$CODEQL_LANGUAGE"
+
+    elif [[ ! -z $CODEQL_BUILD_COMMAND ]]; then
+        debug "Enabling CodeQL manual build"
+        CODEQL_CREATE="$CODEQL_CREATE --command \"${CODEQL_BUILD_COMMAND}\""
     else
         debug "Append only Database"
         debug "CodeQL auto-build is enabled"
-        CODEQL_CREATE="$CODEQL_CREATE $CODEQL_DATABASE"
+        CODEQL_CREATE="$CODEQL_CREATE database create -j=0 -M=$MEMORY --language=$CODEQL_LANGUAGE"
+    fi
+
+    if [[ ! -z "${WORKSPACE}" ]]; then
+        debug "Setting source-root to workspace"
+        CODEQL_CREATE="$CODEQL_CREATE --source-root=$WORKSPACE"
     fi
 
+    # TRAP File caching (C/C++ only for now)
+    if [ "$CODEQL_TRAP_CACHING" = "1" ] && [[ "$CODEQL_LANGUAGE" = "cpp" ]]; then
+        info "CodeQL TRAP Caching is enabled"
+        info "CodeQL TRAP Caching Dir :: $CODEQL_TRAP_CACHING_DIR"
+    
+        CODEQL_TRAP_CACHING_PROJECT="$CODEQL_TRAP_CACHING_DIR/${CODEQL_LANGUAGE}-${CODEQL_GITHUB_REPOSITORY}"
+        # Make sure the directory exists
+        mkdir -p $CODEQL_TRAP_CACHING_PROJECT
+
+        # Set the environment variable
+        export CODEQL_EXTRACTOR_CPP_TRAP_CACHING=true
+
+        CODEQL_CREATE="$CODEQL_CREATE -O=cpp.trap.cache.dir=$CODEQL_TRAP_CACHING_PROJECT -O=cpp.trap.cache.bound=1024 -O=cpp.trap.cache.write=true"
+    fi
+    
+    # Add the database to the command
+    CODEQL_CREATE="$CODEQL_CREATE $CODEQL_DATABASE"
+
     info "Runnning CodeQL database init/create command..."
     debug "CodeQL Create Command :: $CODEQL_CREATE"
 
@@ -111,7 +149,15 @@ for CODEQL_LANGUAGE in ${CODEQL_LANGUAGES//,/ } ; do
         debug "CodeQL Tracer Languages :: $CODEQL_TRACER_LANGUAGES"
     fi
 
-    if [ ! -z ${CODEQL_BUILD_COMMAND+x} ] && [[ "$CODEQL_LANGUAGE" =~ ^(cpp|csharp|java|go)$ ]] ; then
+    if [ "$CODEQL_BUILD_MODE" = "none" ] && [[ "$CODEQL_LANGUAGE" =~ ^(csharp|java)$ ]]; then
+        info "Running buildless mode..."
+        
+        $CODEQL_BINARY database trace-command --use-build-mode $CODEQL_DATABASE
+
+        debug "Finalizing CodeQL Database"
+        $CODEQL_BINARY database finalize $CODEQL_DATABASE | true
+
+    elif [ ! -z ${CODEQL_BUILD_COMMAND+x} ] && [[ "$CODEQL_LANGUAGE" =~ ^(cpp|csharp|java|go)$ ]] ; then
         info "Running build command :: '$CODEQL_BUILD_COMMAND'"
 
         eval ${CODEQL_BUILD_COMMAND}