Add test code (#82)

Added `test` key to each pattern

* Added test code to generic patterns
* Fixed up some of the test data for ./configs
* Fixed test data and patterns more
* Fixed generic data
* Fixed common passwords tests and added "test" key
* Test data for IBAN patterns
* .NET config test data
* Test data for machineKey
* More database test data
* Added DB test files
* Added GPG test data
* Test data for Bearer headers
* Test data for Arc
* Test data for PII
* Test data for keys
* Added test data for Azure Connection String
* Added test data for Grafana
* Added test data for all Sentry patterns
* Added Okta test data
* Added DataDog test data
* Added test data to JWT
* Add test data for hardcoded URI passwords and fixed pattern
* Added test data for IPv4 and ghcr typos
* Update workflow-application-token-action to v3.0.0
* Test unit tests with Actions workflow
* Fixed over-matching in ssh_rsa pattern

Author: aegilops

.github/workflows/pr-markdown.yml

@@ -13,7 +13,7 @@ jobs:
 
       - name: Get Token
         id: get_workflow_token
-        uses: peter-murray/workflow-application-token-action@8e1ba3bf1619726336414f1014e37f17fbadf1db #v2.1.0
+        uses: peter-murray/workflow-application-token-action@dc0413987a085fa17d19df9e47d4677cf81ffef3 #v3.0.0
         with:
           application_id: ${{ secrets.ADVANCED_SECURITY_APP_ID }}
           application_private_key: ${{ secrets.ADVANCED_SECURITY_APP_KEY }}

.github/workflows/unit-tests.yml

@@ -0,0 +1,39 @@
+name: Unit Tests with hyperscan
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main, develop]
+
+  workflow_dispatch:
+
+jobs:
+  run:
+    runs-on: ubuntu-latest
+
+    permissions:
+      security-events: write
+
+    steps:
+      - name: Checkout custom patterns
+        uses: actions/checkout@v4
+
+      - name: Checkout secret-scanning-tools
+        uses: actions/checkout@v4
+        with:
+          repository: "advanced-security/secret-scanning-tools"
+          path: "secret-scanning-tools"
+
+      - name: Install dependencies
+        run: |
+          cd "${GITHUB_WORKSPACE}"/secret-scanning-tools/secretscanning
+          sudo apt-get -qq update
+          sudo apt-get -qq install libpcre3-dev
+          python3 -mpip -q install -r requirements.txt
+          python3 -mpip -q install tqdm
+
+      - name: Unit Test patterns with hyperscan
+        run: |
+          cd "${GITHUB_WORKSPACE}"/secret-scanning-tools/secretscanning
+          python3 ./test.py --tests "${GITHUB_WORKSPACE}" --exclude django_secret_key

.github/workflows/validate.yml

@@ -20,7 +20,7 @@ jobs:
 
       - name: Get Token
         id: get_workflow_token
-        uses: peter-murray/workflow-application-token-action@8e1ba3bf1619726336414f1014e37f17fbadf1db #v2.1.0
+        uses: peter-murray/workflow-application-token-action@dc0413987a085fa17d19df9e47d4677cf81ffef3 #v3.0.0
         with:
           application_id: ${{ secrets.ADVANCED_SECURITY_APP_ID }}
           application_private_key: ${{ secrets.ADVANCED_SECURITY_APP_KEY }}

common/patterns.yml

@@ -19,7 +19,11 @@ patterns:
         # web colours
         - ^\#[0-9]+$
         # long strings, unlikely to be a weak password
-        - ^.{20,} 
+        - ^.{20,}
+    test:
+      data: some_variable="p@55w0rd123"
+      start_offset: 15
+      end_offset: 26
     comments:
       - ⚠️  prone to high numbers of false positives, use with caution ⚠️ 
       - very small common password shortlist from SecLists
@@ -28,79 +32,111 @@ patterns:
       - adds some l33tsp3@k variations
 
     expected:
-      - name: top-passwords-shortlist.txt
-        start_offset: 13
-        end_offset: 21
-      - name: top-passwords-shortlist.txt
-        start_offset: 34
-        end_offset: 40
-      - name: top-passwords-shortlist.txt
-        start_offset: 49
-        end_offset: 57
-      - name: top-passwords-shortlist.txt
-        start_offset: 64
-        end_offset: 70
-      - name: top-passwords-shortlist.txt
-        start_offset: 80
-        end_offset: 86
-      - name: top-passwords-shortlist.txt
-        start_offset: 94
-        end_offset: 100
-      - name: top-passwords-shortlist.txt
-        start_offset: 106
-        end_offset: 113
-      - name: top-passwords-shortlist.txt
-        start_offset: 119
-        end_offset: 125
-      - name: top-passwords-shortlist.txt
-        start_offset: 131
-        end_offset: 137
-      - name: top-passwords-shortlist.txt
-        start_offset: 143
-        end_offset: 151
-      - name: top-passwords-shortlist.txt
-        start_offset: 161
-        end_offset: 169
-      - name: top-passwords-shortlist.txt
-        start_offset: 179
-        end_offset: 187
-      - name: top-passwords-shortlist.txt
-        start_offset: 197
-        end_offset: 204
-      - name: top-passwords-shortlist.txt
-        start_offset: 214
-        end_offset: 222
-      - name: top-passwords-shortlist.txt
-        start_offset: 232
-        end_offset: 238
-      - name: top-passwords-shortlist.txt
-        start_offset: 248
-        end_offset: 254
-      - name: top-passwords-shortlist.txt
-        start_offset: 264
-        end_offset: 271
-      - name: top-passwords-shortlist.txt
-        start_offset: 281
-        end_offset: 287
-      - name: top-passwords-shortlist.txt
-        start_offset: 297
-        end_offset: 303
-      - name: top-passwords-shortlist.txt
-        start_offset: 313
-        end_offset: 320
-      - name: top-passwords-shortlist.txt
-        start_offset: 330
-        end_offset: 335
-      - name: top-passwords-shortlist.txt
-        start_offset: 345
-        end_offset: 352
-      - name: top-passwords-shortlist.txt
-        start_offset: 362
-        end_offset: 367
-      - name: top-passwords-shortlist.txt
-        start_offset: 377
-        end_offset: 384
-      - name: top-passwords-shortlist.txt
-        start_offset: 394
-        end_offset: 403
-
+    - name: top-passwords-shortlist.txt
+      start_offset: 9
+      end_offset: 17
+    - name: top-passwords-shortlist.txt
+      start_offset: 27
+      end_offset: 32
+    - name: top-passwords-shortlist.txt
+      start_offset: 42
+      end_offset: 51
+    - name: top-passwords-shortlist.txt
+      start_offset: 61
+      end_offset: 68
+    - name: top-passwords-shortlist.txt
+      start_offset: 78
+      end_offset: 83
+    - name: top-passwords-shortlist.txt
+      start_offset: 93
+      end_offset: 99
+    - name: top-passwords-shortlist.txt
+      start_offset: 109
+      end_offset: 116
+    - name: top-passwords-shortlist.txt
+      start_offset: 126
+      end_offset: 132
+    - name: top-passwords-shortlist.txt
+      start_offset: 142
+      end_offset: 145
+    - name: top-passwords-shortlist.txt
+      start_offset: 155
+      end_offset: 159
+    - name: top-passwords-shortlist.txt
+      start_offset: 169
+      end_offset: 175
+    - name: top-passwords-shortlist.txt
+      start_offset: 185
+      end_offset: 191
+    - name: top-passwords-shortlist.txt
+      start_offset: 201
+      end_offset: 208
+    - name: top-passwords-shortlist.txt
+      start_offset: 218
+      end_offset: 224
+    - name: top-passwords-shortlist.txt
+      start_offset: 234
+      end_offset: 241
+    - name: top-passwords-shortlist.txt
+      start_offset: 251
+      end_offset: 261
+    - name: top-passwords-shortlist.txt
+      start_offset: 271
+      end_offset: 277
+    - name: top-passwords-shortlist.txt
+      start_offset: 287
+      end_offset: 295
+    - name: top-passwords-shortlist.txt
+      start_offset: 305
+      end_offset: 313
+    - name: top-passwords-shortlist.txt
+      start_offset: 323
+      end_offset: 331
+    - name: top-passwords-shortlist.txt
+      start_offset: 341
+      end_offset: 347
+    - name: top-passwords-shortlist.txt
+      start_offset: 357
+      end_offset: 364
+    - name: top-passwords-shortlist.txt
+      start_offset: 374
+      end_offset: 380
+    - name: top-passwords-shortlist.txt
+      start_offset: 390
+      end_offset: 396
+    - name: top-passwords-shortlist.txt
+      start_offset: 406
+      end_offset: 413
+    - name: top-passwords-shortlist.txt
+      start_offset: 423
+      end_offset: 428
+    - name: top-passwords-shortlist.txt
+      start_offset: 438
+      end_offset: 445
+    - name: top-passwords-shortlist.txt
+      start_offset: 455
+      end_offset: 460
+    - name: top-passwords-shortlist.txt
+      start_offset: 470
+      end_offset: 477
+    - name: top-passwords-shortlist.txt
+      start_offset: 487
+      end_offset: 495
+    - name: top-passwords-shortlist.txt
+      start_offset: 505
+      end_offset: 511
+    - name: top-passwords-shortlist.txt
+      start_offset: 521
+      end_offset: 527
+    - name: top-passwords-shortlist.txt
+      start_offset: 537
+      end_offset: 543
+    - name: top-passwords-shortlist.txt
+      start_offset: 553
+      end_offset: 559
+    - name: top-passwords-shortlist.txt
+      start_offset: 569
+      end_offset: 573
+    - name: top-passwords-shortlist.txt
+      start_offset: 583
+      end_offset: 589

common/top-passwords-shortlist.txt

@@ -0,0 +1,36 @@
+password=p@55w0rd
+password=adm!n
+password=t3mporary
+password=Am3r1cA
+password=indi@
+password=mumb4i
+password=1234567
+password=abcdef
+password=123
+password=1qaz
+password=qwerty
+password=m0nkey
+password=letmein
+password=drag0n
+password=0000000
+password=bA$k3tb4ll
+password=s0cc3r
+password=iloveyou
+password=tru5tn01
+password=Sun5h1ne
+password=m45ter
+password=w3lcome
+password=shad0w
+password=ashl3y
+password=f00tb@l
+password=j35us
+password=m1chAel
+password=n1nja
+password=mu5tang
+password=chrysler
+password=t0yot4
+password=w1nt3r
+password=spr1ng
+password=summ3r
+password=f@ll
+password=4utumn

configs/README.md

@@ -20,14 +20,14 @@ _version: v0.1_
 
 - Supports quoted passwords
 
-- Not case sensative
+- Case insensitive
 
 
 <details>
 <summary>Pattern Format</summary>
 
 ```regex
-[^\r\n\p{Cc}]+
+[^\r\n\x00-\x08]+
 ```
 
 </details>
@@ -36,7 +36,7 @@ _version: v0.1_
 <summary>Start Pattern</summary>
 
 ```regex
-(?:[^0-9A-Za-z]|\A)(?i)(?:postgres|mysql|mysql_root)_password[\t ]*[=:][\t ]*['"]
+(?:[^0-9A-Za-z]|\A)(?i)(?:postgres|mysql|mysql_root)_password[\t ]*[=:][\t ]*['"]?
 ```
 
 </details><details>
@@ -61,7 +61,7 @@ _version: v0.1_
 <summary>Pattern Format</summary>
 
 ```regex
-[^\r\n'"\p{Cc}]+
+[^\r\n'"\x00-\x08]+
 ```
 
 </details>
@@ -70,7 +70,7 @@ _version: v0.1_
 <summary>Start Pattern</summary>
 
 ```regex
-(?:spring\.datasource|jdbc)\.password[ \t]*=[ \t]*['"]?
+(\A|\b)(?:spring\.datasource|jdbc)\.password[ \t]*=[ \t]*['"]?
 ```
 
 </details><details>
@@ -88,11 +88,7 @@ _version: v0.1_
 
 _version: v0.1_
 
-**Comments / Notes:**
-
 
-- _If the secret is at the start of the file, its not picked up_
-  
 
 <details>
 <summary>Pattern Format</summary>
@@ -107,7 +103,7 @@ _version: v0.1_
 <summary>Start Pattern</summary>
 
 ```regex
-\bSECRET_KEY[ \t]*=[ \t]*["']
+(\b|\A)SECRET_KEY[ \t]*=[ \t]*["']
 ```
 
 </details><details>
@@ -204,13 +200,13 @@ _version: v0.1_
 **Comments / Notes:**
 
 
-- Checks for all github action susing a version that isn't a pinned SHA-1 commit hash
+- Checks for all github actions using a version that isn't a pinned SHA-1 commit hash
 
 - Checks for uses: org name / repo name @ string under 40 characters
 
-- Not case sensative
+- Not case sensitive
 
-- exclude all actions in actions, github and advanced-security repo
+- Exclude all actions in actions, github and advanced-security repo
 
 
 <details>