Skip to content

Checkov report parsing enhanced #12398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: dev
Choose a base branch
from

Conversation

shodanwashere
Copy link

@shodanwashere shodanwashere commented May 7, 2025

Description

Implements the suggestions on this issue: #12384 and then some.
To clarify, findings imported from Checkov reports get additional info injected straight from the Palo Alto guidelines documentation.

imagem

Findings now come with a more descriptive description (hehe) on their Description field (hehe), and additional fix suggestions and benchmark guidelines on the Mitigation field.

Test results

Checking docker compose version
Supported docker compose version
Running docker compose unit tests with test case unittests.tools.test_checkov_parser.TestCheckovParser ...
[07/May/2025 13:56:32] INFO [dojo.models:4628] enabling audit logging
Found 6 test(s).
Using existing test database for alias 'default' ('test_defectdojo')...
Operations to perform:
  Synchronize unmigrated apps: dbbackup, django_filters, drf_spectacular, drf_spectacular_sidecar, fontawesomefree, humanize, imagekit, messages, multiselectfield, polymorphic, rest_framework, staticfiles, tagulous
  Apply all migrations: admin, auditlog, auth, authtoken, contenttypes, django_celery_results, dojo, sessions, sites, social_django, tagging, watson
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying dojo.0001_squashed_0090_index_duplicate_finding...Database vendor: postgresql
 OK
  Applying dojo.0091_npm_audit_path_censoring...[07/May/2025 13:56:42] INFO [dojo.db_migrations.0091_npm_audit_path_censoring:20] Removing random hashes from npm audit file_paths
[07/May/2025 13:56:42] INFO [dojo.utils:2239] 0 out of 0 models processed ...
[07/May/2025 13:56:42] INFO [dojo.utils:2244] 0 out of 0 models processed ...
 OK
  Applying dojo.0092_is_mitigated... OK
  Applying dojo.0093_django_tagging_removal... OK
  Applying dojo.0094_remove_system_settings_s_finding_severity_naming... OK
  Applying dojo.0095_remove_old_product_contact_fields... OK
  Applying dojo.0096_grype_name_change... OK
  Applying dojo.0097_engagement_type... OK
  Applying dojo.0098_anchore_vuln_id... OK
  Applying dojo.0099_delete_report... OK
  Applying dojo.0100_dojo_user_for_authv2... OK
  Applying dojo.0101_enable_features... OK
  Applying dojo.0102_dojo_group... OK
  Applying dojo.0103_report_notification... OK
  Applying dojo.0104_endpoint_userinfo_creation... OK
  Applying dojo.0105_endpoint_host_migration...[07/May/2025 13:56:43] INFO [dojo.endpoint.utils:192] There is not broken endpoint.
 OK
  Applying dojo.0106_role_model...Installed 5 object(s) from 1 fixture(s)
 OK
  Applying dojo.0107_global_role... OK
  Applying dojo.0108_blank_fields... OK
  Applying dojo.0109_group_user_role... OK
  Applying dojo.0110_auth_v2_migrate_user_roles... OK
  Applying dojo.0111_group_user_rename... OK
  Applying dojo.0112_group_user_rename_2... OK
  Applying dojo.0113_endpoint_protocol... OK
  Applying dojo.0114_cyclonedx_vuln_uniqu... OK
  Applying dojo.0115_language_types...Installed 462 object(s) from 1 fixture(s)
 OK
  Applying dojo.0116_test_type_active... OK
  Applying dojo.0117_usercontactinfo_force_password_reset... OK
  Applying dojo.0118_remove_finding_images... OK
  Applying dojo.0119_default_group_is_staff... OK
  Applying dojo.0120_sonarqube_test_and_clean...[07/May/2025 13:56:45] WARNING [dojo.db_migrations.0120_sonarqube_test_and_clean:23] No SonarQube tool configuration found, all invalid SonarQube configurations will be removed.
 OK
  Applying dojo.0121_user_restrict... OK
  Applying dojo.0122_cobaltio_product... OK
  Applying dojo.0123_scan_type... OK
  Applying dojo.0124_sonarqube_api_type_length_change... OK
  Applying dojo.0125_sonarqube_clean... OK
  Applying dojo.0126_finding_publish_date... OK
  Applying dojo.0127_remove_hashes...[07/May/2025 13:56:46] INFO [dojo.db_migrations.0127_remove_hashes:15] Content type for auth / user does not exist
[07/May/2025 13:56:46] INFO [dojo.db_migrations.0127_remove_hashes:19] Content type for dojo / cred_user does not exist
 OK
  Applying dojo.0128_pytz_update... OK
  Applying dojo.0129_finding_deprecated_fields... OK
  Applying dojo.0130_product_api_scan_configuration... OK
  Applying dojo.0131_migrate_sonarcube_cobalt... OK
  Applying dojo.0132_remove_configs_from_test... OK
  Applying dojo.0133_finding_service... OK
  Applying dojo.0134_sonarque_cobaltio_removal... OK
  Applying dojo.0135_email_from... OK
  Applying dojo.0136_default_group_help_text... OK
  Applying dojo.0137_system_settings_enable_endpoint_metadata_import... OK
  Applying dojo.0138_remove_authorized_users... OK
  Applying dojo.0139_google_sheets_rules_framework_enable... OK
  Applying dojo.0140_auth_group... OK
  Applying dojo.0141_enable_user_profile_editable... OK
  Applying dojo.0142_environment_delete... OK
  Applying dojo.0143_objects... OK
  Applying dojo.0144_import_action_untouched... OK
  Applying dojo.0145_system_settings_default_group_email_pattern... OK
  Applying dojo.0146_lead_optional... OK
  Applying dojo.0147_rename_sslyze_parser... OK
  Applying dojo.0148_default_notifications... OK
  Applying dojo.0149_harmonize_user_format... OK
  Applying dojo.0150_dedupe_endpoint_status...[07/May/2025 13:56:48] INFO [dojo.db_migrations.0150_dedupe_endpoint_status:22] There is nothing to process
 OK
  Applying dojo.0151_index_endpoint_status... OK
  Applying dojo.0152_notifications_template... OK
  Applying dojo.0153_migrate_endpoint_mitigated...[07/May/2025 13:56:48] INFO [dojo.db_migrations.0153_migrate_endpoint_mitigated:20] There is nothing to process
 OK
  Applying dojo.0154_remove_endpoint_mitigated... OK
  Applying dojo.0155_enable_finding_groups... OK
  Applying dojo.0156_migrate_finding_groups_setting...[07/May/2025 13:56:48] INFO [dojo.db_migrations.0156_migrate_finding_groups_setting:14] Migrating value from FEATURE_FINDING_GROUPS into system settings model
 OK
  Applying dojo.0157_vulnerability_reference... OK
  Applying dojo.0158_vulnerability_id... OK
  Applying dojo.0159_remove_broken_endpoint_statuses...[07/May/2025 13:56:48] INFO [dojo.endpoint.utils:362] There is no broken endpoint_status
 OK
  Applying dojo.0160_set_notnull_endpoint_statuses... OK
  Applying dojo.0161_alter_dojo_group_social_provider... OK
  Applying dojo.0162_created_and_updated... OK
  Applying dojo.0163_system_settings_enable_calendar... OK
  Applying dojo.0164_remove_system_settings_staff_user_email_pattern... OK
  Applying dojo.0165_custom_sla... OK
  Applying dojo.0166_copy_sla_from_system_settings... OK
  Applying dojo.0167_system_settings_add_vulnerability_id_to_jira_label... OK
  Applying dojo.0168_alter_system_settings_time_zone... OK
  Applying dojo.0169_planned_remediation_date... OK
  Applying dojo.0170_jira_project_custom_fields... OK
  Applying dojo.0171_jira_labels_per_product_and_engagement... OK
  Applying dojo.0172_optimize_usage_of_endpoint_status... OK
  Applying dojo.0173_alter_risk_acceptance_name... OK
  Applying dojo.0174_jira_project_default_assignee... OK
  Applying dojo.0175_system_settings_enable_notify_sla...[07/May/2025 13:56:50] INFO [dojo.db_migrations.0175_system_settings_enable_notify_sla:14] Migrating value from SLA_NOTIFY_ACTIVE into system settings model
[07/May/2025 13:56:50] INFO [dojo.db_migrations.0175_system_settings_enable_notify_sla:25] Migrating value from SLA_NOTIFY_ACTIVE_VERIFIED_ONLY into system settings model
[07/May/2025 13:56:50] INFO [dojo.db_migrations.0175_system_settings_enable_notify_sla:36] Migrating value from SLA_NOTIFY_WITH_JIRA_ONLY into system settings model
 OK
  Applying dojo.0176_custom_password_requirements... OK
  Applying dojo.0177_alter_system_settings_time_zone... OK
  Applying dojo.0178_alter_answer_polymorphic_ctype_and_more... OK
  Applying dojo.0179_alter_finding_verified... OK
  Applying dojo.0180_announcement_userannouncement... OK
  Applying dojo.0181_jira_instance_finding_jira_sync... OK
  Applying dojo.0182_alter_jira_instance_default_issue_type... OK
  Applying dojo.0183_system_settings_enable_notify_sla_exponential_backoff_and_more... OK
  Applying dojo.0184_remove_child_rule_parent_rule_delete_fieldrule_and_more... OK
  Applying dojo.0185_product_disable_sla_breach_notifications_and_more... OK
  Applying dojo.0186_system_settings_non_common_password_required... OK
  Applying dojo.0187_nessus_to_tenable...[07/May/2025 13:56:50] WARNING [dojo.db_migrations.0187_nessus_to_tenable:42] We identified 0 Nessus/NessusWAS findings to migrate to Tenable findings
 OK
  Applying dojo.0188_product_enable_product_tag_inheritance_and_more... OK
  Applying dojo.0189_finding_effort_and_remediation_for_fixing... OK
  Applying dojo.0190_system_settings_experimental_fp_history... OK
  Applying dojo.0191_alter_notifications_risk_acceptance_expiration... OK
  Applying dojo.0192_notifications_scan_added_empty... OK
  Applying dojo.0193_remove_system_settings_enable_auditlog... OK
  Applying dojo.0194_alter_finding_component_name... OK
  Applying dojo.0195_alter_announcement_dismissable... OK
  Applying dojo.0196_notifications_sla_breach_combined... OK
  Applying dojo.0197_parser_merge...[07/May/2025 13:56:51] WARNING [dojo.db_migrations.0197_parser_merge:62] We identified 0 OpenVAS CSV/ OpenVAS XML findings to migrate to OpenVAS Parser findings
[07/May/2025 13:56:51] WARNING [dojo.db_migrations.0197_parser_merge:80] We identified 0 Clair Klar Scan findings to migrate to Clair Scan findings
 OK
  Applying dojo.0198_alter_system_settings_enable_deduplication... OK
  Applying dojo.0199_whitesource_to_mend...[07/May/2025 13:56:51] WARNING [dojo.db_migrations.0199_whitesource_to_mend:41] We identified 0 Whitesource findings to migrate to Mend findings
 OK
  Applying dojo.0200_finding_sla_expiration_date_product_async_updating_and_more... OK
  Applying dojo.0201_populate_finding_sla_expiration_date...[07/May/2025 13:56:51] INFO [dojo.db_migrations.0201_populate_finding_sla_expiration_date:18] Calculating SLA expiration dates for all findings
[07/May/2025 13:56:51] INFO [dojo.db_migrations.0201_populate_finding_sla_expiration_date:27] Found 0 findings to be updated
[07/May/2025 13:56:51] INFO [dojo.db_migrations.0201_populate_finding_sla_expiration_date:74] 0 out of 0 findings processed...
 OK
  Applying dojo.0202_alter_dojo_group_social_provider... OK
  Applying dojo.0203_alter_finding_options_finding_epss_percentile_and_more... OK
  Applying dojo.0204_alter_finding_cvssv3_score... OK
  Applying dojo.0205_jira_project_epic_issue_type_name... OK
  Applying dojo.0206_system_settings_api_expose_error_details... OK
  Applying dojo.0207_alter_sonarqube_issue_key... OK
  Applying dojo.0208_merge_acunetix...[07/May/2025 13:56:51] WARNING [dojo.db_migrations.0208_merge_acunetix:38] We identified 0 Acunetix360 Scan findings to migrate to Acunetix Scan findings
 OK
  Applying dojo.0209_alter_finding_severity... OK
  Applying dojo.0210_system_settings_filter_string_matching... OK
  Applying dojo.0211_system_settings_enable_similar_findings... OK
  Applying dojo.0212_sla_configuration_enforce_critical_and_more... OK
  Applying dojo.0213_system_settings_enable_ui_table_based_searching... OK
  Applying dojo.0214_test_type_dynamically_generated... OK
  Applying dojo.0215_webhooks_notifications... OK
  Applying dojo.0216_alter_jira_project_push_all_issues... OK
  Applying dojo.0217_jira_project_enabled... OK
  Applying dojo.0218_system_settings_enforce_verified_status_and_more... OK
  Applying dojo.0219_system_settings_enforce_verified_status_jira_and_more... OK
  Applying dojo.0220_system_settings_disclaimer_notif... OK
  Applying dojo.0221_system_settings_disclaimer_notif... OK
  Applying dojo.0222_clean_old_sessions... OK
  Applying dojo.0223_aws_sechub_update_endpoints... OK
  Applying dojo.0224_alter_regulation_category... OK
  Applying dojo.0225_alter_product_revenue... OK
  Applying dojo.0226_import_history_left_untouched_rename... OK
  Applying dojo.0227_migrate_tags... OK
  Applying dojo.0228_alter_jira_username_password... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying social_django.0001_initial... OK
  Applying social_django.0002_add_related_name... OK
  Applying social_django.0003_alter_email_max_length... OK
  Applying social_django.0004_auto_20160423_0400... OK
  Applying social_django.0005_auto_20160727_2333... OK
  Applying social_django.0006_partial... OK
  Applying social_django.0007_code_timestamp... OK
  Applying social_django.0008_partial_timestamp... OK
  Applying social_django.0009_auto_20191118_0520... OK
  Applying social_django.0010_uid_db_index... OK
  Applying social_django.0011_alter_id_fields... OK
  Applying social_django.0012_usersocialauth_extra_data_new... OK
  Applying social_django.0013_migrate_extra_data... OK
  Applying social_django.0014_remove_usersocialauth_extra_data... OK
  Applying social_django.0015_rename_extra_data_new_usersocialauth_extra_data... OK
  Applying social_django.0016_alter_usersocialauth_extra_data... OK
  Applying tagging.0001_initial... OK
  Applying tagging.0002_on_delete... OK
  Applying tagging.0003_adapt_max_tag_length... OK
  Applying watson.0001_initial... OK
  Applying watson.0002_alter_searchentry_object_id... OK
System check identified no issues (0 silenced).
test_parse_file_with_multiple_check_type_has_multiple_check_type (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_multiple_check_type_has_multiple_check_type) ... ERROR
test_parse_file_with_multiple_vuln_has_multiple_findings (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_multiple_vuln_has_multiple_findings) ... ERROR
test_parse_file_with_no_vuln_has_no_findings (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_no_vuln_has_no_findings) ... ok
test_parse_file_with_no_vuln_has_no_findings_v2 (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_no_vuln_has_no_findings_v2) ... ok
test_parse_file_with_one_vuln_has_one_finding (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_one_vuln_has_one_finding) ... ERROR
test_parse_file_with_specified_severity (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_specified_severity) ... ERROR

======================================================================
ERROR: test_parse_file_with_multiple_check_type_has_multiple_check_type (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_multiple_check_type_has_multiple_check_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 974, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0xffffa16d6f90>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/general_4.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa16d6f90>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/unittests/tools/test_checkov_parser.py", line 35, in test_parse_file_with_multiple_check_type_has_multiple_check_type
    findings = parser.get_findings(testfile, Test())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 70, in get_findings
    findings += self.get_items(tree, test, check_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 106, in get_items
    item = get_item(node, test, check_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 126, in get_item
    plain_req = requests.get(plain_url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/general_4.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa16d6f90>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

======================================================================
ERROR: test_parse_file_with_multiple_vuln_has_multiple_findings (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_multiple_vuln_has_multiple_findings)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 974, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0xffffa1d1d750>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/ensure-the-key-vault-is-recoverable.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa1d1d750>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/unittests/tools/test_checkov_parser.py", line 29, in test_parse_file_with_multiple_vuln_has_multiple_findings
    findings = parser.get_findings(testfile, Test())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 70, in get_findings
    findings += self.get_items(tree, test, check_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 106, in get_items
    item = get_item(node, test, check_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 126, in get_item
    plain_req = requests.get(plain_url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/ensure-the-key-vault-is-recoverable.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa1d1d750>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

======================================================================
ERROR: test_parse_file_with_one_vuln_has_one_finding (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_one_vuln_has_one_finding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 974, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0xffffa1d5a450>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/ensure-the-key-vault-is-recoverable.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa1d5a450>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/unittests/tools/test_checkov_parser.py", line 23, in test_parse_file_with_one_vuln_has_one_finding
    findings = parser.get_findings(testfile, Test())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 70, in get_findings
    findings += self.get_items(tree, test, check_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 106, in get_items
    item = get_item(node, test, check_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 126, in get_item
    plain_req = requests.get(plain_url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/ensure-the-key-vault-is-recoverable.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa1d5a450>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

======================================================================
ERROR: test_parse_file_with_specified_severity (unittests.tools.test_checkov_parser.TestCheckovParser.test_parse_file_with_specified_severity)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 974, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0xffffa08f9a50>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/bc_gcp_gcs_2.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa08f9a50>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/unittests/tools/test_checkov_parser.py", line 85, in test_parse_file_with_specified_severity
    findings = parser.get_findings(testfile, Test())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 70, in get_findings
    findings += self.get_items(tree, test, check_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 106, in get_items
    item = get_item(node, test, check_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/checkov/parser.py", line 126, in get_item
    plain_req = requests.get(plain_url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.bridgecrew.io', port=443): Max retries exceeded with url: /docs/bc_gcp_gcs_2.plain.html (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa08f9a50>: Failed to resolve 'docs.bridgecrew.io' ([Errno -2] Name or service not known)"))

----------------------------------------------------------------------
Ran 6 tests in 0.109s

FAILED (errors=4)
Preserving test database for alias 'default' ('test_defectdojo')...
exit status 1

My theory is these tests fail because they use scan files generated from back when Checkov was still owned by Bridgecrew, whose endpoints have since gone offline. Now findings use documentation hosted by Palo Alto Networks.

Checklist

This checklist is for your information.

  • Make sure to rebase your PR against the very latest dev.
  • Features/Changes should be submitted against the dev.
  • Bugfixes should be submitted against the bugfix branch.
  • Give a meaningful name to your PR, as it may end up being used in the release notes.
  • Your code is flake8 compliant.
  • If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
  • Your code is python 3.11 compliant.
  • Model changes must include the necessary migrations in the dojo/db_migrations folder.
  • Add applicable tests to the unit tests.
  • Add the proper label to categorize your PR.

Copy link

dryrunsecurity bot commented May 7, 2025

DryRun Security

This pull request contains a low-confidence potential information disclosure risk in the Checkov parser where adding a description field might inadvertently expose sensitive implementation details from an untrusted source.

💭 Unconfirmed Findings (1)
Vulnerability Potential Information Disclosure Risk
Description Located in dojo/tools/checkov/parser.py, this risk involves adding a 'description' field that could expose sensitive implementation details or internal system information by including vulnerability descriptions from an untrusted source.

All finding details can be found in the DryRun Security Dashboard.

@shodanwashere
Copy link
Author

i'm not expecting this PR to be merged immediately, in fact it shouldn't: the unit tests for Checkov need to be updated as the scan files being used are now quite outdated.

@shodanwashere shodanwashere changed the title Checkov info enhancement Checkov report parsing enhanced May 7, 2025
@shodanwashere
Copy link
Author

Vulnerability External Web Request without Error Handling
Description Located in dojo/tools/checkov/parser.py, this finding indicates a potential security risk with requests.get() call that lacks proper error handling or timeout mechanisms, which could lead to request failures, performance issues, or unhandled network-related exceptions.

Warning

requests.get() is used to obtain documentation directly from Palo Alto Networks referring to the findings obtained on the report. However, as stated by Dry Run, we can shut down this PR temporarily to apply mitigations to improper error handling and timeouts.

Vulnerability Potential Information Disclosure via Web Scraping
Description Found in dojo/tools/checkov/parser.py, this issue involves using BeautifulSoup to parse external HTML, which could expose the code to risks from external website HTML structure changes and potentially leak unintended information.

Note

BeautifulSoup is being used specifically to obtain snippets of the documentation for different fields by parsing the plain HTML content.

Vulnerability Potential Dependency Pinning Security Consideration
Description Located in requirements.txt, this finding highlights a potential security risk with the beautifulsoup4 library version 4.13.4, recommending verification that this specific library version does not contain known security vulnerabilities.

Caution

I included the latest version of this library on this PR, but I do agree, I should have checked to see if Beautiful Soup has any vulnerabilities that could open up Defect Dojo specifically with Checkov reports.

@mwager
Copy link
Contributor

mwager commented May 8, 2025

Nice PR!

My notes:

requests.get misses error handling if I see correctly.

But, not sure if the maintainers want (or if if should be in general) to have this kind of logic in a parser... needs to be discussed. Could also be done before importing to DD.

@shodanwashere
Copy link
Author

requests.get misses error handling if I see correctly.

Was taking a quick gander and maybe, we could have it try to requests.get from the guideline documentation, and if it fails, just don't try to include it in the finding?

@valentijnscholten
Copy link
Member

valentijnscholten commented May 8, 2025

@shodanwashere Thanks a lot for the PR, really happy with contributions. After consideration the team have decided not to accept the PR as-is. Experience has learned us that reaching out to external systems to enhance findings during import has certain risks and maintenance costs that we're unable to support currently.

We have a GitHub repository with community contributions: https://github.com/DefectDojo/Community-Contribs where useful scripts are shared with the community. We would welcome a checkov-enhancer.sh/py/go/rb/... script that performs the enhancement currently implemented in the PR. Fans of the enhancement could use the script to enhance the Checkov report before uploading it to Defect Dojo.

Another approach could be to suggest to Prismacloud to add more text to the report instead of just a link to their website. The benefit of the link is that the website will always have the latest and greatest information.

In issue #12384 it was suggested to include the related benchmark items from the report to the finding in Defect Dojo. That would be a useful and welcome addition to have a PR for.

@shodanwashere
Copy link
Author

just pushed a number of commits based on the obtained feedback. I'll be maintaining the benchmark references in the findings as mitigation info and will also be future proofing to include the description if it's in the original finding. I'm currently in conversations with PrismaCloud Support to understand why Checkov is omitting information for findings, so we'll be able to make this PR more robust in the future.

Copy link
Member

@valentijnscholten valentijnscholten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @shodanwashere . Could you look at the test/linting results?

Specifically, this fixes Q000, F821 and F541 identified previously on
lines 126, 128 and 130
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants