Add katana input#7545
Conversation
WalkthroughChangesThe PR adds a Katana JSONL input format that parses crawl records and bare URLs into request responses, registers the format with the HTTP provider, and documents and tests raw-request preservation, request synthesis, tolerant parsing, and callback termination. Katana JSONL input
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant KatanaJSONL
participant KatanaFormat
participant RequestResponse
participant ParseCallback
KatanaJSONL->>KatanaFormat: Provide crawl record or bare URL
KatanaFormat->>RequestResponse: Parse raw request or synthesize and reparse
RequestResponse->>ParseCallback: Emit parsed request response
ParseCallback-->>KatanaFormat: Continue or stop parsing
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
pkg/input/formats/katana/katana_test.go (1)
20-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrefer
require.NoErrorfor idiomatic error checking.For asserting that an error is
nilintestify/require, it is more idiomatic to userequire.NoError. It will automatically format the error message alongside your custom message, matching your usage on line 28.♻️ Proposed refactor
- require.Nilf(t, err, "error opening katana input file: %v", err) + require.NoError(t, err, "error opening katana input file")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/input/formats/katana/katana_test.go` at line 20, Replace the require.Nilf assertion in the Katana input test with require.NoError, preserving the existing custom context message and error value.pkg/input/formats/katana/katana.go (1)
158-172: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDynamically add a
Content-Lengthheader if it's missing.If
body != ""but the original Katana headers do not include aContent-LengthorTransfer-Encodingheader,types.ParseRawRequestWithURL(which relies on standardhttp.ReadRequestlogic) may fail to read the body and silently truncate it. It's safer to synthesize theContent-Lengthdynamically if it wasn't provided in the captured headers.♻️ Proposed refactor to ensure body sizing is present
// Emit headers in a deterministic order, skipping Host (already written). + var hasBodySizing bool for _, key := range sortedHeaderKeys(headers) { if strings.EqualFold(key, "Host") { continue } + if strings.EqualFold(key, "Content-Length") || strings.EqualFold(key, "Transfer-Encoding") { + hasBodySizing = true + } sb.WriteString(key) sb.WriteString(": ") sb.WriteString(headers[key]) sb.WriteString("\r\n") } + if body != "" && !hasBodySizing { + sb.WriteString(fmt.Sprintf("Content-Length: %d\r\n", len(body))) + } sb.WriteString("\r\n") if body != "" { sb.WriteString(body) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/input/formats/katana/katana.go` around lines 158 - 172, Update the raw request serialization around the header-emission loop to add a Content-Length header when body is non-empty and the original headers contain neither Content-Length nor Transfer-Encoding. Compute the length from the serialized body using the appropriate byte length, while preserving existing headers and avoiding duplicate Content-Length values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/input/formats/katana/katana.go`:
- Around line 48-54: Update the Headers field in katanaRequest to use
map[string][]string, matching Katana’s JSONL request.headers shape so valid
entries unmarshal correctly. Keep Body and Raw as plain string fields without
adding base64 handling.
---
Nitpick comments:
In `@pkg/input/formats/katana/katana_test.go`:
- Line 20: Replace the require.Nilf assertion in the Katana input test with
require.NoError, preserving the existing custom context message and error value.
In `@pkg/input/formats/katana/katana.go`:
- Around line 158-172: Update the raw request serialization around the
header-emission loop to add a Content-Length header when body is non-empty and
the original headers contain neither Content-Length nor Transfer-Encoding.
Compute the length from the serialized body using the appropriate byte length,
while preserving existing headers and avoiding duplicate Content-Length values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c2e344b4-276e-47a6-8bc4-565e83572d12
📒 Files selected for processing (5)
pkg/input/formats/README.mdpkg/input/formats/katana/katana.gopkg/input/formats/katana/katana_test.gopkg/input/formats/testdata/katana.jsonlpkg/input/provider/http/multiformat.go
| type katanaRequest struct { | ||
| Method string `json:"method"` | ||
| Endpoint string `json:"endpoint"` | ||
| Body string `json:"body"` | ||
| Headers map[string]string `json:"headers"` | ||
| Raw string `json:"raw"` | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does ProjectDiscovery Katana export headers as map[string]string or map[string][]string in its JSONL output, and does it base64-encode the raw and body fields?
💡 Result:
In Katana's JSONL output, the headers field is represented as a map[string][]string [1]. Regarding the raw and body fields, they are not base64-encoded by default in the standard JSONL output; they are typically represented as plain strings [1][2][3]. However, Katana provides specific flags for handling these fields. While there is an issue referencing an -irrb (-include-response-base64) flag [4], standard usage generally relies on the --omit-raw and --omit-body flags to manage or exclude this data from the output [2][5][6]. The raw and body fields contain the request/response data as captured during the crawl [1][5].
Citations:
- 1: https://docs.projectdiscovery.io/opensource/katana/running
- 2: https://github.com/projectdiscovery/katana/
- 3: https://github.com/projectdiscovery/katana/blob/main/README.md
- 4: adding -irr (-include-response) and -irrb (-include-response-base64) flags look like httpx for saving raw req/res in json output katana#1014
- 5: Customizable json output to reduce file size / Omit raw / body katana#447
- 6: fix(headless): preserve response body/raw and wait after form submission katana#1585
🏁 Script executed:
sed -n '1,180p' pkg/input/formats/katana/katana.goRepository: projectdiscovery/nuclei
Length of output: 5752
🌐 Web query:
ProjectDiscovery Katana JSONL navigation.Request headers map[string][]string raw body plain string source
💡 Result:
ProjectDiscovery Katana provides structured output in JSONL format, which includes request and response details [1][2]. The navigation.Request object (often represented as a "request" field in the JSONL output) contains information about the HTTP request, including headers and the body [3]. Key details regarding your request: 1. Request Headers: These are represented as a map[string][]string within the navigation.Request structure [3]. When exported to JSONL, they appear as a JSON object where each key is a header name and the value is an array of strings [1][4]. 2. Raw Body: The body is typically included as a plain string field [1][2]. 3. Customization: Katana allows you to omit the raw request/response data or the body entirely from the JSONL output using the -or (--omit-raw) and -ob (--omit-body) flags, respectively [2][4]. This is useful for reducing the size of the generated output file [5]. 4. Field Inspection: You can list all available fields that can be included or excluded from the JSONL output by using the -lof (--list-output-fields) flag [6][7][2]. You can subsequently exclude specific fields from the output using the -eof (--exclude-output-fields) flag [7][2]. For detailed usage, you can run katana -lof to see the full schema of available fields [7][8].
Citations:
- 1: https://docs.projectdiscovery.io/opensource/katana/running
- 2: https://github.com/projectdiscovery/katana/blob/main/README.md
- 3: https://pkg.go.dev/github.com/projectdiscovery/katana/pkg/navigation
- 4: https://docs.projectdiscovery.io/opensource/katana/running.md
- 5: Customizable json output to reduce file size / Omit raw / body katana#447
- 6: https://github.com/projectdiscovery/katana
- 7: feat(output): Add
-eofand-lofflags for JSON output filtering katana#1390 - 8: https://www.codeline.co/thoughts/repo-review/2024/katana-next-gen-web-crawler-for-automation-pipelines
pkg/input/formats/katana/katana.go:48-54 — Match katanaRequest.Headers to Katana's JSONL shape
Katana emits request.headers as map[string][]string, so this map[string]string field will fail to unmarshal and skip valid entries. Body and Raw are plain strings; no base64 handling is needed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/input/formats/katana/katana.go` around lines 48 - 54, Update the Headers
field in katanaRequest to use map[string][]string, matching Katana’s JSONL
request.headers shape so valid entries unmarshal correctly. Keep Body and Raw as
plain string fields without adding base64 handling.
1 participant
Parse katana JSONL crawl output as nuclei input for DAST scanning. Closes #7548.