Skip to content

Commit 92c48e3

Browse files
pulorsokpulorsok
authored
Add showcase of Quark Script Generate using Quark Script Agent (#39)
1 parent a5aaf04 commit 92c48e3

File tree

3 files changed

+296
-22
lines changed

3 files changed

+296
-22
lines changed

‎README.md

Lines changed: 178 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,20 +19,25 @@ git clone https://github.com/quark-engine/quark-script.git
1919
pip install -r requirements.txt
2020
```
2121

22-
3. Run the script:
22+
3. Add your OpenAI API key in quarkscriptAgent.py.
23+
```python
24+
os.environ["OPENAI_API_KEY"] = {your API Key}
25+
```
26+
27+
4. Run the script:
2328
```
2429
python quarkScriptAgent.py
2530
```
2631

27-
4. Result:
32+
5. Result:
2833

29-
<img width="1440" alt="截圖 2024-07-26 下午3 39 12" src="https://github.com/user-attachments/assets/9c8ba9d3-c8b5-4583-8cb8-750f8c3bf2a7">
34+
<img width="1440" alt="截圖 2024-07-26 下午3 39 12" src="https://hackmd.io/_uploads/SJZZpf9FR.png">
3035

3136
### Decode the Prompts
3237
Here are two prompts, each for executing different analysis processes.
3338

3439
```
35-
1st Prompt: Initialize the rule instance with the rule path set to "rule.json"
40+
1st Prompt: Initialize the rule instance with the rule path set to "constructCryptoGraphicKey.json"
3641
```
3742
Used Quark Script APIs/Tools that LLM used: `loadRule`
3843

@@ -44,10 +49,175 @@ Used Quark Script APIs/Tools that LLM used: `runQuarkAnalysis`, `getBehaviorOccu
4449

4550
The `loadRule`, `runQuarkAnalysis`, `getBehaviorOccurList`, `getParameterValues`, and `isHardCoded` functions are treated as **tools** within LangChain, enabling them to be invoked through the `gpt-4o` model to analyze and identify [CWE-798](https://cwe.mitre.org/data/definitions/798.html) vulnerabilities in the [ovaa.apk](https://github.com/oversecured/ovaa) sample.
4651

47-
<img width="845" alt="截圖 2024-07-27 下午8 45 25" src="https://github.com/user-attachments/assets/1dd8fb68-9ab4-4afc-a15a-006ff468a883">
52+
<img width="845" alt="截圖 2024-07-27 下午8 45 25" src="https://hackmd.io/_uploads/BkplMjvYR.png">
53+
54+
55+
## Showcase: Generating Quark Script with Quark Script Agent
56+
57+
Quark Script Agent allows you to automate the creation of Quark Script code. This lets users concentrate on designing the detection process, while the LLM and Quark Script API handle the coding. This approach greatly enhances efficiency and streamlines the workflow.
58+
59+
Here's an example for generating a Quark Script to detect CWE-798.
60+
61+
### Demo Video
62+
63+
https://github.com/user-attachments/assets/dc0e782b-3500-4260-a961-c499c14e495c
64+
65+
### Decode the Prompts
66+
67+
Now, let's instruct the Quark Script Agent to detect CWE-798 and write the Quark Script for the detection process using the following prompt.
68+
69+
#### Assign a role to the Agent
70+
71+
To make the Quark Script Agent's responses more precise and relevant to vulnerability detection, we assign it the role of an Android APK vulnerability analyst.
72+
73+
```!
74+
User:
75+
Now you are an Android APK vulnerability analyst. Please follow the steps below to analyze a sample.
76+
```
77+
78+
```!
79+
Agent:
80+
Sure, I'd be happy to help with that. Please provide the sample APK file and the detection rule file that you would like to use for the analysis.
81+
```
82+
83+
#### Instruct the detection process
84+
85+
Instruct the Quark Script Agent to detect CWE-798 using the following detection process.
86+
87+
```!
88+
Detection Process:
89+
1. Define the behavior "Construct Cryptographic Key" in the rule instance.
90+
2. Run Quark Analysis using the rule instance on the APK sample.
91+
3. Obtain all instances of detected behaviors from the Quark Analysis results.
92+
4. Retrieve the parameter values for each occurred behavior.
93+
5. Check if any parameters are hard-coded. If found, print the hard-coded parameters.
94+
```
95+
96+
>Note: The prompt will include details like the sample or rule path.
97+
98+
99+
1. Define the behavior "Construct Cryptographic Key" in the rule instance.
100+
```!
101+
User:
102+
1. Define the behavior "Construct Cryptographic Key" in the rule instance, with the rule path set to "constructCryptoGraphicKey.json".
103+
```
104+
105+
```!
106+
Agent:
107+
The behavior "Construct Cryptographic Key" has been successfully defined in the rule instance with the rule path set to "constructCryptoGraphicKey.json". How would you like to proceed?
108+
```
109+
110+
2. Run Quark Analysis using the rule instance on the APK sample.
111+
```!
112+
User:
113+
2. Run Quark Analysis using the rule instance on the APK sample "ovaa.apk".
114+
```
115+
```!
116+
Agent:
117+
The Quark analysis on the APK sample "ovaa.apk" has been completed successfully. What would you like to do next?
118+
```
119+
120+
3. Obtain all instances of detected behaviors from the Quark Analysis results.
121+
```!
122+
User:
123+
3. Obtain all instances of detected behaviors from the Quark Analysis results in the sample "ovaa.apk".
124+
```
125+
126+
```!
127+
Agent
128+
I have obtained all instances of detected behaviors from the Quark Analysis results in the sample "ovaa.apk". If you need further details or specific information about these behaviors, please let me know!
129+
```
130+
131+
4. Retrieve the parameter values for each occurred behavior.
132+
```!
133+
User:
134+
4. Retrieve the parameter values for each occurred behavior.
135+
```
136+
137+
```!
138+
Agent:
139+
The parameter values for the occurred behavior are:
140+
1. "49u5gh249gh24985ghf429gh4ch8f23f"
141+
2. "AES"
142+
143+
Would you like to check if any of these values are hard-coded?
144+
```
145+
146+
5. Check if any parameters are hard-coded. If found, print the hard-coded parameters.
147+
```!
148+
User:
149+
5. Check if any parameter values are hard-coded. If yes, display the hard-coded values.
150+
```
151+
152+
```!
153+
Agent:
154+
The following parameter values are hard-coded:
155+
156+
1. `49u5gh249gh24985ghf429gh4ch8f23f`
157+
2. `AES`
158+
```
159+
160+
#### Generate Quark Script
161+
Finally, ask the Quark Script Agent to write and execute the Python code based on the detection process outlined above.
162+
163+
```!
164+
User:
165+
Using the Quark Script API, implement the above logic in Python and save it to 'detectCWE798.py', with the rule path set to 'constructCryptoGraphicKey.json' and the sample path to 'ovaa.apk'.
166+
```
167+
168+
```!
169+
Agent:
170+
The code has been written to `detectCWE798.py`. You can now execute this script to perform the analysis.
171+
```
172+
173+
Here is the generated code:
174+
```python
175+
from quark.script import Rule, runQuarkAnalysis
176+
177+
# Load the rule
178+
rule_path = 'constructCryptoGraphicKey.json'
179+
rule_instance = Rule(rule_path)
180+
181+
# Define the sample path
182+
sample_path = 'ovaa.apk'
183+
184+
# Run the Quark analysis
185+
quark_result = runQuarkAnalysis(sample_path, rule_instance)
186+
187+
# Extract the behavior occurrence list
188+
behavior_occur_list = quark_result.behaviorOccurList
189+
190+
# Iterate through each behavior and get parameter values
191+
for behavior in behavior_occur_list:
192+
param_values = behavior.getParamValues()
193+
for param in param_values:
194+
# Check if the parameter value is hardcoded
195+
is_hardcoded = quark_result.isHardcoded(param)
196+
if is_hardcoded:
197+
print(f'Hardcoded value found: {param}')
198+
```
199+
200+
```!
201+
User:
202+
Execute detectCWE798.py
203+
```
204+
205+
Here is the result.
206+
```shell
207+
Hardcoded value found: 49u5gh249gh24985ghf429gh4ch8f23f
208+
Hardcoded value found: AES
209+
```
210+
211+
We added 2 tools, `writeCodeInFile` and `executeCode`, to enable Quark Script Agent to generate and execute Quark Script code.
212+
213+
###
214+
<img width="845" alt="截圖 2024-07-27 下午8 45 25" src="https://github.com/user-attachments/assets/d8fd805a-86c9-4eff-b120-d340fc43d792">
48215

49216

50217
* Notes:
51-
1. Since LangChain currently does not support passing Python instances between tools, we are temporarily using global variables to pass parameters between tools in `quarkScriptAgent.py`.
52-
2. Place the rules, samples, and `quarkScriptAgent.py` in the same folder; the LLM will automatically find files with matching names.
53-
3. A web GUI is under construction, please stay tuned!
218+
1. The code is generated by OpenAI's GPT model, and the output may not always match the documentation exactly.
219+
2. Since LangChain currently does not support passing Python instances between tools, we are temporarily using global variables to pass parameters between tools in `quarkScriptAgent.py`.
220+
3. Place the rules, samples, and `quarkScriptAgent.py` in the same folder; the LLM will automatically find files with matching names.
221+
4. A web GUI is under construction, please stay tuned!
222+
223+

‎rule.json renamed to ‎constructCryptoGraphicKey.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"crime": "Detect APK using SecretKeySpec.",
2+
"crime": "Construct cryptographic key",
33
"permission": [],
44
"api": [
55
{

0 commit comments

Comments
 (0)