Cannot scan Azure Gov OpenAI GPT-4.1 LLM

[MarChristensen]

Steps to reproduce

How'd you do it?

1. set OPENAI_API_KEY

2. set OPENAI_ENDPOINT

3. garak --model_type openai --model_name gpt-4.1 --probes packagehallucination.Dart

4. ...This section should also tell us any relevant information about the
   environment; for example, if a probe that used to work is failing,
   tell us the target application and possibly model versions.
   When running the garak scan we get this error:
   Traceback (most recent call last):
   File "", line 198, in run_module_as_main
   File "", line 88, in run_code
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\Scripts\garak.exe_main.py", line 7, in
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak_main.py", line 9, in main
   cli.main(sys.argv[1:])
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\cli.py", line 627, in main
   command.probewise_run(
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\command.py", line 259, in probewise_run
   probewise_h.run(generator, probe_names, evaluator, buffs)
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\harnesses\probewise.py", line 109, in run
   super().run(model, [probe], detectors, evaluator, announce_probe=False)
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\harnesses\base.py", line 149, in run
   attempt_results = probe.probe(model)
   ^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\probes\base.py", line 441, in probe
   attempts_completed = self._execute_all(attempts_todo)
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\probes\base.py", line 352, in _execute_all
   result = self._execute_attempt(this_attempt)
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\probes\base.py", line 294, in _execute_attempt
   this_attempt.outputs = self.generator.generate(
   ^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\generators\base.py", line 162, in generate
   outputs = self._call_model(prompt, generations_this_call)
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\backoff_sync.py", line 105, in retry
   ret = target(*args, kwargs)
   ^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\garak\generators\openai.py", line 270, in _call_model
   response = generator.create(create_args)
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\openai_utils_utils.py", line 286, in wrapper
   return func(*args, kwargs)
   ^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\openai\resources\chat\completions\completions.py", line 1147, in create
   return self._post(
   ^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\openai_base_client.py", line 1259, in post
   return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\IRTS.Marissa.Christe\AppData\Roaming\Python\Python312\site-packages\openai_base_client.py", line 1047, in request
   raise self._make_status_error_from_response(err.response) from None
   openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: f65927ee**************d2d1. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

Were you following a specific guide/tutorial or reading documentation?

Yes, garak documentation
If yes link the guide/tutorial or documentation you were following here, otherwise you may omit this section.
https://github.com/NVIDIA/garak

Expected behavior

Scan should run

Current behavior

invalid_request_error

garak version

garak LLM vulnerability scanner v0.13.2

Additional Information

1. Operating system: Windows
2. Python version: 3.12
3. Install method (pypi, pip based repo install, direct repository checkout with git): Pip based repo install
4. Logs from execution run report.html / report.jsonl / hitlog.jsonl and if possible garak.log

5. Details of execution config such as command line flags or config files
6. Any relevant hardware or resource information

[MarChristensen]

I have tested the API key with powershell and it works, it just doesn't work in garak.

[jmartin-tech]

--model_type openai would not target Azure Gov OpenAI it would exercise the public commercial service. I suspect the key provided is not valid for the public consumer service.

There is a specific generator for Azure hosted models. The azure.AzureOpenAIGenerator requires a number of additional environment variables to navigate the azure authentication requirements, though I do not have a way to test this with Gov cloud based Azure targets. The generator has been tested with a model hosted in a standard Azure cloud account.

[MarChristensen]

I am trying to run this through the CLI, do I just pass in something in the command line like,
--generators azure.AzureOpenAIGenerator?

Thanks for your help!

[jmartin-tech]

Generators are the input value for the --target_type parameter, I am not sure what the env for OpenAI gov looks like but the following may offer a basic template.

azure.AzureOpenAIGenerator is the default generator from the azure module offering access via --target_type azure.

As noted in the generator docs there are 3 env variable that are needed for Azure proxy of models:

• AZURE_API_KEY
• AZURE_ENDPOINT - see docs for expected url
• AZURE_MODEL_NAME - defined in your azure deployment

These can be passed in configuration files as well however for simplicity here I just use the env vars:

export AZURE_API_KEY="your super secret key"
export AZURE_ENDPOINT="https://YOUR_RESOURCE_NAME.openai.azure.com/"
export AZURE_MODEL_NAME="your_deployment_name"
garak --target_type azure --target_name your_deployment_name --probes packagehallucination.Dart