Lookout v3.0.1: Parser fixes for Streaming/Polling API, workbook updates, threat hunting notebooks

[fgravato]

Summary

Updates Lookout solution v3.0.1 with parser fixes to support the actual streaming API data structure and adds threat hunting notebooks.

Changes Made

Parser v3.1.0

• ✅ Added coalesce() support for Streaming API (device.info.email, device.hardware.*, device.software.*, device.status.*)
• ✅ Added support for Polling API (same nested structure)
• ✅ Added support for REST API (device.email_address, device.manufacturer, etc.)
• ✅ Handles all 4 event types: THREAT, DEVICE, SMISHING_ALERT, AUDIT

Workbooks

• ✅ Updated all 5 workbooks to use correct streaming API field paths
• ✅ Fixed queries to use tostring() for safe field extraction

Notebooks (NEW)

• ✅ Lookout-ThreatHunting-MobileMalware.ipynb - Mobile malware investigation
• ✅ Lookout-ThreatHunting-Smishing.ipynb - SMS phishing campaign analysis
• ✅ Lookout-ThreatHunting-DeviceCompliance.ipynb - Device compliance hunting
• ✅ Lookout-ThreatHunting-AuditInsider.ipynb - Audit & insider threat analysis

Cleanup

• ✅ Removed development/documentation files
• ✅ Removed test scripts and validation files
• ✅ Fixed gated URLs (replaced enterprise.support.lookout.com with public URLs)

Technical Details

The Lookout MRA v2 Streaming API (/mra/stream/v2/events) returns data with a different nested structure than documented:

Field	REST API Docs	Streaming API (Actual)
Email	device.email_address	device.info.email
Manufacturer	device.manufacturer	device.hardware.manufacturer
OS Version	device.os_version	device.software.os_version

The parser now uses coalesce() to handle both:

DeviceEmailAddress = coalesce(tostring(threat.device.email), tostring(device.info.email), tostring(device.email_address))

Testing

• ✅ All analytic rules use LookoutAPI connector ID
• ✅ Parser handles multiple API structures
• ✅ Version 3.0.1 consistent across metadata files
• ✅ No hardcoded locations
• ✅ No gated URLs

Files Changed

• 29 files (Lookout solution only)
• Net: -3,134 lines (cleanup of dev files)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[fgravato]

Fix Applied

Fixed Microsoft Sentinel branding validation error:

• Replaced 'Azure Sentinel' with 'Microsoft Sentinel' in 4 files:
  • Workbooks/LookoutEvents.json
  • Workbooks/LookoutEventsV2.json
  • Package/mainTemplate.json
  • Package/createUiDefinition.json

CI should pass now.

[v-shukore]

Hi @fgravato, could you repackage this solution using the V3 tool and generate an updated package with current version? Currently, only the maintemplate and createui files were updated; the package also needs to be updated. Thanks!

[fgravato]

Repackaged using V3 Tool

As requested by @v-shukore, I've repackaged the solution using the V3 tool (createSolutionV3.ps1).

Changes:

• ✅ Regenerated 3.0.1.zip with all solution components
• ✅ Updated mainTemplate.json
• ✅ Updated createUiDefinition.json
• ✅ All ARM-TTK validations passed

Ready for review!

[fgravato]

Hi @v-shuklasumit, could you please review this PR when you have a chance?

The latest commit fixes the Microsoft Sentinel branding validation failure - updated 'Azure Sentinel Solution' to 'Microsoft Sentinel Solution' in the workbook descriptions.

Thank you!

[v-shukore]

Hi @fgravato, we checked the following URL: https://www.lookout.com/hc/en-us/articles/115002741773-Mobile-Risk-API-Guide, and it is returning a 404 error.

Could you please share the updated workbook, parser screenshots, and screenshots of the newly added notebooks running?
Thanks!