Screen HUD

Real-time screen monitoring and plugin-based desktop HUD tool Based on Tauri + React + Rust, integrated Python plugin system, supports multi-display and high-performance rendering

English | 中文

---

👥 User Guide

Use Cases

• Real-time screen monitoring and analysis: Real-time monitoring of screen content, analysis and display of HUD overlay
• Multi-display support: Support for independent rendering across multiple displays
• Plugin-based extension: Extend functionality through plugins

Core Features

• Plugin-based architecture:
  • Support for custom Python plugin development
• Multi-display support:
• Real-time processing:
  • Low-latency screen capture
  • Configurable polling interval
• Security:
  • Open source and auditable
  • Local processing, no network connection

Plugin List

🎭 FaceRecognition - Face Recognition Plugin

• High-precision face detection and recognition with masking based on InsightFace
• Supports known face database and real-time recognition
• GPU/CPU adaptive acceleration

🖼️ FaceMask - Face Mosaic Plugin

• Real-time face detection and masking using OpenCV

🚀 HelloWorld - Example Plugin

• Simple text display example
• Complete standardized interface implementation

Installation and Running

Method 1: Direct Download and Run (Recommended)

1. Go to the Releases page to download the release package, extract to any directory
2. Ensure the directory structure is as follows:

   your-directory/
   ├── screen-hud.exe
   ├── config.toml
   └── plugins/
       ├── HelloWorld/
       │   ├── screen_hud.html
       │   ├── screen_hud.py
       │   └── README.md
       ├── FaceMask/
       │   ├── screen_hud.html
       │   ├── screen_hud.py
       │   └── README.md
       └── FaceRecognition/
           ├── screen_hud.html
           ├── screen_hud.py
           ├── README.md
           └── known_faces/
               ├── 张三/
               │   ├── photo1.jpg
               │   └── photo2.jpg
               └── 李四/
                   ├── photo1.jpg
                   └── photo2.jpg
   

3. Double-click screen-hud.exe to run

Environment Requirements:

• Windows 10/11 - Currently supported platform
• Python 3.8+ - Plugin runtime environment (automatically detected on first startup)
• WebView2 - Runtime (usually pre-installed on Windows 10/11)

Environment Management Architecture:

• Rust Backend Responsibility: Automatically detect and install system Python and pip on application startup
• Plugin Responsibility: Only need to initialize their own package dependencies (such as OpenCV, numpy, etc.)
• Smart Fallback: Prioritize system packages, create virtual environment only when system packages are unavailable

First startup will automatically:

• Detect system Python; create venv and install dependencies if unavailable
• Subsequent startups will first verify if venv dependencies are complete, skip installation if complete

Method 2: Source Code Compilation and Run

# Clone repository
git clone https://github.com/your-username/screen-hud.git
cd screen-hud

# Install frontend dependencies
npm install

# Development mode startup
npm run tauri dev

Prerequisites:

• Node.js 18+ - Frontend development environment
• Rust 1.70+ - Tauri backend compilation
• Python 3.8+ - Plugin runtime environment
• Windows 10/11 - Currently supported platform

Build release version:

# Build frontend
npm run build

# Build Tauri application
npm run tauri build

Usage Instructions

Basic Usage Flow

1. Start application: Double-click screen-hud.exe to start the application
2. Wait for initialization: The application will automatically initialize all plugins
3. Select plugin: Choose the plugin to use in the main interface
4. Select display: Choose the display to monitor
5. Start monitoring: Click the start button to begin monitoring
6. View effects: View plugin effects on the selected display

Plugin Configuration

• FaceRecognition: Add known face photos in the plugins/FaceRecognition/known_faces/ directory
• FaceMask: No additional configuration required, ready to use out of the box
• HelloWorld: No additional configuration required, suitable for testing and learning

Configuration Options

The following parameters can be adjusted in config.toml:

[monitoring]
interval = 8               # Detection interval (ms)
capture_scale = 1.0        # Screenshot downsampling ratio

[system]
log_level = "info"        # Log level

👨‍💻 Development Guide

Plugin Architecture

Plugin Architecture Overview

Screen HUD adopts a plugin-based architecture, each plugin contains the following components:

• Python backend: Process screenshot data, implement core logic
• HTML frontend: Render plugin output, display on screen

Environment Management Architecture

• Rust Backend Responsibilities:
  • Automatically detect system Python and pip availability on application startup
  • Install ensurepip if system Python/pip is unavailable
  • Provide stable Python runtime environment for plugins
• Plugin Responsibilities:
  • Only need to initialize their own package dependencies in plugin_init()
  • Prioritize system packages, create virtual environment only when system packages are unavailable
  • Focus on business logic implementation, no need to worry about Python environment management

Plugin Workflow

1. Application startup: Pre-create overlay window pool
2. User selection: Select plugin and display to start monitoring
3. Exclusive monitoring: Automatically stop other monitoring on the same display
4. Load plugin: Get window from pool and load plugin
5. Data processing: Plugin processes screenshot data and returns rendering information
6. Real-time rendering: Frontend renders plugin output in real-time

Plugin Structure

plugins/
├── YourPlugin/
│   ├── screen_hud.html    # HUD display page
│   ├── screen_hud.py      # Python plugin logic
│   └── README.md          # Plugin documentation

Standardized Interface Specification

Required Interfaces

• plugin_init(): Plugin initialization, focus on package dependency installation and model loading (Python environment guaranteed by Rust backend)
• plugin_ready(): Check if plugin is ready
• plugin_interval(): Return polling interval (milliseconds)
• plugin_process(image_data): Process screenshot data
• plugin_cleanup(): Plugin cleanup and resource release

Optional Interfaces

• plugin_info(): Return detailed plugin information (JSON string)

Frontend Communication Mechanism

Python to HTML Data Transfer

The return value of Python plugin's plugin_process() will be directly passed to HTML frontend, no additional serialization steps required:

def plugin_process(image_data) -> str:
    """Process screenshot data, return JSON string to frontend"""
    return json.dumps({
        "type": "text",
        "content": "Hello World!",
        "position": {"x": 100, "y": 100},
        "style": {"fontSize": 24, "color": "#FF0000"}
    })

HTML Frontend Message Reception

HTML frontend uses window.addEventListener to listen for messages from Rust backend:

<!DOCTYPE html>
<html>
<head>
    <title>Plugin HUD</title>
</head>
<body>
    <canvas id="canvas"></canvas>
    
    <script>
        // Listen for messages from Rust backend
        window.addEventListener('message', function(event) {
            const data = JSON.parse(event.data);
            
            // Process plugin returned data
            if (data.type === 'text') {
                renderText(data.content, data.position, data.style);
            } else if (data.type === 'image') {
                renderImage(data.src, data.position, data.angle);
            }
        });
        
        function renderText(content, position, style) {
            const canvas = document.getElementById('canvas');
            const ctx = canvas.getContext('2d');
            
            ctx.font = `${style.fontSize}px ${style.fontFamily || 'Arial'}`;
            ctx.fillStyle = style.color;
            ctx.fillText(content, position.x, position.y);
        }
    </script>
</body>
</html>

Why use window.addEventListener instead of Tauri API?

1. Overlay window isolation: Overlay windows run in independent WebView2 processes, isolated from the main Tauri application process
2. Decoupled design: Plugin frontend is decoupled from the main application, runs independently, safe and stable, does not affect the main window

Complete Development Example

Python Backend (screen_hud.py)

import json

class PluginState:
    def __init__(self):
        self.ready = False
        self.message_count = 0

_plugin_state = PluginState()

def plugin_init() -> bool:
    """Plugin initialization"""
    _plugin_state.ready = True
    return True

def plugin_ready() -> bool:
    """Check if plugin is ready"""
    return _plugin_state.ready

def plugin_interval() -> int:
    """Return polling interval (milliseconds)"""
    return 1000

def plugin_process(image_data) -> str:
    """Process screenshot data"""
    _plugin_state.message_count += 1
    return json.dumps({
        "type": "text",
        "content": f"Hello World! ({_plugin_state.message_count})",
        "position": {"x": 100, "y": 100},
        "style": {"fontSize": 24, "color": "#FF0000"}
    })

def plugin_cleanup() -> bool:
    """Plugin cleanup"""
    return True

def plugin_info() -> str:
    """Return plugin information"""
    info = {
        "name": "YourPlugin",
        "version": "1.0.0",
        "description": "Plugin description",
        "features": ["Feature 1", "Feature 2"],
        "status": {"ready": _plugin_state.ready}
    }
    return json.dumps(info, ensure_ascii=False, indent=2)

HTML Frontend (screen_hud.html)

<!DOCTYPE html>
<html>
<head>
    <title>Plugin HUD</title>
    <style>
        body {
            margin: 0;
            padding: 0;
            background: transparent;
            overflow: hidden;
        }
        #canvas {
            position: absolute;
            top: 0;
            left: 0;
        }
    </style>
</head>
<body>
    <canvas id="canvas"></canvas>
    
    <script>
        const canvas = document.getElementById('canvas');
        const ctx = canvas.getContext('2d');
        
        // Set canvas size
        function resizeCanvas() {
            canvas.width = window.innerWidth;
            canvas.height = window.innerHeight;
        }
        
        // Listen for window size changes
        window.addEventListener('resize', resizeCanvas);
        resizeCanvas();
        
        // Listen for messages from Rust backend
        window.addEventListener('message', function(event) {
            const data = JSON.parse(event.data);
            
            // Clear canvas
            ctx.clearRect(0, 0, canvas.width, canvas.height);
            
            // Process plugin returned data
            if (data.type === 'text') {
                renderText(data.content, data.position, data.style);
            } else if (data.type === 'image') {
                renderImage(data.src, data.position, data.angle);
            }
        });
        
        function renderText(content, position, style) {
            ctx.font = `${style.fontSize}px ${style.fontFamily || 'Arial'}`;
            ctx.fillStyle = style.color;
            ctx.fillText(content, position.x, position.y);
        }
        
        function renderImage(src, position, angle) {
            const img = new Image();
            img.onload = function() {
                ctx.save();
                ctx.translate(position.x, position.y);
                ctx.rotate(angle * Math.PI / 180);
                ctx.drawImage(img, -img.width/2, -img.height/2);
                ctx.restore();
            };
            img.src = src;
        }
    </script>
</body>
</html>

Development Steps

1. Create Plugin Directory

Create a new folder in the plugins/ directory, for example MyPlugin/

2. Implement Python Backend

Create screen_hud.py file, implement all required interfaces:

• Use PluginState class to manage plugin state
• Implement all required interface functions
• Return JSON format data to frontend

3. Create HTML Frontend

Create screen_hud.html file:

• Use window.addEventListener('message') to receive data
• Implement data rendering logic
• Handle different types of rendering requirements

Debugging and Testing

Log Configuration

Configure log_level = "debug" in config.toml to enable detailed logs:

[system]
log_level = "debug"  # Options: trace, debug, info, warn, error

Log output locations:

• Rust backend logs: Output to log files in logs/ directory
• Frontend console: When log_level = "debug", automatically opens browser developer tools console for screen_hud.html
  • Reason: HUD HTML is a transparent window with event penetration, normally cannot manually open console

Debugging Methods

1. Python plugin debugging:

   def plugin_process(image_data) -> str:
       print(f"[Plugin] Processing image: {len(image_data.get('data', []))} bytes")
       # Processing logic...
       result = {"type": "text", "content": "Debug info"}
       print(f"[Plugin] Returning: {result}")
       return json.dumps(result)

2. HTML frontend debugging:

   window.addEventListener('message', function(event) {
       console.log('Received data:', event.data);
       const data = JSON.parse(event.data);
       console.log('Parsed data:', data);
       // Process data...
   });

   Note: console.log() output is only visible in the automatically opened console when log_level = "debug"

Performance Optimization Suggestions

• Set reasonable polling intervals
• Optimize image processing algorithms
• Use GPU acceleration (if available)
• Avoid unnecessary resource consumption

---

Screen HUD - Powerful screen monitoring and plugin-based extension platform 🚀