Add translation to Grab Frame and Edit Text Window using Windows AI #602
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add TranslateText method to WindowsAiUtilities.cs using LanguageModel - Add translation settings to Settings files - Add translation toggle button and menu with language selection to GrabFrame.xaml - Implement translation logic in GrabFrame.xaml.cs with timer-based translation - Store original texts and restore them when translation is disabled - Support 12 target languages (English, Spanish, French, German, Italian, Portuguese, Russian, Japanese, Chinese, Korean, Arabic, Hindi) Co-authored-by: TheJoeFin <[email protected]>
- Add detailed comments explaining the translation approach - Use TextRewriter with custom translation prompt as workaround - Note that Microsoft.Extensions.AI could be added for better translation support Co-authored-by: TheJoeFin <[email protected]>
- Auto-freeze frame when translation is enabled for static content - Implement parallel translation of word borders for better performance - Use Dispatcher.InvokeAsync for thread-safe UI updates Co-authored-by: TheJoeFin <[email protected]>
- Hide translation button when Windows AI is not available - Update tooltip to show current target language - Improve user experience with dynamic tooltip updates Co-authored-by: TheJoeFin <[email protected]>
- Remove Task.Run with async lambdas to avoid thread pool issues - Add debug logging for translation errors - Replace hard-coded array index with robust menu search - Simplify translation logic to sequential processing Co-authored-by: TheJoeFin <[email protected]>
- Add semaphore to limit concurrent translations (max 3) - Extract translation logic into testable async Task methods - Replace magic string with constant - Add comprehensive XML documentation about translation limitations - Properly dispose semaphore on unload Co-authored-by: TheJoeFin <[email protected]>
- Replace hardcoded translation prompt with template constant - Improve code maintainability and allow for future customization Co-authored-by: TheJoeFin <[email protected]>
Copilot
AI
changed the title
[WIP] Add real-time translation in Grab Frame functionality
Add real-time translation to Grab Frame using Windows AI
Dec 27, 2025
Owner
|
@copilot add a translation to the Local AI menu on the Edit Text Window as well |
- Add Translate submenu with 12 language options to Local AI menu - Implement TranslateMenuItem_Click handler following existing pattern - Uses WindowsAiUtilities.TranslateText for translation - Supports both selected text and full text translation - Displays loading indicator during translation Co-authored-by: TheJoeFin <[email protected]>
Contributor
Author
Added translation to the Local AI menu in Edit Text Window (commit 3df768b). Menu Structure: Usage:
The implementation follows the same pattern as other Local AI features (Summarize, Rewrite, Convert to Table) and uses the |
- Show warning dialog if translation fails or returns unchanged text - Catch and display exceptions with appropriate error message - Improve user feedback when translation doesn't work Co-authored-by: TheJoeFin <[email protected]>
- Keep basic exception handling with error dialog - Remove check for unchanged text to match other AI handlers - More consistent with SummarizeMenuItem, RewriteMenuItem patterns Co-authored-by: TheJoeFin <[email protected]>
Copilot
AI
changed the title
Add real-time translation to Grab Frame using Windows AI
Add translation to Grab Frame and Edit Text Window using Windows AI
Dec 27, 2025
…ithub.com/TheJoeFin/Text-Grab into copilot/add-real-time-translation-feature
Updated error message display to explicitly use System.Windows.MessageBox to avoid potential naming conflicts.
Updated translation prompt for clarity and consistency, added a dedicated system prompt constant, and streamlined method documentation.
Enables Windows AI-powered translation for each word, adds progress feedback and cancellation, and optimizes resource usage with shared model and fast language detection.
Enable translating grabbed text to system language using Windows AI if available. Adds UI option and language detection for translation workflow.
Enables quick translation to system language, improves error handling, and updates UI menu dynamically. Includes minor formatting fixes.
Improves robustness by safely handling cases where recognizedText.Lines is null, preventing possible null reference exceptions.
There was a problem hiding this comment.
Pull request overview
This PR implements translation functionality using Windows AI LanguageModel API in both Grab Frame (real-time) and Edit Text Window (on-demand). The implementation uses TextRewriter as a workaround since WindowsAppSDK 1.8 lacks dedicated translation APIs. The feature includes language detection heuristics to optimize performance and supports 12 target languages including English, Spanish, French, German, Italian, Portuguese, Russian, Japanese, Chinese, Korean, Arabic, and Hindi.
Key Changes
- Translation API with shared model instance, concurrency control (max 3 simultaneous), and result cleaning to remove AI instruction echoes
- Real-time translation in Grab Frame with auto-freeze, timer-based updates, progress tracking, and cancellation support
- On-demand translation in Edit Text Window and FullscreenGrab with system language auto-detection
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| Text-Grab/Utilities/WindowsAiUtilities.cs | Adds core translation API with language detection heuristics, result cleaning, shared model management, and semaphore-controlled concurrency |
| Text-Grab/Views/GrabFrame.xaml.cs | Implements real-time translation logic with timer-based updates, cancellation support, original text restoration, and progress tracking |
| Text-Grab/Views/GrabFrame.xaml | Adds translation toggle button, language menu with 12 options, and progress indicator UI to toolbar |
| Text-Grab/Views/EditTextWindow.xaml.cs | Implements on-demand translation handlers for menu items with loading indicators and error handling |
| Text-Grab/Views/EditTextWindow.xaml | Adds translation menu items under Local AI menu following existing pattern |
| Text-Grab/Views/FullscreenGrab.xaml.cs | Adds post-capture translation option to auto-translate to system language |
| Text-Grab/Views/FullscreenGrab.xaml | Adds translation menu item to post-capture actions dropdown |
| Text-Grab/Utilities/LanguageUtilities.cs | Adds GetSystemLanguageForTranslation helper to map system language to translation-friendly names |
| Text-Grab/Properties/Settings.settings | Adds GrabFrameTranslationEnabled and GrabFrameTranslationLanguage settings for state persistence |
| Text-Grab/Properties/Settings.Designer.cs | Generated code for new translation settings properties |
| Text-Grab/Models/WinAiOcrLinesWords.cs | Adds null check for recognizedText.Lines to prevent NullReferenceException |
| Text-Grab/Controls/WordBorder.xaml.cs | Adds context menu translation option with system language detection and undo support |
| Text-Grab/Controls/WordBorder.xaml | Adds translation menu item to word border context menu |
Files not reviewed (1)
- Text-Grab/Properties/Settings.Designer.cs: Language not supported
Comment on lines
25
to
26
| private static readonly SemaphoreSlim _modelInitializationLock = new(1, 1); | ||
|
|
There was a problem hiding this comment.
The static semaphore '_modelInitializationLock' is never disposed, which could lead to a resource leak. Since this is a static field in a utility class that lives for the application lifetime, consider implementing a cleanup method that can be called during application shutdown, or document that this is intentional for application-lifetime resources.
Suggested change
| private static readonly SemaphoreSlim _modelInitializationLock = new(1, 1); | |
| private static readonly SemaphoreSlim _modelInitializationLock = new(1, 1); | |
| private static bool _disposed; | |
| /// <summary> | |
| /// Releases resources held by static members of <see cref="WindowsAiUtilities"/>. | |
| /// Should be called once during application shutdown. | |
| /// </summary> | |
| public static void Cleanup() | |
| { | |
| if (_disposed) | |
| return; | |
| _modelInitializationLock.Dispose(); | |
| _disposed = true; | |
| } |
Text-Grab/Views/GrabFrame.xaml.cs
Outdated
Comment on lines
2972
to
2983
| if (cancellationToken.IsCancellationRequested) | ||
| return; | ||
|
|
||
| string translatedText = await WindowsAiUtilities.TranslateText(originalText, translationTargetLanguage); | ||
|
|
||
| if (!cancellationToken.IsCancellationRequested) | ||
| { | ||
| wordBorder.Word = translatedText; | ||
|
|
||
| translatedWordsCount++; | ||
| await Dispatcher.InvokeAsync(() => UpdateTranslationProgress()); | ||
| } |
There was a problem hiding this comment.
Race condition potential: The cancellationToken is checked at line 2972 after acquiring the semaphore, but by the time the translation call at line 2975 executes, the token could be requested for cancellation. Consider passing the cancellationToken to WindowsAiUtilities.TranslateText if it supports cancellation, or check the token immediately before the translation call and throw OperationCanceledException to abort early.
Suggested change
| if (cancellationToken.IsCancellationRequested) | |
| return; | |
| string translatedText = await WindowsAiUtilities.TranslateText(originalText, translationTargetLanguage); | |
| if (!cancellationToken.IsCancellationRequested) | |
| { | |
| wordBorder.Word = translatedText; | |
| translatedWordsCount++; | |
| await Dispatcher.InvokeAsync(() => UpdateTranslationProgress()); | |
| } | |
| // Ensure cancellation is honored immediately before starting translation | |
| cancellationToken.ThrowIfCancellationRequested(); | |
| string translatedText = await WindowsAiUtilities.TranslateText(originalText, translationTargetLanguage); | |
| // If cancellation was requested during translation, abort before updating UI state | |
| cancellationToken.ThrowIfCancellationRequested(); | |
| wordBorder.Word = translatedText; | |
| translatedWordsCount++; | |
| await Dispatcher.InvokeAsync(() => UpdateTranslationProgress()); |
Text-Grab/Views/GrabFrame.xaml.cs
Outdated
Comment on lines
3025
to
3034
| foreach (var item in TranslationMenuItem.Items) | ||
| { | ||
| if (item is MenuItem menuItem && menuItem.Header.ToString() == TargetLanguageMenuHeader) | ||
| { | ||
| foreach (var langItem in menuItem.Items) | ||
| { | ||
| if (langItem is MenuItem langMenuItem && langMenuItem.Tag is string tag) | ||
| langMenuItem.IsChecked = tag == translationTargetLanguage; | ||
| } | ||
| break; |
There was a problem hiding this comment.
Potential NullReferenceException: When Windows AI is not available (line 3009 sets TranslationMenuItem.Visibility to Collapsed), the code at lines 3025-3036 still attempts to iterate through TranslationMenuItem.Items. If TranslationMenuItem is not properly initialized in XAML or is null, this will throw an exception. Consider adding a null check before the foreach loop or restructuring to only run this logic when canUseWinAI is true.
Suggested change
| foreach (var item in TranslationMenuItem.Items) | |
| { | |
| if (item is MenuItem menuItem && menuItem.Header.ToString() == TargetLanguageMenuHeader) | |
| { | |
| foreach (var langItem in menuItem.Items) | |
| { | |
| if (langItem is MenuItem langMenuItem && langMenuItem.Tag is string tag) | |
| langMenuItem.IsChecked = tag == translationTargetLanguage; | |
| } | |
| break; | |
| if (canUseWinAI && TranslationMenuItem != null) | |
| { | |
| foreach (var item in TranslationMenuItem.Items) | |
| { | |
| if (item is MenuItem menuItem && menuItem.Header.ToString() == TargetLanguageMenuHeader) | |
| { | |
| foreach (var langItem in menuItem.Items) | |
| { | |
| if (langItem is MenuItem langMenuItem && langMenuItem.Tag is string tag) | |
| langMenuItem.IsChecked = tag == translationTargetLanguage; | |
| } | |
| break; | |
| } |
Comment on lines
3574
to
3630
| private async void TranslateMenuItem_Click(object sender, RoutedEventArgs e) | ||
| { | ||
| if (sender is not MenuItem menuItem || menuItem.Tag is not string targetLanguage) | ||
| return; | ||
|
|
||
| string textToTranslate = GetSelectedTextOrAllText(); | ||
|
|
||
| SetToLoading($"Translating to {targetLanguage}..."); | ||
|
|
||
| try | ||
| { | ||
| string translatedText = await WindowsAiUtilities.TranslateText(textToTranslate, targetLanguage); | ||
|
|
||
| if (PassedTextControl.SelectionLength == 0) | ||
| PassedTextControl.Text = translatedText; | ||
| else | ||
| PassedTextControl.SelectedText = translatedText; | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| System.Windows.MessageBox.Show($"Translation failed: {ex.Message}", | ||
| "Translation Error", MessageBoxButton.OK, MessageBoxImage.Warning); | ||
| } | ||
| finally | ||
| { | ||
| SetToLoaded(); | ||
| } | ||
| } | ||
|
|
||
| private async void TranslateToSystemLanguageMenuItem_Click(object sender, RoutedEventArgs e) | ||
| { | ||
| string textToTranslate = GetSelectedTextOrAllText(); | ||
|
|
||
| // Get system language using the helper from LanguageUtilities | ||
| string systemLanguage = LanguageUtilities.GetSystemLanguageForTranslation(); | ||
|
|
||
| SetToLoading($"Translating to {systemLanguage}..."); | ||
|
|
||
| try | ||
| { | ||
| string translatedText = await WindowsAiUtilities.TranslateText(textToTranslate, systemLanguage); | ||
|
|
||
| if (PassedTextControl.SelectionLength == 0) | ||
| PassedTextControl.Text = translatedText; | ||
| else | ||
| PassedTextControl.SelectedText = translatedText; | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| System.Windows.MessageBox.Show($"Translation failed: {ex.Message}", | ||
| "Translation Error", MessageBoxButton.OK, MessageBoxImage.Warning); | ||
| } | ||
| finally | ||
| { | ||
| SetToLoaded(); | ||
| } | ||
| } |
There was a problem hiding this comment.
The TranslateToSystemLanguageMenuItem_Click and TranslateMenuItem_Click methods contain nearly identical code blocks (lines 3614-3619 and 3585-3590). Consider extracting this duplicated logic into a shared helper method to improve maintainability. The only difference between them is how the target language is obtained (from system language vs. MenuItem.Tag), so you could create a method like 'PerformTranslationAsync(string targetLanguage)' that both methods could call.
Text-Grab/Views/GrabFrame.xaml.cs
Outdated
Comment on lines
462
to
464
| // Dispose the shared translation model when translation is disabled | ||
| if (!isTranslationEnabled) | ||
| WindowsAiUtilities.DisposeTranslationModel(); |
There was a problem hiding this comment.
The logic for disposing the translation model appears incorrect. The condition 'if (!isTranslationEnabled)' means the model is disposed when translation is disabled, but it should be disposed regardless of the toggle state when the window unloads. Since GrabFrame_Unloaded runs during cleanup, the model should always be disposed to prevent resource leaks. Remove the condition and always call WindowsAiUtilities.DisposeTranslationModel().
Suggested change
| // Dispose the shared translation model when translation is disabled | |
| if (!isTranslationEnabled) | |
| WindowsAiUtilities.DisposeTranslationModel(); | |
| // Dispose the shared translation model during cleanup to prevent resource leaks | |
| WindowsAiUtilities.DisposeTranslationModel(); |
Text-Grab/Views/GrabFrame.xaml.cs
Outdated
Comment on lines
3045
to
3049
| foreach (WordBorder wb in wordBorders) | ||
| { | ||
| if (originalTexts.TryGetValue(wb, out string? originalText)) | ||
| wb.Word = originalText; | ||
| } |
There was a problem hiding this comment.
This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.
Comment on lines
406
to
412
| if (_translationLanguageModel != null) | ||
| return; | ||
|
|
||
| await _modelInitializationLock.WaitAsync(); | ||
| try | ||
| { | ||
| if (_translationLanguageModel == null) |
There was a problem hiding this comment.
Condition is always true because of call to operator !=.
Suggested change
| if (_translationLanguageModel != null) | |
| return; | |
| await _modelInitializationLock.WaitAsync(); | |
| try | |
| { | |
| if (_translationLanguageModel == null) | |
| if (!object.ReferenceEquals(_translationLanguageModel, null)) | |
| return; | |
| await _modelInitializationLock.WaitAsync(); | |
| try | |
| { | |
| if (object.ReferenceEquals(_translationLanguageModel, null)) |
Comment on lines
85
to
96
| if (hasLatin && text.Length > 10) | ||
| { | ||
| // Check for common English words as additional heuristic | ||
| if (targetCode == "en") | ||
| { | ||
| string lowerText = text.ToLowerInvariant(); | ||
| string[] commonEnglishWords = { " the ", " and ", " or ", " is ", " are ", " was ", " were ", " in ", " on ", " at ", " to ", " of ", " for ", " with " }; | ||
| int englishWordCount = commonEnglishWords.Count(w => lowerText.Contains(w)); | ||
| // If text contains multiple common English words, likely already English | ||
| if (englishWordCount >= 2) | ||
| return true; | ||
| } |
There was a problem hiding this comment.
These 'if' statements can be combined.
Suggested change
| if (hasLatin && text.Length > 10) | |
| { | |
| // Check for common English words as additional heuristic | |
| if (targetCode == "en") | |
| { | |
| string lowerText = text.ToLowerInvariant(); | |
| string[] commonEnglishWords = { " the ", " and ", " or ", " is ", " are ", " was ", " were ", " in ", " on ", " at ", " to ", " of ", " for ", " with " }; | |
| int englishWordCount = commonEnglishWords.Count(w => lowerText.Contains(w)); | |
| // If text contains multiple common English words, likely already English | |
| if (englishWordCount >= 2) | |
| return true; | |
| } | |
| if (hasLatin && text.Length > 10 && targetCode == "en") | |
| { | |
| // Check for common English words as additional heuristic | |
| string lowerText = text.ToLowerInvariant(); | |
| string[] commonEnglishWords = { " the ", " and ", " or ", " is ", " are ", " was ", " were ", " in ", " on ", " at ", " to ", " of ", " for ", " with " }; | |
| int englishWordCount = commonEnglishWords.Count(w => lowerText.Contains(w)); | |
| // If text contains multiple common English words, likely already English | |
| if (englishWordCount >= 2) | |
| return true; |
Comment on lines
3587
to
3590
| if (PassedTextControl.SelectionLength == 0) | ||
| PassedTextControl.Text = translatedText; | ||
| else | ||
| PassedTextControl.SelectedText = translatedText; |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Comment on lines
3616
to
3619
| if (PassedTextControl.SelectionLength == 0) | ||
| PassedTextControl.Text = translatedText; | ||
| else | ||
| PassedTextControl.SelectedText = translatedText; |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Owner
|
@copilot apply changes based on the comments in this thread |
…de quality issues - Add Cleanup() method to WindowsAiUtilities for proper disposal of static semaphore - Fix race condition in translation cancellation by using ThrowIfCancellationRequested() - Add null check for TranslationMenuItem to prevent NullReferenceException - Extract common translation logic into PerformTranslationAsync() helper method - Always dispose translation model on window unload regardless of translation state - Remove unused isTranslating field from GrabFrame - Replace duplicate GetSystemLanguageName() with call to LanguageUtilities.GetSystemLanguageForTranslation() - Fix indentation in WindowsAiUtilities.cs (TranslateText method and documentation) - Fix indentation in GrabFrame.xaml.cs (CancelTranslationButton_Click method) - Use explicit .Where() filtering in foreach loops for better clarity - Use object.ReferenceEquals() for null checks in double-checked locking pattern - Combine nested if statements for targetCode == "en" check - Keep explicit if/else for text assignment for better readability Co-authored-by: TheJoeFin <[email protected]>
Contributor
Author
Applied all code review feedback in commit ef3f6ad. Key changes: Resource Management:
Thread Safety:
Code Quality:
All 17 review comments have been addressed. |
- Add Cleanup() method for proper disposal of static semaphore - Use object.ReferenceEquals() for null checks in double-checked locking - Use explicit .Where() filtering in foreach loop - Combine nested if statements for targetCode check Co-authored-by: TheJoeFin <[email protected]>
Added GitHub Sponsors, Buy Me a Coffee, and updated PayPal link.
Improves UI structure, type safety, and translation feature gating based on Windows AI availability.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Translation Feature - Code Review Improvements ✅
This PR implements translation functionality using Windows AI LanguageModel API in both Grab Frame (real-time) and Edit Text Window (on-demand), with all 17 code review improvements applied.
📝 Code Review Changes Applied
✅ Resource Management
Cleanup()method toWindowsAiUtilitiesfor proper disposal of static semaphore✅ Thread Safety & Cancellation
ThrowIfCancellationRequested()before and after translation✅ Null Safety
TranslationMenuItembefore iteration to prevent NullReferenceExceptionobject.ReferenceEquals()for null checks in double-checked locking pattern✅ Code Quality & Maintainability
PerformTranslationAsync()helper methodisTranslatingfieldGetSystemLanguageName()withLanguageUtilities.GetSystemLanguageForTranslation().Where()filtering in foreach loops for clarity✨ Features
Grab Frame Translation
Edit Text Window Translation
🛠️ Technical Implementation
Architecture:
LanguageModelAPI throughTextRewriterCode Quality:
🧪 Testing Requirements
Requires: Windows 11, ARM64 (or
OverrideAiArchCheck), WindowsAppSDK 1.8+Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.