Skip to content

Implement mass addition of bib information #12025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 75 commits into from
Jun 7, 2025
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
887dd35
feat: enhance right-click context menu with new actions (extract refe…
BojiZhang Oct 19, 2024
656facf
fix: Restore MERGE_WITH_FETCHED_ENTRY and add BATCH_MERGE_WITH_FETCHE…
BojiZhang Oct 19, 2024
0409856
Refactor and optimize FetchAndMergeEntry class
BojiZhang Oct 19, 2024
489b512
Merge branch 'JabRef:main' into fix-for-issue-#372
BojiZhang Oct 19, 2024
dc138ce
Updated CHANGELOG.md
BojiZhang Oct 19, 2024
0532df4
Added a period at the end of the description in CHANGELOG.md.
BojiZhang Oct 19, 2024
012dd9a
Updated JabRef_en.properties
BojiZhang Oct 19, 2024
f8267ac
Move and rename "Mass Getting bibliographic data" action
BojiZhang Oct 20, 2024
74f2473
Refactor FetchAndMergeEntry class for improved functionality and read…
BojiZhang Oct 20, 2024
29759a6
Merge branch 'main' into fix-for-issue-#372
koppor Oct 20, 2024
37cfeae
Refactor entry fetching and merging process
BojiZhang Oct 20, 2024
77e0d51
Updated CHANGELOG.md
BojiZhang Oct 20, 2024
e6f0ad2
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 20, 2024
c5b7b4c
refactor: Separate UI and core logic in entry merging system
BojiZhang Oct 25, 2024
5b829b1
Merge remote-tracking branch 'upstream/fix-for-issue-#372' into fix-f…
BojiZhang Oct 25, 2024
350502e
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 25, 2024
fe46c0c
Refactored updateFieldIfNecessaryWithUndo method in MergeEntriesHelpe…
BojiZhang Oct 25, 2024
8eac6a9
Merge remote-tracking branch 'upstream/fix-for-issue-#372' into fix-f…
BojiZhang Oct 25, 2024
5d54083
refactor: Replace String.format with string concatenation in MergingI…
BojiZhang Oct 25, 2024
93072cf
refactor: Fixed incorrect placeholder in updateProgress Method
BojiZhang Oct 25, 2024
262d25d
Updated JabRef_en.properties to fix the failing LocalizationConsisten…
BojiZhang Oct 25, 2024
4667124
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 26, 2024
ae4df14
refactor: optimize BibEntry merging and fetching logic:
BojiZhang Oct 26, 2024
5ed3d70
Roll Back: FetchAndMergeEntry.java and MergeWithFetchedEntryAction.java.
BojiZhang Oct 27, 2024
74a6285
Merge branch 'main' into fix-for-issue-#372
koppor Oct 27, 2024
8d9bdce
Refactor: improve MergingIdBasedFetcher, BatchEntryMergeWithFetchedD…
BojiZhang Oct 27, 2024
95e9ae8
Fixed the minor import format issue.
BojiZhang Oct 27, 2024
9fdbbb4
Merge remote-tracking branch 'upstream/fix-for-issue-#372' into fix-f…
BojiZhang Oct 27, 2024
ce1ea3e
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 27, 2024
7305f3c
Apply suggestions from code review
BojiZhang Oct 27, 2024
9dce81a
Refactor: Reorder parameters to (fetcher, library) in merge operations
BojiZhang Oct 27, 2024
5f24815
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 27, 2024
d17fd88
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 30, 2024
61f5a8f
Refactor: Deleted the empty line before package
BojiZhang Oct 31, 2024
86051ac
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 31, 2024
198f2d0
Merge branch 'main' into fix-for-issue-#372
BojiZhang Oct 31, 2024
ace1975
Merge branch 'main' into fix-for-issue-#372
BojiZhang Nov 13, 2024
74681e1
Refactor: Deleted the empty line before package
BojiZhang Nov 16, 2024
3197ef4
refactor: Extract background processing into BatchEntryMergeTask, opt…
BojiZhang Nov 16, 2024
c04a2fa
Updated JabRef_en.properties
BojiZhang Nov 16, 2024
d2566db
Merge branch 'main' into fix-for-issue-#372
BojiZhang Nov 17, 2024
4d1b88d
Apply suggestions from code review
BojiZhang Nov 17, 2024
e8645a2
Modify log messages correspondingly.
BojiZhang Nov 17, 2024
ac8c59d
Refactor: Move string normalization from StringNormalizer to StringUtil
BojiZhang Nov 17, 2024
5172bc2
Merge branch 'main' into fix-for-issue-#372
BojiZhang Nov 18, 2024
05b3c42
Merge branch 'main' into fix-for-issue-#372
BojiZhang Dec 5, 2024
81043b7
Merge branch 'main' into fix-for-issue-#372
BojiZhang Feb 7, 2025
1a7bbb7
Merge branch 'main' into fix-for-issue-#372
BojiZhang Feb 23, 2025
b345e6b
Merge branch 'main' into fix-for-issue-#372
BojiZhang Mar 1, 2025
0ce2092
Modified the StringUtilClassIsSmall() test in StringUtilTest class.
BojiZhang Mar 1, 2025
2b208de
Added the cancellation feature to batch updating
BojiZhang Mar 1, 2025
e018398
Attempted to fix a failed test by modifying CHANGELOG.md.
BojiZhang Mar 1, 2025
cd4a318
Attempted to fix a conflict by modifying CHANGELOG.md.
BojiZhang Mar 10, 2025
96ff453
Merge branch 'main' into fix-for-issue-#372
BojiZhang Mar 10, 2025
c37cc82
Merge branch 'main' into fix-for-issue-#372
subhramit Mar 10, 2025
7c47166
Merge branch 'main' into fix-for-issue-#372
BojiZhang Mar 10, 2025
5ad79bb
Merge branch 'main' into fix-for-issue-#372
BojiZhang Mar 29, 2025
bcd1318
Attempted to fix an issue in RightClickMenu
BojiZhang Mar 29, 2025
ae9f742
Use named groups
subhramit Jun 5, 2025
9d0d1c4
Fix codespaces indent
subhramit Jun 5, 2025
b919860
Merge remote-tracking branch 'origin/main' into fix-for-issue-#372
koppor Jun 5, 2025
b0da898
Fix logger import
koppor Jun 5, 2025
a0d2c8f
Remove unnecessary separator
koppor Jun 5, 2025
a70195d
Remove obsolete comment
koppor Jun 5, 2025
8057a82
Reorder lookup menu
koppor Jun 5, 2025
da44233
Compilefix
koppor Jun 5, 2025
7f0d9c6
Merge branch 'main' into fix-for-issue-#372
koppor Jun 5, 2025
23b6172
add test case
calixtus Jun 6, 2025
1a7cf26
Remove normalize method
calixtus Jun 7, 2025
983091a
Refactor RemoveWhitespace to NonSpaceWhitespaceRemover
calixtus Jun 7, 2025
db3e91d
Fix CHANGELOG.md
calixtus Jun 7, 2025
e394341
Remove argument record
calixtus Jun 7, 2025
4dd5004
Simplify
calixtus Jun 7, 2025
c223e24
Rewrite
calixtus Jun 7, 2025
88ce43f
l10n
calixtus Jun 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
- We added automatic browser extension install on Windows for Chrome and Edge. [#6076](https://github.com/JabRef/jabref/issues/6076)
- We added a search bar for filtering keyboard shortcuts. [#11686](https://github.com/JabRef/jabref/issues/11686)
- By double clicking on a local citation in the Citation Relations Tab you can now jump the linked entry. [#11955](https://github.com/JabRef/jabref/pull/11955)
- Added batch fetching of bibliographic data for multiple entries in the "Lookup" menu. [#372](https://github.com/JabRef/jabref/issues/372)
- We use the menu icon for background tasks as a progress indicator to visualise an import's progress when dragging and dropping several PDF files into the main table. [#12072](https://github.com/JabRef/jabref/pull/12072)

### Changed
Expand Down
1 change: 1 addition & 0 deletions src/main/java/org/jabref/gui/actions/StandardActions.java
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ public enum StandardActions implements Action {
OPEN_URL(Localization.lang("Open URL or DOI"), IconTheme.JabRefIcons.WWW, KeyBinding.OPEN_URL_OR_DOI),
SEARCH_SHORTSCIENCE(Localization.lang("Search ShortScience")),
MERGE_WITH_FETCHED_ENTRY(Localization.lang("Get bibliographic data from %0", "DOI/ISBN/...")),
BATCH_MERGE_WITH_FETCHED_ENTRY(Localization.lang("Get bibliographic data from %0 (fully automated)", "DOI/ISBN/...")),
ATTACH_FILE(Localization.lang("Attach file"), IconTheme.JabRefIcons.ATTACH_FILE),
ATTACH_FILE_FROM_URL(Localization.lang("Attach file from URL"), IconTheme.JabRefIcons.DOWNLOAD_FILE),
PRIORITY(Localization.lang("Priority"), IconTheme.JabRefIcons.PRIORITY),
Expand Down
16 changes: 15 additions & 1 deletion src/main/java/org/jabref/gui/frame/MainMenu.java
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
import org.jabref.gui.linkedfile.RedownloadMissingFilesAction;
import org.jabref.gui.maintable.NewLibraryFromPdfActionOffline;
import org.jabref.gui.maintable.NewLibraryFromPdfActionOnline;
import org.jabref.gui.mergeentries.BatchEntryMergeWithFetchedDataAction;
import org.jabref.gui.mergeentries.MergeEntriesAction;
import org.jabref.gui.plaincitationparser.PlainCitationParserAction;
import org.jabref.gui.preferences.GuiPreferences;
Expand Down Expand Up @@ -273,7 +274,20 @@ private void createMenu() {

new SeparatorMenuItem(),

factory.createMenuItem(StandardActions.FIND_UNLINKED_FILES, new FindUnlinkedFilesAction(dialogService, stateManager))
factory.createMenuItem(StandardActions.FIND_UNLINKED_FILES, new FindUnlinkedFilesAction(dialogService, stateManager)),

new SeparatorMenuItem(),

factory.createMenuItem(
StandardActions.BATCH_MERGE_WITH_FETCHED_ENTRY,
new BatchEntryMergeWithFetchedDataAction(
frame::getCurrentLibraryTab,
preferences,
dialogService,
stateManager,
taskExecutor
)
)
);

final MenuItem pushToApplicationMenuItem = factory.createMenuItem(pushToApplicationCommand.getAction(), pushToApplicationCommand);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@

package org.jabref.gui.mergeentries;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.function.Supplier;

import javafx.application.Platform;

import org.jabref.gui.LibraryTab;
import org.jabref.gui.StateManager;
import org.jabref.gui.actions.ActionHelper;
import org.jabref.gui.actions.SimpleCommand;
import org.jabref.gui.preferences.GuiPreferences;
import org.jabref.gui.undo.NamedCompound;
import org.jabref.logic.importer.fetcher.MergingIdBasedFetcher;
import org.jabref.logic.l10n.Localization;
import org.jabref.logic.util.BackgroundTask;
import org.jabref.logic.util.NotificationService;
import org.jabref.logic.util.TaskExecutor;
import org.jabref.model.entry.BibEntry;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Handles the merging of multiple bibliography entries with fetched data.
* This action performs background fetching and merging of entries while
* providing progress updates to the user.
*/
public class BatchEntryMergeWithFetchedDataAction extends SimpleCommand {

private static final Logger LOGGER = LoggerFactory.getLogger(BatchEntryMergeWithFetchedDataAction.class);

private final Supplier<LibraryTab> tabSupplier;
private final GuiPreferences preferences;
private final NotificationService notificationService;
private final TaskExecutor taskExecutor;

public BatchEntryMergeWithFetchedDataAction(Supplier<LibraryTab> tabSupplier,
GuiPreferences preferences,
NotificationService notificationService,
StateManager stateManager,
TaskExecutor taskExecutor) {
this.tabSupplier = tabSupplier;
this.preferences = preferences;
this.notificationService = notificationService;
this.taskExecutor = taskExecutor;
this.executable.bind(ActionHelper.needsDatabase(stateManager));
}

@Override
public void execute() {
LibraryTab libraryTab = tabSupplier.get();
if (libraryTab == null) {
LOGGER.error("Cannot perform batch merge: no library is open.");
return;
}

List<BibEntry> entries = libraryTab.getDatabase().getEntries();
if (entries.isEmpty()) {
notificationService.notify(Localization.lang("empty library"));
return;
}

MergeContext context = new MergeContext(
libraryTab,
entries,
new MergingIdBasedFetcher(preferences.getImportFormatPreferences()),
notificationService
);

taskExecutor.execute(createBackgroundTask(context));
}

private static BackgroundTask<List<String>> createBackgroundTask(MergeContext context) {
BackgroundTask<List<String>> task = new BackgroundTask<>() {
@Override
public List<String> call() {
return processMergeEntries(context, this);
}
};

task.setTitle(Localization.lang("Fetching and merging entries"));
task.showToUser(true);

task.onSuccess(updatedEntries ->
handleSuccess(updatedEntries, context));

task.onFailure(ex ->
handleFailure(ex, context.notificationService));

return task;
}

private static List<String> processMergeEntries(MergeContext context, BackgroundTask<?> task) {
int totalEntries = context.entries().size();

for (int i = 0; i < totalEntries && !task.isCancelled(); i++) {
BibEntry entry = context.entries().get(i);
updateProgress(i, totalEntries, task);
fetchAndMergeEntry(context, entry);
}

finalizeCompoundEdit(context);
return context.updatedEntries();
}

private static void fetchAndMergeEntry(MergeContext context, BibEntry entry) {
LOGGER.debug("Processing entry: {}", entry);

context.fetcher().fetchEntry(entry)
.filter(MergingIdBasedFetcher.FetcherResult::hasChanges)
.ifPresent(result -> Platform.runLater(() -> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is runLater() needed? The whole thing runs in a background task - and there is no interaction with the UI, is it?

MergeEntriesHelper.mergeEntries(entry, result.mergedEntry(), context.compoundEdit());
entry.getCitationKey().ifPresent(context.updatedEntries()::add);
}));
}

private static void finalizeCompoundEdit(MergeContext context) {
if (!context.updatedEntries().isEmpty()) {
Platform.runLater(() -> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary?

context.compoundEdit().end();
context.libraryTab().getUndoManager().addEdit(context.compoundEdit());
});
}
}

private static void updateProgress(int currentIndex, int totalEntries, BackgroundTask<?> task) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static method are a strong indicator for a bad design.

Please, simply create a new class inherting from BackgroundTask. See org.jabref.gui.externalfiles.UnlinkedFilesCrawler for an example.

Platform.runLater(() -> {
task.updateMessage(Localization.lang("Processing entry %0 of %1", currentIndex + 1, totalEntries));
task.updateProgress(currentIndex, totalEntries);
});
}

private static void handleSuccess(List<String> updatedEntries, MergeContext context) {
Platform.runLater(() -> {
String message = updatedEntries.isEmpty()
? Localization.lang("No updates found.")
: Localization.lang("Batch update successful. %0 entries updated.",
updatedEntries.size());

LOGGER.debug("Batch update completed. {}", message);
context.notificationService().notify(message);
});
}

private static void handleFailure(Exception ex, NotificationService notificationService) {
LOGGER.error("Error during fetch and merge", ex);
Platform.runLater(() ->
notificationService.notify(
Localization.lang("Error while fetching and merging entries: %0", ex.getMessage())
)
);
}

private record MergeContext(
LibraryTab libraryTab,
List<BibEntry> entries,
MergingIdBasedFetcher fetcher,
NamedCompound compoundEdit,
List<String> updatedEntries,
NotificationService notificationService
) {
MergeContext(LibraryTab libraryTab, List<BibEntry> entries, MergingIdBasedFetcher fetcher, NotificationService notificationService) {
this(
libraryTab,
entries,
fetcher,
new NamedCompound(Localization.lang("Merge entries")),
Collections.synchronizedList(new ArrayList<>()),
notificationService
);
}
}
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No empty line before "package"

package org.jabref.gui.mergeentries;

import java.util.Arrays;
Expand Down
103 changes: 103 additions & 0 deletions src/main/java/org/jabref/gui/mergeentries/MergeEntriesHelper.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No empty line before package

package org.jabref.gui.mergeentries;

import java.util.LinkedHashSet;
import java.util.Optional;
import java.util.Set;

import org.jabref.gui.undo.NamedCompound;
import org.jabref.gui.undo.UndoableChangeType;
import org.jabref.gui.undo.UndoableFieldChange;
import org.jabref.model.entry.BibEntry;
import org.jabref.model.entry.field.Field;
import org.jabref.model.entry.field.FieldFactory;
import org.jabref.model.entry.types.EntryType;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Helper class for merging bibliography entries with undo support.
* Source entry data is merged into the library entry, with longer field values preferred
* and obsolete fields removed.
*/
public final class MergeEntriesHelper {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no test for this. Please add a JUNit test (ctrl+shift+t in IntelliJ gets you started). -- Reading a unit test makes it easier to check whether the logic s correct.


private static final Logger LOGGER = LoggerFactory.getLogger(MergeEntriesHelper.class);

private MergeEntriesHelper() {
}

/**
* Merges two BibEntry objects with undo support.
*
* @param entryFromLibrary The entry to be updated (target, from the library)
* @param entryFromFetcher The entry containing new information (source, from the fetcher)
* @param undoManager Compound edit to collect undo information
*/
public static void mergeEntries(BibEntry entryFromLibrary, BibEntry entryFromFetcher, NamedCompound undoManager) {
LOGGER.debug("Entry from library: {}", entryFromLibrary);
LOGGER.debug("Entry from fetcher: {}", entryFromFetcher);

mergeEntryType(entryFromLibrary, entryFromFetcher, undoManager);
mergeFields(entryFromLibrary, entryFromFetcher, undoManager);
removeFieldsNotPresentInFetcher(entryFromLibrary, entryFromFetcher, undoManager);
}

private static void mergeEntryType(BibEntry entryFromLibrary, BibEntry entryFromFetcher, NamedCompound undoManager) {
EntryType libraryType = entryFromLibrary.getType();
EntryType fetcherType = entryFromFetcher.getType();

if (!libraryType.equals(fetcherType)) {
LOGGER.debug("Updating type {} -> {}", libraryType, fetcherType);
entryFromLibrary.setType(fetcherType);
undoManager.addEdit(new UndoableChangeType(entryFromLibrary, libraryType, fetcherType));
}
}

private static void mergeFields(BibEntry entryFromLibrary, BibEntry entryFromFetcher, NamedCompound undoManager) {
Set<Field> allFields = new LinkedHashSet<>();
allFields.addAll(entryFromFetcher.getFields());
allFields.addAll(entryFromLibrary.getFields());

for (Field field : allFields) {
Optional<String> fetcherValue = entryFromFetcher.getField(field);
Optional<String> libraryValue = entryFromLibrary.getField(field);

fetcherValue.ifPresent(newValue -> {
if (shouldUpdateField(libraryValue, newValue)) {
LOGGER.debug("Updating field {}: {} -> {}", field, libraryValue.orElse(null), newValue);
entryFromLibrary.setField(field, newValue);
undoManager.addEdit(new UndoableFieldChange(entryFromLibrary, field, libraryValue.orElse(null), newValue));
}
});
}
}

private static boolean shouldUpdateField(Optional<String> libraryValue, String fetcherValue) {
return libraryValue.map(value -> fetcherValue.length() > value.length())
.orElse(true);
}

/**
* Removes fields from the library entry that are not present in the fetcher entry.
* This ensures the merged entry only contains fields that are considered current
* according to the fetched data.
*/
private static void removeFieldsNotPresentInFetcher(BibEntry entryFromLibrary, BibEntry entryFromFetcher, NamedCompound undoManager) {
Set<Field> obsoleteFields = new LinkedHashSet<>(entryFromLibrary.getFields());
obsoleteFields.removeAll(entryFromFetcher.getFields());

for (Field field : obsoleteFields) {
if (FieldFactory.isInternalField(field)) {
continue;
}

entryFromLibrary.getField(field).ifPresent(value -> {
LOGGER.debug("Removing obsolete field {} with value {}", field, value);
entryFromLibrary.clearField(field);
undoManager.addEdit(new UndoableFieldChange(entryFromLibrary, field, value, null));
});
}
}
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No empty line at the beginning of a file (not sure why this file was touched by you)

package org.jabref.gui.mergeentries;

import javax.swing.undo.UndoManager;
Expand Down
Loading