Skip to content

Conversation

@quicksandznzn
Copy link
Contributor

Description

Fixes: #448

Summary

Add a CLI to migrate plugin storage from local filesystem to a cloud object store, plus bilingual docs and README links.

Motivation

  • Simplify moving existing local plugin data (packages, assets, installed bundles) to cloud storage.
  • Standardize on the same OSS abstraction used by the daemon.
  • Provide safe, idempotent migration with preview mode.

Key Changes

  • New CLI: cmd/migrate_storage/main.go
    • Recursively migrates three categories: plugin_packages, assets, plugin
    • Idempotent uploads: skips destination objects that already exist
    • Flags:
      • --dry-run (preview only)
      • --only packages,assets,installed (scope control)
      • --source-root (override local root)
    • Safety: restricts to local → cloud only (rejects local → local)
    • Fix: corrects prefix handling when listing/loading/saving under a given path
  • Documentation (bilingual):
    • English: docs/storage-migration/guide.md
    • 中文: docs/storage-migration/guide_cn.md
  • README:
    • New “Storage Migration” section with links to both guides

How To Use

  • Configure environment (or .env) with the same variables used by the daemon:
    • PLUGIN_STORAGE_TYPE (must be non-local)
    • PLUGIN_STORAGE_OSS_BUCKET
    • Cloud provider credentials (AWS/COS/OSS/Azure/GCS/OBS/TOS)
    • PLUGIN_STORAGE_LOCAL_ROOT (defaults to ./storage)
  • Run:
    # Preview
    go run ./cmd/migrate_storage --dry-run
    
    # Execute
    go run ./cmd/migrate_storage
    
    # Or build and run
    go build -o migrate-storage ./cmd/migrate_storage
    ./migrate-storage --only packages,assets,installed

Notes and Constraints

  • Only local → cloud is supported; the tool exits if PLUGIN_STORAGE_TYPE=local.
  • Requires network access and write permission to the destination bucket/container.
  • No database changes are made; does not alter install_type or installation records.

File References

  • cmd/migrate_storage/main.go
  • docs/storage-migration/guide.md
  • docs/storage-migration/guide_cn.md
  • README.md (adds “Storage Migration” section)

Known Limitations

  • For very large datasets, many head/list operations can be slower on some providers.
  • Migration operates single-process; concurrency tuning can be considered later if needed.

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Performance improvement
  • Other

Essential Checklist

Testing

  • I have tested the changes locally and confirmed they work as expected
  • I have added unit tests where necessary and they pass successfully

Bug Fix (if applicable)

  • I have used GitHub syntax to close the related issue (e.g., Fixes #123 or Closes #123)

Additional Information

1. Migration Command Screenshot

Migration Command

2. Cloud OSS Screenshot

Cloud OSS Configuration

3. Change Plugin Storage Type

Plugin Storage Type Configuration

4. Restart Plugin Daemon

I deleted the cwd folder to test the initialization of loading the plugin from the cloud. The plugin daemon runs successfully

Plugin Daemon Restart

5. Test the Plugin in Dify

Plugin Testing in Dify

Please provide any additional context that would help reviewers understand the changes.

@quicksandznzn
Copy link
Contributor Author

@Yeuoly hi

This PR centralizes OSS construction behind a small factory to remove duplicated config wiring and keep provider options consistent. No behavior change; just refactoring for maintainability.

Changes

  • Add internal/storagefactory/factory.go exposing storagefactory.New(t string, cfg *app.Config) (oss.OSS, error) and centralizing app.Configoss.OSSArgs mapping.
  • Update internal/server/server.go to construct storage via storagefactory.New(...) inside initOSS.
  • Update cmd/migrate_storage/main.go to build both source and destination storage via storagefactory.New(...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to migrate storage methods, local migration to OSS

2 participants