feat: introduce trigger (#482)

* feat(plugins): add FetchPluginReadme endpoint and update launch configurations

* feat: add PluginReadme database model

* feat: implement readme extracting and storage

* feat: implement readme endpoint

* feat: add plugin asset extraction endpoint with caching support

* Implement trigger functionality and clean up dynamic select code

- Added new trigger-related access types and actions in access.go.
- Introduced new HTTP routes for trigger operations in http_server.gen.go.
- Updated plugin declaration to include triggers in plugin_entities.
- Removed unused dynamic select service and controller files.
- Enhanced message handling in debugging_runtime to support trigger registration.

This update enhances the plugin system by integrating trigger capabilities while cleaning up legacy code.

* Refactor trigger-related types and enhance dynamic select functionality

- Updated TriggerProviderIdentity and TriggerProviderConfiguration to improve structure and validation.
- Renamed TriggerConfiguration to TriggerDeclaration for clarity.
- Added CredentialType to RequestDynamicParameterSelect for better request handling.
- Enhanced PluginDecoderHelper to read and unmarshal trigger files correctly.

These changes streamline the trigger system and improve the overall request handling in the plugin architecture.

* Add trigger functionality and enhance database integration

- Introduced TriggerInstallation model for managing trigger installations in the database.
- Updated autoMigrate function to include trigger installations in the migration process.
- Added new HTTP routes for listing and retrieving triggers in the HTTP server.
- Implemented ListTriggers and GetTrigger controller functions for handling trigger requests.
- Enhanced plugin management functions to create, update, and delete trigger installations during plugin lifecycle events.

These changes integrate trigger capabilities into the system, improving the overall plugin functionality and management.

* feat: add remapping for trigger icons in MediaBucket

- Enhanced the RemapAssets function to include remapping of trigger identity icons and dark icons.
- Added error handling for remapping failures to ensure robust asset management.

These changes improve the handling of trigger assets within the plugin system, ensuring icons are correctly remapped as needed.

* feat: add Multiple field to TriggerParameter for enhanced configuration

- Introduced a new Multiple field in the TriggerParameter struct to allow for multiple values in trigger configurations.
- This addition improves the flexibility of trigger parameters within the plugin system.

These changes enhance the capability of trigger parameters, enabling more complex configurations.

* feat: add Multiple field to ProviderConfig for enhanced configuration

- Introduced a new Multiple field in the ProviderConfig struct to allow for multiple values in provider configurations.
- This addition improves the flexibility of provider options within the plugin system.

These changes enhance the capability of provider configurations, enabling more complex setups.

* fix(plugin): update validation error messages in ManifestValidate method

- Enhanced error messages in the ManifestValidate function to include 'trigger' in the validation checks for plugin declarations.
- Updated logic to ensure that all relevant fields are considered when validating the presence of mutually exclusive parameters.

* feat(trigger): add CHECKBOX parameter type to plugin entities  and refactor the trigger provider strcuture

- Introduced a new CHECKBOX parameter type in constant.go for plugin entities.
- Updated tool_declaration.go and trigger_declaration.go to include TOOL_PARAMETER_TYPE_CHECKBOX and TRIGGER_PARAMETER_TYPE_CHECKBOX respectively.
- Enhanced validation logic to accommodate the new CHECKBOX type in parameter checks.

* fix(trigger): update SubscriptionSchema validation in TriggerProviderDeclaration

- Changed SubscriptionSchema validation from 'omitempty' to 'required' in TriggerProviderDeclaration to ensure it is always provided.
- Updated SubscriptionConstructor field to be a pointer to allow for optional inclusion in the trigger provider configuration.

* fix(trigger): rename ParametersSchema to Parameters in SubscriptionConstructor

- Updated the SubscriptionConstructor struct to rename the ParametersSchema field to Parameters for consistency.
- Adjusted related JSON and YAML marshaling logic to reflect the new field name, ensuring proper handling of trigger parameters.

* refactor(trigger): enhance YAML unmarshalling for SubscriptionConstructor and SubscriptionSchema

- Introduced a new helper function to convert YAML nodes to ProviderConfig lists, improving the handling of subscription_schema and credentials_schema.
- Updated the UnmarshalYAML method to utilize the new function, simplifying the logic for parsing different YAML formats.
- Ensured proper initialization of SubscriptionConstructor fields to prevent nil pointer dereferences.

* fix(trigger): update SubscriptionConstructor validation in TriggerProviderDeclaration

- Changed the validation for SubscriptionConstructor in TriggerProviderDeclaration from 'omitempty,dive' to 'omitempty' to simplify the validation logic.
- Ensured that the SubscriptionConstructor field remains optional while maintaining its intended functionality.

* refactor(trigger): rename Trigger to Event in plugin entities and related structures

- Updated the naming conventions in trigger_declaration.go to replace 'Trigger' with 'Event' for better clarity and consistency.
- Adjusted related types, validation functions, and unmarshalling logic to reflect the new 'Event' terminology.
- Ensured that all references to triggers in the codebase are updated to events, including in the SubscriptionConstructor and response structures.

* refactor(trigger): rename TriggerInvoke to TriggerInvokeEvent and update related structures

- Renamed TriggerInvoke function and associated request/response types to TriggerInvokeEvent for improved clarity.
- Updated routing and controller methods to reflect the new naming convention.
- Ensured all references to the trigger invoke functionality are consistent with the new event terminology.

* refactor(trigger): remove Subscription struct from trigger_declaration.go and update TriggerDispatchEventRequest

- Removed the Subscription struct from trigger_declaration.go to streamline the codebase.
- Added Credentials field to TriggerDispatchEventRequest for enhanced functionality and clarity.
- Ensured that the changes maintain consistency with the existing naming conventions and structures.

* fix(trigger): improve nil checks for SubscriptionConstructor in TriggerProviderDeclaration

- Added nil checks for SubscriptionConstructor before accessing its fields to prevent potential nil pointer dereferences.
- Ensured that Parameters and CredentialsSchema are initialized only if SubscriptionConstructor is not nil, enhancing code robustness.

* fix(plugin): add recovery mechanism in OnTraffic to handle panics

- Introduced a deferred function in OnTraffic to recover from panics, logging the error and stack trace for better debugging.
- This enhancement improves the stability of the DifyServer by preventing crashes due to unexpected runtime errors.

* feat(trigger): add Subscription field to TriggerInvokeEventRequest

- Introduced a new Subscription field in the TriggerInvokeEventRequest struct to accommodate subscription data.
- Ensured the field is marked as required, enhancing the request's functionality and validation requirements.

* refactor(event): simplify EventDescription structure in EventDeclaration

- Removed the EventDescription struct and replaced it with a direct I18nObject field in EventDeclaration.
- This change streamlines the event configuration by reducing complexity while maintaining required validation for the description.

* feat(trigger): add UserID field to TriggerDispatchEventResponse

- Introduced a new UserID field in the TriggerDispatchEventResponse struct to include user identification in the response.
- The field is marked as optional, enhancing the response's flexibility while maintaining existing functionality.

* feat: add payload to TriggerDispatchEventResponse

* fix

* feat(trigger): update TriggerDispatchEventResponse structure

* fix: avoid path collusion

* fix: missing )

* fix: query param

* fix: form param

* fix: remove redundant dynamic parameter access type

* fix: remove dynamic parameter access type from validation

* Update internal/server/controllers/plugins.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: Stream <[email protected]>
Co-authored-by: Yeuoly <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Author: 4 people

.gitignore

@@ -14,4 +14,4 @@ working
 cwd/
 bin/
 .venv/
-integration_test_cwd/
+integration_test_cwd/

internal/core/plugin_daemon/access_types/access.go

@@ -3,13 +3,14 @@ package access_types
 type PluginAccessType string
 
 const (
-	PLUGIN_ACCESS_TYPE_TOOL           PluginAccessType = "tool"
-	PLUGIN_ACCESS_TYPE_MODEL          PluginAccessType = "model"
-	PLUGIN_ACCESS_TYPE_ENDPOINT       PluginAccessType = "endpoint"
-	PLUGIN_ACCESS_TYPE_AGENT_STRATEGY PluginAccessType = "agent_strategy"
-	PLUGIN_ACCESS_TYPE_OAUTH          PluginAccessType = "oauth"
-	PLUGIN_ACCESS_TYPE_DATASOURCE     PluginAccessType = "datasource"
+	PLUGIN_ACCESS_TYPE_TOOL              PluginAccessType = "tool"
+	PLUGIN_ACCESS_TYPE_MODEL             PluginAccessType = "model"
+	PLUGIN_ACCESS_TYPE_ENDPOINT          PluginAccessType = "endpoint"
+	PLUGIN_ACCESS_TYPE_AGENT_STRATEGY    PluginAccessType = "agent_strategy"
+	PLUGIN_ACCESS_TYPE_OAUTH             PluginAccessType = "oauth"
+	PLUGIN_ACCESS_TYPE_DATASOURCE        PluginAccessType = "datasource"
 	PLUGIN_ACCESS_TYPE_DYNAMIC_PARAMETER PluginAccessType = "dynamic_parameter"
+	PLUGIN_ACCESS_TYPE_TRIGGER           PluginAccessType = "trigger"
 )
 
 func (p PluginAccessType) IsValid() bool {
@@ -18,8 +19,9 @@ func (p PluginAccessType) IsValid() bool {
 		p == PLUGIN_ACCESS_TYPE_ENDPOINT ||
 		p == PLUGIN_ACCESS_TYPE_AGENT_STRATEGY ||
 		p == PLUGIN_ACCESS_TYPE_OAUTH ||
+		p == PLUGIN_ACCESS_TYPE_DATASOURCE ||
 		p == PLUGIN_ACCESS_TYPE_DYNAMIC_PARAMETER ||
-		p == PLUGIN_ACCESS_TYPE_DATASOURCE
+		p == PLUGIN_ACCESS_TYPE_TRIGGER
 }
 
 type PluginAccessAction string
@@ -51,7 +53,13 @@ const (
 	PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DOCUMENT_DATASOURCE_GET_PAGE_CONTENT PluginAccessAction = "invoke_online_document_datasource_get_page_content"
 	PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DRIVE_BROWSE_FILES                   PluginAccessAction = "invoke_online_drive_browse_files"
 	PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DRIVE_DOWNLOAD_FILE                  PluginAccessAction = "invoke_online_drive_download_file"
-	PLUGIN_ACCESS_ACTION_DYNAMIC_PARAMETER_FETCH_OPTIONS PluginAccessAction = "fetch_parameter_options"
+	PLUGIN_ACCESS_ACTION_DYNAMIC_PARAMETER_FETCH_OPTIONS                    PluginAccessAction = "fetch_parameter_options"
+	PLUGIN_ACCESS_ACTION_INVOKE_TRIGGER_EVENT                               PluginAccessAction = "invoke_trigger_event"
+	PLUGIN_ACCESS_ACTION_DISPATCH_TRIGGER_EVENT                             PluginAccessAction = "dispatch_trigger_event"
+	PLUGIN_ACCESS_ACTION_SUBSCRIBE_TRIGGER                                  PluginAccessAction = "subscribe_trigger"
+	PLUGIN_ACCESS_ACTION_UNSUBSCRIBE_TRIGGER                                PluginAccessAction = "unsubscribe_trigger"
+	PLUGIN_ACCESS_ACTION_REFRESH_TRIGGER                                    PluginAccessAction = "refresh_trigger"
+	PLUGIN_ACCESS_ACTION_VALIDATE_TRIGGER_CREDENTIALS                       PluginAccessAction = "validate_trigger_credentials"
 )
 
 func (p PluginAccessAction) IsValid() bool {
@@ -81,5 +89,11 @@ func (p PluginAccessAction) IsValid() bool {
 		p == PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DOCUMENT_DATASOURCE_GET_PAGES ||
 		p == PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DOCUMENT_DATASOURCE_GET_PAGE_CONTENT ||
 		p == PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DRIVE_BROWSE_FILES ||
-		p == PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DRIVE_DOWNLOAD_FILE
+		p == PLUGIN_ACCESS_ACTION_INVOKE_ONLINE_DRIVE_DOWNLOAD_FILE ||
+		p == PLUGIN_ACCESS_ACTION_INVOKE_TRIGGER_EVENT ||
+		p == PLUGIN_ACCESS_ACTION_DISPATCH_TRIGGER_EVENT ||
+		p == PLUGIN_ACCESS_ACTION_SUBSCRIBE_TRIGGER ||
+		p == PLUGIN_ACCESS_ACTION_UNSUBSCRIBE_TRIGGER ||
+		p == PLUGIN_ACCESS_ACTION_REFRESH_TRIGGER ||
+		p == PLUGIN_ACCESS_ACTION_VALIDATE_TRIGGER_CREDENTIALS
 }

internal/core/plugin_daemon/dynamic_select.gen.go renamed to internal/core/plugin_daemon/dynamic_parameter.gen.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"encoding/base64"
 	"fmt"
+	"runtime/debug"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -155,6 +156,13 @@ func (s *DifyServer) OnShutdown(c gnet.Engine) {
 }
 
 func (s *DifyServer) OnTraffic(c gnet.Conn) (action gnet.Action) {
+	defer func() {
+		if r := recover(); r != nil {
+			traceback := string(debug.Stack())
+			log.Error("panic in OnTraffic: %v\n%s", r, traceback)
+		}
+	}()
+
 	codec := c.Context().(*codec)
 	messages, err := codec.Decode(c)
 	if err != nil {
@@ -276,7 +284,8 @@ func (s *DifyServer) onMessage(runtime *RemotePluginRuntime, message []byte) {
 				!runtime.endpointsRegistrationTransferred &&
 				!runtime.toolsRegistrationTransferred &&
 				!runtime.agentStrategyRegistrationTransferred &&
-				!runtime.datasourceRegistrationTransferred {
+				!runtime.datasourceRegistrationTransferred &&
+				!runtime.triggersRegistrationTransferred {
 				closeConn([]byte("no registration transferred, cannot initialize\n"))
 				return
 			}
@@ -442,6 +451,24 @@ func (s *DifyServer) onMessage(runtime *RemotePluginRuntime, message []byte) {
 				declaration.Datasource = &datasources[0]
 				runtime.Config = declaration
 			}
+		} else if registerPayload.Type == plugin_entities.REGISTER_EVENT_TYPE_TRIGGER_DECLARATION {
+			if runtime.triggersRegistrationTransferred {
+				return
+			}
+
+			triggers, err := parser.UnmarshalJsonBytes2Slice[plugin_entities.TriggerProviderDeclaration](registerPayload.Data)
+			if err != nil {
+				closeConn([]byte(fmt.Sprintf("triggers register failed, invalid triggers declaration: %v\n", err)))
+				return
+			}
+
+			runtime.triggersRegistrationTransferred = true
+
+			if len(triggers) > 0 {
+				declaration := runtime.Config
+				declaration.Trigger = &triggers[0]
+				runtime.Config = declaration
+			}
 		}
 	} else {
 		// continue handle messages if handshake completed

internal/core/plugin_daemon/trigger.gen.go

@@ -59,6 +59,7 @@ type RemotePluginRuntime struct {
 	endpointsRegistrationTransferred     bool
 	agentStrategyRegistrationTransferred bool
 	datasourceRegistrationTransferred    bool
+	triggersRegistrationTransferred      bool
 	assetsTransferred                    bool
 
 	// tenant id

internal/core/plugin_manager/debugging_runtime/hooks.go

@@ -6,6 +6,7 @@ import (
 	"os"
 	"strings"
 
+	lru "github.com/hashicorp/golang-lru/v2"
 	"github.com/langgenius/dify-cloud-kit/oss"
 	"github.com/langgenius/dify-plugin-daemon/internal/core/dify_invocation"
 	"github.com/langgenius/dify-plugin-daemon/internal/core/dify_invocation/real"
@@ -79,6 +80,15 @@ func InitGlobalManager(oss oss.OSS, configuration *app.Config) *PluginManager {
 		config:           configuration,
 	}
 
+	// initialize plugin asset cache
+	if pluginAssetCache == nil {
+		c, err := lru.New[string, []byte](256)
+		if err != nil {
+			log.Panic("init plugin asset cache failed: %s", err.Error())
+		}
+		pluginAssetCache = c
+	}
+
 	return manager
 }
 
@@ -259,3 +269,38 @@ func (p *PluginManager) GetDeclaration(
 		plugin_unique_identifier, runtime_type,
 	)
 }
+
+var (
+	pluginAssetCache *lru.Cache[string, []byte]
+)
+
+func pluginAssetCacheKey(
+	pluginUniqueIdentifier plugin_entities.PluginUniqueIdentifier,
+	path string,
+) string {
+	return fmt.Sprintf("%s/%s", pluginUniqueIdentifier.String(), path)
+}
+func (p *PluginManager) ExtractPluginAsset(
+	pluginUniqueIdentifier plugin_entities.PluginUniqueIdentifier,
+	path string,
+) ([]byte, error) {
+	key := pluginAssetCacheKey(pluginUniqueIdentifier, path)
+	cached, ok := pluginAssetCache.Get(key)
+	if ok {
+		return cached, nil
+	}
+	pkgBytes, err := manager.GetPackage(pluginUniqueIdentifier)
+	if err != nil {
+		return nil, err
+	}
+	zipDecoder, err := decoder.NewZipPluginDecoder(pkgBytes)
+	if err != nil {
+		return nil, err
+	}
+	assets, err := zipDecoder.Assets()
+	if err != nil {
+		return nil, err
+	}
+	pluginAssetCache.Add(key, assets[path])
+	return assets[path], nil
+}

internal/core/plugin_manager/debugging_runtime/type.go

@@ -124,6 +124,22 @@ func (m *MediaBucket) RemapAssets(declaration *plugin_entities.PluginDeclaration
 		}
 	}
 
+	if declaration.Trigger != nil {
+		if declaration.Trigger.Identity.Icon != "" {
+			declaration.Trigger.Identity.Icon, err = remap(declaration.Trigger.Identity.Icon)
+			if err != nil {
+				return nil, errors.Join(err, fmt.Errorf("failed to remap trigger icon"))
+			}
+		}
+
+		if declaration.Trigger.Identity.IconDark != "" {
+			declaration.Trigger.Identity.IconDark, err = remap(declaration.Trigger.Identity.IconDark)
+			if err != nil {
+				return nil, errors.Join(err, fmt.Errorf("failed to remap trigger icon dark"))
+			}
+		}
+	}
+
 	if declaration.Icon != "" {
 		declaration.Icon, err = remap(declaration.Icon)
 		if err != nil {

internal/core/plugin_manager/manager.go

@@ -21,6 +21,8 @@ func autoMigrate() error {
 		models.InstallTask{},
 		models.TenantStorage{},
 		models.AgentStrategyInstallation{},
+		models.TriggerInstallation{},
+		models.PluginReadme{},
 	)
 
 	if err != nil {
@@ -45,6 +47,7 @@ func autoMigrate() error {
 		"tool_installations",
 		"ai_model_installations",
 		"agent_strategy_installations",
+		"trigger_installations",
 	}
 
 	for _, table := range tables {