"unspecified protocol error" when using GCS as S3 backend

[bcspragu]

Hi there!

I'm setting up a Stalwart deployment, using GCS's S3 interop as a backend. I have a config like this:

[store."s3"]
type = "s3"
endpoint = "https://storage.googleapis.com"
bucket = "<bucket>"
region = "auto"
access-key = "<access key>"
secret-key = "<secret key>"
timeout = "30s"
disable = false

When I receive an email, I see the following error in the logs

ERROR jmap::blob::upload: Failed to store blob. event="error" context="put_blob" error=InternalError("S3 error: reqwest: error sending request for url (https://storage.googleapis.com/<bucket>/<obj name>): http2 error: stream error received: unspecific protocol error detected")

I looked through the S3Store implementation and didn't see anything obvious, but when I test this process manually using both Go and Rust clients running on the same machine as Stalwart (where it is failing), they work fine.

Googling around, I see this error occurring variously with Google services, seems to be related to rust-tls, but I'm not sure how that comes into play here.

Rust example code

#[tokio::main]
async fn main() {
    let bucket_name = "<bucket>";
    let google_access_key_id = "<access key>";
    let google_access_key_secret = "<secret key>";

    let bucket = s3::bucket::Bucket::new(
        bucket_name,
        s3::region::Region::Custom {
            region: "auto".to_owned(),
            endpoint: "https://storage.googleapis.com".to_owned(),
        },
        s3::creds::Credentials::new(
            Some(google_access_key_id),
            Some(google_access_key_secret),
            None,
            None,
            None,
        )
        .unwrap(),
    )
    .unwrap()
    .with_path_style();

    bucket
        .put_object("rust-test", "some data".as_bytes())
        .await
        .unwrap();
}

Go example code

package main

import (
	"context"
	"fmt"
	"log"
	"strings"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	bucketName := "<bucket>"
	googleAccessKeyID := "<access key>"
	googleAccessKeySecret := "<secret key>"

	sess := session.Must(session.NewSession(&aws.Config{
		Region:      aws.String("auto"),
		Endpoint:    aws.String("https://storage.googleapis.com"),
		Credentials: credentials.NewStaticCredentials(googleAccessKeyID, googleAccessKeySecret, ""),
	}))

	client := s3.New(sess)
	ctx := context.Background()

	fmt.Println(client.PutObjectWithContext(ctx, &s3.PutObjectInput{
		Body:   strings.NewReader("hello world!"),
		Bucket: aws.String(bucketName),
		Key:    aws.String("test-obj"),
	}))

	return nil
}

[mdecimus]

Are you using Rust S3 with the following features?

rust-s3 = { version = "0.33.0", default-features = false, features = ["tokio-rustls-tls"] }

[bcspragu]

I was able to hack around this with tokio-native-tls, commit here

[mdecimus]

Please, can you open an issue on the rust S3 crate repository including the above code snippet? Unfortunately we can't use native tls in Stalwart as it will break compatibility with some platforms.

[bcspragu]

Understood, filed upstream. My guess is that the actual issue is in hyper, but it's worth leaving a breadcrumb trail for other folks who might hit this

[mdecimus]

Thank you! In the meantime, in case you haven't done this already, you can recompile Stalwart and change the rust-s3 features to native-tls.

[bcspragu]

Indeed, in the above linked commit I switched to tokio-native-tls and that does indeed work (after updating Debian to Bookworm and pulling in libssl3 in the Docker image)