Sending a stream message from client succeeds, even if no network is available

[KaneCheshire]

Describe the bug

While investigating how grpc-swift handles different scenarios while client streams are open (like backgrounding the app, or during a poor/dropped network connection), I encountered an issue that seems like a bug to me.

While a client stream is open, if the network is disabled on the iOS device, before the next keepalive ping fails, sending messages to the stream returns a success straight away.

This means there's no way for the client to know whether a message was actually successfully sent because grpc-swift is reporting "ok" regardless of whether it was actually sent. If the message can't actually be sent to the server (i.e. because there's no network) then I'd expect the client to be able to know about that reliably, just like any regular HTTP call would.

This is using grpc-swift 1.7.3 as a Swift package, on iOS 15.2.1

To reproduce

1. Open a ClientStreamingCall with a server.

• I used a server hosted on a separate Mac to my development machine on my local network.

3. Send a message to the stream and verify that the message is sent and received by the server ok.
4. Drop the network connection somehow (i.e. by turning off mobile data and wifi, or simulate a poor network via Xcode).

• It's important at this stage not to let the app become suspended (inactive is fine), because it does report the connection as timed out if the app is put into the background for too long. So to achieve this, you can use Control Center to manipulate the device connectivity, or just use Xcode to set the network condition.

5. Try to send another message to the stream.

• Note how the Future still completes with success immediately, despite it not being possible to send. The server doesn't receive the message, unless the network is re-established within a minute or so.

Expected behaviour

I would expect the Future to indicate that the message to the stream cannot be sent, otherwise there's no way to verify that a call has succeeded reliably before sending the next message.

Additional information

I realise that there is a mechanism to make the connection fail faster by modifying GRPCChannelPool.Configuation's keepalive config, setting the timeout and interval to a smaller number, which at least makes the connection drop sooner than ~60 seconds after the message is sent, but this seems like a plaster(🇬🇧)/band-aid(🇺🇸) over the underlying problem and still doesn't provide a truly reliable and robust way of knowing if a message was actually sent successfully. Using this way, you'd need to wait a minimum interval between sending messages to ensure that the connection is still alive before sending the next one, which means setting the keepalive interval to a very small number.

The same behaviour exists even if just on a very poor network, if 100% packet loss is occurring while still having an apparent connection then the Future is completing immediately despite it not actually sending to the server. In this case, we don't really get a connection timeout either so I guess the keepalive pings are hanging or something similar.

The 100% packet loss issue is more of a concern than, say, if the device was in airplane mode. Because it's very common for users to have a data signal but a poor data throughput (at least here in the UK it's very common), robust checking of the state of a message send is crucial. An example of this could just be a congested network, or someone on a train going through a tunnel.

For some extra info, I had the server (also using grpc-swift) hosted on a machine on my local network under port 8888 and it receives RPCs fine, so I don't think it's an issue with the server implementation.

Here's how I set up the channel client-side:

try! GRPCChannelPool.with(target: .host("computername.local", port: 8888), transportSecurity: .plaintext, eventLoopGroup: PlatformSupport.makeEventLoopGroup(loopCount: 1)) { [weak self] in
  $0.errorDelegate = self
}

And the general code for sending messages client-side:

let stream = service.openAppInfoClientStream(callOptions: nil)
let info = SomeInfo.with { $0.foo = "bar" }
stream.sendMessage(info).whenComplete { status in
  switch status {
  case .success:
    print("Successfully sent")
  case .failure(let error):
    print("Failed to send", error)
  }
}

Unlikely to be related by for completeness I actually iterate over an AsyncStream of app states to send to the stream, so when the app becomes inactive, a message gets sent, and when the app becomes activate again, another message gets sent. This is just to make investigating a bit quicker without having to tap buttons.

let stream = service.openAppInfoClientStream(callOptions: nil)
let stateObserver = AppStateObserver() // Custom object to observe app states
let task = Task {
  for await state in stateObserver.states {
    let info = SomeInfo.with { $0.state = state }
    stream.sendMessage(info).whenComplete { status in
      // .. switch on status here
    }
  }
}

// Here we can wait for the stream state and then when the stream is closed, 
// cancel the task above so the stream states are no longer awaited
stream.status.whenComplete { result in
  switch result {
  case .success(let status):
    print(status)
  case .failure(let error):
    print(error)
  }
  task.cancel()
}

[KaneCheshire]

A small piece of extra information for completeness:

If no connection has already been made and no network is available then sending a message to a stream waits to call the Future until the connection timeout deadline has passed, and then does reliably notify that sending the message failed.

This is the same behaviour I'd expect when sending a message on an already-open connection, i.e. for some internal mechanism to wait to call the Future with success or fail until it's known that it's likely to have actually been sent.

[dannyhertz]

I am also finding this bug very frustrating! Let me know if you find any workarounds.

[glbrntt]

While a client stream is open, if the network is disabled on the iOS device, before the next keepalive ping fails, sending messages to the stream returns a success straight away.

I suspect the underlying NIOTSConnectionChannel is still open and accepting writes which is the the futures succeed before the keep alive fires and closes the channel.

Could you provide trace logs for this bug?

---

This means there's no way for the client to know whether a message was actually successfully sent because grpc-swift is reporting "ok" regardless of whether it was actually sent. If the message can't actually be sent to the server (i.e. because there's no network) then I'd expect the client to be able to know about that reliably, just like any regular HTTP call would.

gRPC can't guarantee that a message on a stream made it to the server, only that a write has been accepted to be written (i.e. gRPC thinks it can write the message, it is by no means a guarantee). One way to get this behaviour is to build additional logic into your RPC by having the server ack each request it receives.

[Lukasa]

If the message can't actually be sent to the server (i.e. because there's no network) then I'd expect the client to be able to know about that reliably, just like any regular HTTP call would.

To be very concrete, it is not possible to know this reliably. It is always possible that a connection fails in the window between the socket accepting a write and the server getting the data. There is no API surface that allows us to complete the promise only when the underlying TCP connection acknowledges the data transfer, and even if there was that API could still not confirm that the remote server had actually processed the request. Idempotency is a tricky thing.