Metrics mismatch between response codes and actions

[robbavey]

The Elasticsearch output bulk_request metrics are intended to show the results of interactions with Elasticsearch, measuring response codes returned from the bulk request, and measuring whether a bulk request was a "success" (all actions in the bulk request processed successfully), a "failure" (none of the documents processed), or "with_errors" (bulk request succeeded, but individual actions in the bulk request generated errors).

Ordinarily, we would expect the total number of responses to match, or at least very closely match, the total of failures', successes, and with_errors`. However there are situations where these numbers do not match:

"bulk_requests": {
              "responses": {
                "200": 2514,
                "429": 1
              },
              "with_errors": 274,
              "failures": 1,
              "successes": 1602
            },
          }

We should fix these metrics, so that we have correlations to understand what is happening for each of the responses from Elasticsearch.

[robbavey]

Note:

From a conversation, @yaauie posits:

The cleanest path I can see to the mismatch is the multiple bulk requests when a batch is oversized; each bulk will increment the responses.200, but the responses from those separate bulks are then zipped back together, and the batch-level successes/failures/with_errors is incremented from the composite response.