Reconcile envoyproxy/data-plane-api and envoyproxy/envoy (#3036)

This PR implements the planned merge of envoyproxy/data-plane-api into envoyproxy/envoy as described in #2934 and https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/envoy-dev/KcVHFH-zQwQ.

Risk Level: Medium (there might be unintentional breakage of dependent builds).
Testing: CI passes. There is now an additional bazel.api do_ci.sh target to build and run API tests.

Fixes #2934.

Signed-off-by: Harvey Tuch <[email protected]>

Author: htuch

.circleci/config.yml

@@ -37,6 +37,15 @@ jobs:
      steps:
        - checkout
        - run: ci/do_circle_ci.sh bazel.tsan
+   api:
+     docker:
+       - image: *envoy-build-image
+     resource_class: xlarge
+     working_directory: /source
+     steps:
+       - checkout
+       - run: ci/do_circle_ci.sh bazel.api
+       - run: ci/api_mirror.sh
    ipv6_tests:
      machine: true
      steps:
@@ -96,8 +105,11 @@ jobs:
      steps:
        - run: sleep 30 # workaround GH sync issue
        - checkout
-       - add_ssh_keys
        - run: ci/do_circle_ci.sh docs
+       - add_ssh_keys
+       - run: docs/publish.sh
+       - store_artifacts:
+           path: generated/docs
    mac:
      macos:
        xcode: "9.3.0"
@@ -118,6 +130,7 @@ workflows:
               only: /^v.*/
       - asan
       - tsan
+      - api
       - ipv6_tests
       - coverage
       - format

BUILD

@@ -1 +1,3 @@
 licenses(["notice"])  # Apache 2
+
+exports_files(["VERSION"])

CONTRIBUTING.md

@@ -68,9 +68,8 @@ maximize the chances of your PR being merged.
 * PRs are expected to have 100% test coverage for added code. This can be verified with a coverage
   build. If your PR cannot have 100% coverage for some reason please clearly explain why when you
   open it.
-* Any PR that changes user-facing behavior **must** have associated documentation in
-  [data-plane-api](https://github.com/envoyproxy/data-plane-api/tree/master/docs) as well as
-  [release notes](https://github.com/envoyproxy/data-plane-api/blob/master/docs/root/intro/version_history.rst).
+* Any PR that changes user-facing behavior **must** have associated documentation in [docs](docs) as
+  well as [release notes](docs/root/intro/version_history.rst).
 * All code comments and documentation are expected to have proper English grammar and punctuation.
   If you are not a fluent English speaker (or a bad writer ;-)) please let us know and we will try
   to find some help but there are no guarantees.

GOVERNANCE.md

@@ -69,33 +69,27 @@
 * We do releases approximately every 3 months as described in the
   [release cadence documentation](CONTRIBUTING.md#release-cadence).
 * Decide on the somewhat arbitrary time that a release will occur.
-* Begin marshalling the ongoing PR flow in both this repo and
-  [data-plane-api](https://github.com/envoyproxy/data-plane-api). Ask maintainers to hold off
-  merging any particularly risky PRs until after the release is tagged. This is because we currently
-  don't use release branches and assume that master is RC quality at all times. At the same time,
-  try to make sure that data-plane-api doc PRs are only merged *after* the Envoy PR so that we don't
-  wind up with stale docs.
-* Do a final check of the [release notes](https://github.com/envoyproxy/data-plane-api/blob/master/docs/root/intro/version_history.rst)
-  and make any needed corrections.
-* Switch the [data-plane-api VERSION](https://github.com/envoyproxy/data-plane-api/blob/master/VERSION) from a
-  "dev" variant to a final variant. E.g., "1.6.0-dev" to "1.6.0". Get a review and merge.
-* Update the [data-plane-api SHA in Envoy](https://github.com/envoyproxy/envoy/blob/ed312500ec38876446ce8ee70a06f7cda4adc937/bazel/repository_locations.bzl#L79)
-  to the final release SHA. Get the PR approved and merge.
+* Begin marshalling the ongoing PR flow in this repo. Ask maintainers to hold off merging any
+  particularly risky PRs until after the release is tagged. This is because we currently don't use
+  release branches and assume that master is RC quality at all times.
+* Do a final check of the [release notes](docs/root/intro/version_history.rst) and make any needed
+  corrections.
+* Switch the [VERSION](VERSION) from a "dev" variant to a final variant. E.g., "1.6.0-dev" to
+  "1.6.0". Get a review and merge.
 * **Wait for tests to pass on master.**
 * Create a [tagged release](https://github.com/envoyproxy/envoy/releases). The release should
-  start with "v" and be followed by the version number. E.g., "v1.6.0". **This must match
-  the [data-plane-api VERSION](https://github.com/envoyproxy/data-plane-api/blob/master/VERSION).**
+  start with "v" and be followed by the version number. E.g., "v1.6.0". **This must match the
+  [VERSION](VERSION).**
 * Monitor the CircleCI tag build to make sure that the final docker images get pushed along with
   the final docs. The final documentation will end up in the
   [envoyproxy.github.io repository](https://github.com/envoyproxy/envoyproxy.github.io/tree/master/docs/envoy).
 * Contact rdl@ on Slack so that the website can be updated for the new release.
 * Craft a witty/uplifting email and send it to all the email aliases including envoy-announce@.
 * If possible post on Twitter (either have Matt do it or contact caniszczyk@ on Slack and have the
   Envoy account post).
-* Do a new PR to update the [data-plane-api VERSION](https://github.com/envoyproxy/data-plane-api/blob/master/VERSION)
-  to the next development release. E.g., "1.7.0-dev". At the same time, also add a new empty section
-  to the [release notes](https://github.com/envoyproxy/data-plane-api/blob/master/docs/root/intro/version_history.rst)
-  for the following version. E.g., "1.7.0".
+* Do a new PR to update [VERSION](VERSION) to the next development release. E.g., "1.7.0-dev". At
+  the same time, also add a new empty section to the [release
+  notes](docs/root/intro/version_history.rst) for the following version. E.g., "1.7.0".
 * Update [DEPRECATED.md](DEPRECATED.md) to remove the '(pending)' comment on the current version,
   replacing it with the release date. Add a placeholder for the next version.
 

PULL_REQUEST_TEMPLATE.md

@@ -36,22 +36,17 @@ if you are unsure. A good rule of thumb is the riskier the change, the
 more comprehensive the testing should be.
 
 *Docs Changes*:
->Link to [Data Plane PR](https://github.com/envoyproxy/data-plane-api/pulls)]
-if your PR involves documentation changes. Please write in N/A if there were no
-documentation changes.
+Description of documentation changes. These should be made in [docs/root](docs/root) and/or inline
+with the API protos. Please write in N/A if there were no documentation changes.
 
 *Release Notes*:
->If this change is user impacting you **must** add a release note via a discrete PR to
-[version_history.rst](https://github.com/envoyproxy/data-plane-api/blob/master/docs/root/intro/version_history.rst).
-Please include any relevant links. Each release note should be prefixed with the relevant subsystem
-in alphabetical order (see existing examples as a guide) and include links to relevant parts of
-the documentation. Often times, this PR can be done concurrently with the main documentation PR
-for the feature. Thank you! Please write in N/A if there are no release notes.
+>If this change is user impacting you **must** add a release note to
+[version_history.rst](docs/root/intro/version_history.rst). Please include any relevant links. Each
+release note should be prefixed with the relevant subsystem in alphabetical order (see existing
+examples as a guide) and include links to relevant parts of the documentation. Thank you! Please
+write in N/A if there are no release notes.
 
 [Optional Fixes #Issue]
 
-[Optional *API Changes*:]
->Link to [Data Plane PR](https://github.com/envoyproxy/data-plane-api/pulls)]
-
 [Optional *Deprecated*:]
->Description of what is deprecated.
+>Description of what is [deprecated](DEPRECATED.md).

README.md

@@ -22,7 +22,8 @@ to find out more about the origin story and design philosophy of Envoy
 
 ## Related
 
-* [data-plane-api](https://github.com/envoyproxy/data-plane-api): v2 API definitions.
+* [data-plane-api](https://github.com/envoyproxy/data-plane-api): v2 API definitions as a standalone
+  repository. This is a read-only mirror of [api](api/).
 * [envoy-perf](https://github.com/envoyproxy/envoy-perf): Performance testing framework.
 * [envoy-filter-example](https://github.com/envoyproxy/envoy-filter-example): Example of how to add new filters
   and link to the main repository.

REPO_LAYOUT.md

@@ -5,11 +5,12 @@ as well as to clearly specify how extensions are added to the repository. The to
 are:
 
 * [.circleci/](.circleci/): Configuration for [CircleCI](https://circleci.com/gh/envoyproxy).
+* [api/](api/): Envoy data plane API.
 * [bazel/](bazel/): Configuration for Envoy's use of [Bazel](https://bazel.build/).
 * [ci/](ci/): Scripts used both during CI as well as to build Docker containers.
 * [configs/](configs/): Example Envoy configurations.
-* [docs/](docs/): Project level documentation as well as scripts for publishing final docs during
-  releases.
+* [docs/](docs/): End user facing Envoy proxy and data plane API documentation as well as scripts
+  for publishing final docs during releases.
 * [examples/](examples/): Larger Envoy examples using Docker and Docker Compose.
 * [include/](include/): "Public" interface headers for "core" Envoy. In general,
   these are almost entirely 100% abstract classes. There are a few cases of not-abstract classes in

VERSION

@@ -0,0 +1 @@
+1.7.0-dev

api/API_OVERVIEW.md

@@ -0,0 +1,128 @@
+# Envoy v2 APIs for developers
+
+## Goals
+
+This repository contains both the implemented and draft v2 JSON REST and gRPC
+[Envoy](https://github.com/envoyproxy/envoy/) APIs.
+
+Version 2 of the Envoy API evolves existing APIs and introduces new APIs to:
+
+* Allow for more advanced load balancing through load and resource utilization reporting to management servers.
+* Improve N^2 health check scalability issues by optionally offloading health checking to other Envoy instances.
+* Support Envoy deployment in edge, sidecar and middle proxy deployment models via changes to the listener model and CDS/SDS APIs.
+* Allow streaming updates from the management server on change, instead of polling APIs from Envoy. gRPC APIs will be supported
+  alongside JSON REST APIs to provide for this.
+* Ensure all Envoy runtime configuration is dynamically discoverable via API
+  calls, including listener configuration, certificates and runtime settings, which are today sourced from the filesystem. There
+  will still remain a static bootstrap configuration file that will specify items
+  unlikely to change during runtime, including the Envoy node identity, xDS
+  management server addresses, administration interface and tracing
+  configuration.
+* Revisit and where appropriate cleanup any v1 technical debt.
+
+## Status
+
+See
+[here](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview.html#status)
+for the current status of the v2 APIs.
+
+See [here](CONTRIBUTING.md#api-changes) for the v2 API change process.
+
+## Principles
+
+* [Proto3](https://developers.google.com/protocol-buffers/docs/proto3) will be
+  used to specify the canonical API. This will provide directly the gRPC API and
+  via gRPC-JSON transcoding the JSON REST API. A textual YAML input will be
+  supported for filesystem configuration files (e.g. the bootstrap file), in
+  addition to JSON, as a syntactic convenience. YAML file contents will be
+  internally converted to JSON and then follow the standard JSON-proto3
+  conversion during Envoy config ingestion.
+
+* xDS APIs should support eventual consistency. For example, if RDS references a
+  cluster that has not yet been supplied by CDS, it should be silently ignored
+  and traffic not forwarded until the CDS update occurs. Stronger consistency
+  guarantees are possible if the management server is able to sequence the xDS
+  APIs carefully (for example by using the ADS API below). By following the
+  `[CDS, EDS, LDS, RDS]` sequence for all pertinent resources, it will be
+  possible to avoid traffic outages during configuration update.
+
+* The API is primarily intended for machine generation and consumption. It is
+  expected that the management server is responsible for mapping higher level
+  configuration concepts to API responses. Similarly, static configuration
+  fragments may be generated by templating tools, etc. The APIs and tools
+  used to generate xDS configuration are beyond the scope of the definitions in
+  this repository.
+
+* REST-JSON API equivalents will be provided for the basic singleton xDS
+  subscription services CDS/EDS/LDS/RDS/SDS. Advanced APIs such as HDS, ADS and
+  EDS multi-dimensional LB will be gRPC only. This avoids having to map
+  complicated bidirectional stream semantics onto REST.
+
+* Listeners will be immutable. Any updates to a listener via LDS will require
+  the draining of existing connections for the specific bound IP/port. As a
+  result, new requests will only be guaranteed to observe the new configuration
+  after existing connections have drained or the drain timeout.
+
+* Versioning will be expressed via [proto3 package
+  namespaces](https://developers.google.com/protocol-buffers/docs/proto3#packages),
+  i.e. `package envoy.api.v2;`.
+
+* Custom components (e.g. filters, resolvers, loggers) will use a reverse DNS naming scheme,
+  e.g. `com.google.widget`, `com.lyft.widget`.
+
+## APIs
+
+Unless otherwise stated, the APIs with the same names as v1 APIs have a similar role.
+
+* [Cluster Discovery Service (CDS)](envoy/api/v2/cds.proto).
+* [Endpoint Discovery Service (EDS)](envoy/api/v2/eds.proto). This has the same role as SDS in the [v1 API](https://www.envoyproxy.io/docs/envoy/latest/api-v1/cluster_manager/sds),
+  the new name better describes what the API does in practice. Advanced global load balancing capable of utilizing N-dimensional upstream metrics is now supported.
+* [Health Discovery Service (HDS)](envoy/service/discovery/v2/hds.proto). This new API supports efficient endpoint health discovery by the management server via the Envoy instances it manages. Individual Envoy instances
+  will typically receive HDS instructions to health check a subset of all
+  endpoints. The health check subset may not be a subset of the Envoy instance's
+  EDS endpoints.
+* [Listener Discovery Service (LDS)](envoy/api/v2/lds.proto). This new API supports dynamic discovery of the listener configuration (which ports to bind to, TLS details, filter chains, etc.).
+* [Metric Service (MS)](envoy/service/metrics/v2/metrics_service.proto). This new API allows Envoy to push (stream) metrics forever for servers to consume.
+* [Rate Limit Service (RLS)](envoy/service/ratelimit/v2/rls.proto)
+* [Route Discovery Service (RDS)](envoy/api/v2/rds.proto).
+* [Secret Discovery Service (SDS)](envoy/service/discovery/v2/sds.proto).
+
+In addition to the above APIs, an aggregation API will be provided to allow for
+fine grained control over the sequencing of API updates across discovery
+services:
+
+* [Aggregated Discovery Service (ADS)](envoy/api/v2/discovery.proto). See
+  the [ADS overview](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview#aggregated-discovery-service).
+
+A protocol description for the xDS APIs is provided [here](XDS_PROTOCOL.md).
+
+## Terminology
+
+Some relevant [existing terminology](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/terminology.html) is
+repeated below and some new v2 terms introduced.
+
+* Cluster: A cluster is a group of logically similar endpoints that Envoy
+  connects to. In v2, RDS routes points to clusters, CDS provides cluster configuration and
+  Envoy discovers the cluster members via EDS.
+
+* Downstream: A downstream host connects to Envoy, sends requests, and receives responses.
+
+* Endpoint: An endpoint is an upstream host that is a member of one or more clusters. Endpoints are discovered via EDS.
+
+* Listener: A listener is a named network location (e.g., port, unix domain socket, etc.) that can be connected to by downstream clients. Envoy exposes one or more listeners that downstream hosts connect to.
+
+* Locality: A location where an Envoy instance or an endpoint runs. This includes
+  region, zone and sub-zone identification.
+
+* Management server: A logical server implementing the v2 Envoy APIs. This is not necessarily a single physical machine since it may be replicated/sharded and API serving for different xDS APIs may be implemented on different physical machines.
+
+* Region: Geographic region where a zone is located.
+
+* Sub-zone: Location within a zone where an Envoy instance or an endpoint runs.
+  This allows for multiple load balancing targets within a zone.
+
+* Upstream: An upstream host receives connections and requests from Envoy and returns responses.
+
+* xDS: CDS/EDS/HDS/LDS/RLS/RDS/SDS APIs.
+
+* Zone: Availability Zone (AZ) in AWS, Zone in GCP.

api/BUILD

@@ -0,0 +1,77 @@
+# Contributing guide
+
+## API changes
+
+All API changes should follow the [style guide](STYLE.md).
+
+API changes are regular PRs in https://github.com/envoyproxy/envoy for the API/configuration
+changes. They may be as part of a larger implementation PR. Please follow the standard Bazel and CI
+process for validating build/test sanity of `api/` before submitting a PR.
+
+*Note: New .proto files should be also included to [build.sh](https://github.com/envoyproxy/envoy/blob/master/docs/build.sh) and
+[BUILD](https://github.com/envoyproxy/envoy/blob/master/api/docs/BUILD) in order to get the RSTs generated.*
+
+## Documentation changes
+
+The Envoy project takes documentation seriously. We view it as one of the reasons the project has
+seen rapid adoption. As such, it is required that all features have complete documentation. This is
+generally going to be a combination of API documentation as well as architecture/overview
+documentation.
+
+### Building documentation locally
+
+The documentation can be built locally in the root of https://github.com/envoyproxy/envoy via:
+
+```
+docs/build.sh
+```
+
+Or to use a hermetic docker container:
+
+```
+./ci/run_envoy_docker.sh './ci/do_ci.sh docs'
+```
+
+This process builds RST documentation directly from the proto files, merges it with the static RST
+files, and then runs [Sphinx](http://www.sphinx-doc.org/en/stable/rest.html) over the entire tree to
+produce the final documentation. The generated RST files are not committed as they are regenerated
+every time the documentation is built.
+
+### Viewing documentation
+
+Once the documentation is built, it is available rooted at `generated/docs/index.html`. The
+generated RST files are also viewable in `generated/rst`.
+
+Note also that the generated documentation can be viewed in CI:
+
+1. Open docs job in CircleCI.
+2. Navigate to "artifacts" tab.
+3. Expand files and click on `index.html`.
+
+If you do not see an artifacts tab this is a bug in CircleCI. Try logging out and logging back in.
+
+### Documentation guidelines
+
+The following are some general guidelines around documentation.
+
+* Cross link as much as possible. Sphinx is fantastic at this. Use it! See ample examples with the
+  existing documentation as a guide.
+* Please use a **single space** after a period in documentation so that all generated text is
+  consistent.
+* Comments can be left inside comments if needed (that's pretty deep, right?) via the `[#comment:]`
+  special tag. E.g.,
+
+  ```
+  // This is a really cool field!
+  // [#comment:TODO(mattklein123): Do something cooler]
+  string foo_field = 3;
+  ```
+
+* Prefer *italics* for emphasis as `backtick` emphasis is somewhat jarring in our Sphinx theme.
+* All documentation is expected to use proper English grammar with proper punctuation. If you are
+  not a fluent English speaker please let us know and we will help out.
+* Tag messages/enum/files with `[#proto-status: draft|experimental|frozen]` to
+  reflect their [API
+  status](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview#status).
+  Frozen entities do not need to be tagged except when overriding an outer scope
+  draft or experimental status.