diff --git a/docs/client-concepts/client-concepts.asciidoc b/docs/client-concepts/client-concepts.asciidoc new file mode 100644 index 00000000000..6b333b44615 --- /dev/null +++ b/docs/client-concepts/client-concepts.asciidoc @@ -0,0 +1,26 @@ +[[client-concepts]] += Client concepts + +The .NET client for {es} maps closely to the original {es} API. All +requests and responses are exposed through types, making it ideal for getting up and running quickly. + +[[serialization]] +== Serialization + +By default, the .NET client for {es} uses the Microsoft System.Text.Json library for serialization. The client understands how to serialize and +deserialize the request and response types correctly. It also handles (de)serialization of user POCO types representing documents read or written to {es}. + +The client has two distinct serialization responsibilities - serialization of the types owned by the `Elastic.Clients.Elasticsearch` library and serialization of source documents, modeled in application code. The first responsibility is entirely internal; the second is configurable. + +[[source-serialization]] +=== Source serialization + +Source serialization refers to the process of (de)serializing POCO types in consumer applications as source documents indexed and retrieved from {es}. A source serializer implementation handles serialization, with the default implementation using the `System.Text.Json` library. As a result, you may use `System.Text.Json` attributes and converters to control the serialization behavior. + +* <> + +* <> + +include::serialization/modeling-documents-with-types.asciidoc[] + +include::serialization/custom-serialization.asciidoc[] diff --git a/docs/client-concepts/serialization/custom-serialization.asciidoc b/docs/client-concepts/serialization/custom-serialization.asciidoc new file mode 100644 index 00000000000..3ca00988e20 --- /dev/null +++ b/docs/client-concepts/serialization/custom-serialization.asciidoc @@ -0,0 +1,232 @@ +[[customizing-source-serialization]] +==== Customizing source serialization + +The built-in source serializer handles most POCO document models correctly. Sometimes, you may need further control over how your types are serialized. + +NOTE: The built-in source serializer uses the https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview[Microsoft `System.Text.Json` library] internally. You can apply `System.Text.Json` attributes and converters to control the serialization of your document types. + +[discrete] +[[system-text-json-attributes]] +===== Using `System.Text.Json` attributes + +`System.Text.Json` includes attributes that can be applied to types and properties to control their serialization. These can be applied to your POCO document types to perform actions such as controlling the name of a property or ignoring a property entirely. Visit the https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview[Microsoft documentation for further examples]. + +We can model a document to represent data about a person using a regular class (POCO), applying `System.Text.Json` attributes as necessary. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=person-class-with-attributes] +---- +<1> The `JsonPropertyName` attribute ensures the `FirstName` property uses the JSON name `forename` when serialized. +<2> The `JsonIgnore` attribute prevents the `Age` property from appearing in the serialized JSON. + +We can then index an instance of the document into {es}. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-person-with-attributes] +---- + +The index request is serialized, with the source serializer handling the `Person` type, serializing the POCO property named `FirstName` to the JSON object member named `forename`. The `Age` property is ignored and does not appear in the JSON. + +[source,javascript] +---- +{ + "forename": "Steve" +} +---- + +[discrete] +[[configuring-custom-jsonserializeroptions]] +===== Configuring custom `JsonSerializerOptions` + +The default source serializer applies a set of standard `JsonSerializerOptions` when serializing source document types. In some circumstances, you may need to override some of our defaults. This is achievable by creating an instance of `DefaultSourceSerializer` and passing an `Action`, which is applied after our defaults have been set. This mechanism allows you to apply additional settings or change the value of our defaults. + +The `DefaultSourceSerializer` includes a constructor that accepts the current `IElasticsearchClientSettings` and a `configureOptions` `Action`. + +[source,csharp] +---- +public DefaultSourceSerializer(IElasticsearchClientSettings settings, Action configureOptions); +---- + +Our application defines the following `Person` class, which models a document we will index to {es}. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=person-class] +---- + +We want to serialize our source document using Pascal Casing for the JSON properties. Since the options applied in the `DefaultSouceSerializer` set the `PropertyNamingPolicy` to `JsonNamingPolicy.CamelCase`, we must override this setting. After configuring the `ElasticsearchClientSettings`, we index our document to {es}. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=custom-options-local-function] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=create-client] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-person] +---- +<1> A local function can be defined, accepting a `JsonSerializerOptions` parameter. Here, we set `PropertyNamingPolicy` to `null`. This returns to the default behavior for `System.Text.Json`, which uses Pascal Case. +<2> When creating the `ElasticsearchClientSettings`, we supply a `SourceSerializerFactory` using a lambda. The factory function creates a new instance of `DefaultSourceSerializer`, passing in the `settings` and our `ConfigureOptions` local function. We have now configured the settings with a custom instance of the source serializer. + +The `Person` instance is serialized, with the source serializer serializing the POCO property named `FirstName` using Pascal Case. + +[source,javascript] +---- +{ + "FirstName": "Steve" +} +---- + +As an alternative to using a local function, we could store an `Action` into a variable instead, which can be passed to the `DefaultSouceSerializer` constructor. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=custom-options-action] +---- + +[discrete] +[[registering-custom-converters]] +===== Registering custom `System.Text.Json` converters + +In certain more advanced situations, you may have types which require further customization during serialization than is possible using `System.Text.Json` property attributes. In these cases, the recommendation from Microsoft is to leverage a custom `JsonConverter`. Source document types serialized using the `DefaultSourceSerializer` can leverage the power of custom converters. + +For this example, our application has a document class that should use a legacy JSON structure to continue operating with existing indexed documents. Several options are available, but we'll apply a custom converter in this case. + +Our class is defined, and the `JsonConverter` attribute is applied to the class type, specifying the type of a custom converter. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-with-jsonconverter-attribute] +---- +<1> The `JsonConverter` attribute signals to `System.Text.Json` that it should use a converter of type `CustomerConverter` when serializing instances of this class. + +When serializing this class, rather than include a string value representing the value of the `CustomerType` property, we must send a boolean property named `isStandard`. This requirement can be achieved with a custom JsonConverter implementation. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=converter-usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-converter] +---- +<1> When reading, this converter reads the `isStandard` boolean and translate this to the correct `CustomerType` enum value. +<2> When writing, this converter translates the `CustomerType` enum value to an `isStandard` boolean property. + +We can then index a customer document into {es}. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-customer-with-converter] +---- + +The `Customer` instance is serialized using the custom converter, creating the following JSON document. + +[source,javascript] +---- +{ + "customerName": "Customer Ltd", + "isStandard": false +} +---- + +[discrete] +[[creating-custom-system-text-json-serializer]] +===== Creating a custom `SystemTextJsonSerializer` + +The built-in `DefaultSourceSerializer` includes the registration of `JsonConverter` instances which apply during source serialization. In most cases, these provide the proper behavior for serializing source documents, including those which use `Elastic.Clients.Elasticsearch` types on their properties. + +An example of a situation where you may require more control over the converter registration order is for serializing `enum` types. The `DefaultSourceSerializer` registers the `System.Text.Json.Serialization.JsonStringEnumConverter`, so enum values are serialized using their string representation. Generally, this is the preferred option for types used to index documents to {es}. + +In some scenarios, you may need to control the string value sent for an enumeration value. That is not directly supported in `System.Text.Json` but can be achieved by creating a custom `JsonConverter` for the `enum` type you wish to customize. In this situation, it is not sufficient to use the `JsonConverterAttribute` on the `enum` type to register the converter. `System.Text.Json` will prefer the converters added to the `Converters` collection on the `JsonSerializerOptions` over an attribute applied to an `enum` type. It is, therefore, necessary to either remove the `JsonStringEnumConverter` from the `Converters` collection or register a specialized converter for your `enum` type before the `JsonStringEnumConverter`. + +The latter is possible via several techniques. When using the {es} .NET library, we can achieve this by deriving from the abstract `SystemTextJsonSerializer` class. + +Here we have a POCO which uses the `CustomerType` enum as the type for a property. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-without-jsonconverter-attribute] +---- + +To customize the strings used during serialization of the `CustomerType`, we define a custom `JsonConverter` specific to our `enum` type. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-type-converter] +---- +<1> When reading, this converter translates the string used in the JSON to the matching enum value. +<2> When writing, this converter translates the `CustomerType` enum value to a custom string value written to the JSON. + +We create a serializer derived from `SystemTextJsonSerializer` to give us complete control of converter registration order. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=derived-converter-usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=my-custom-serializer] +---- +<1> Inherit from `SystemTextJsonSerializer`. +<2> In the constructor, use the factory method `DefaultSourceSerializer.CreateDefaultJsonSerializerOptions` to create default options for serialization. No default converters are registered at this stage because we pass `false` as an argument. +<3> Register our `CustomerTypeConverter` as the first converter. +<4> To apply any default converters, call the `DefaultSourceSerializer.AddDefaultConverters` helper method, passing the options to modify. +<5> Implement the `CreateJsonSerializerOptions` method returning the stored `JsonSerializerOptions`. + +Because we have registered our `CustomerTypeConverter` before the default converters (which include the `JsonStringEnumConverter`), our converter takes precedence when serializing `CustomerType` instances on source documents. + +The base `SystemTextJsonSerializer` class handles the implementation details of binding, which is required to ensure that the built-in converters can access the `IElasticsearchClientSettings` where needed. + +We can then index a customer document into {es}. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-customer-without-jsonconverter-attribute] +---- + +The `Customer` instance is serialized using the custom `enum` converter, creating the following JSON document. + +[source,javascript] +---- +{ + "customerName": "Customer Ltd", + "customerType": "premium" // <1> +} +---- +<1> The string value applied during serialization is provided by our custom converter. + +[discrete] +[[creating-custom-serializers]] +===== Creating a custom `Serializer` + +Suppose you prefer using an alternative JSON serialization library for your source types. In that case, you can inject an isolated serializer only to be called for the serialization of `_source`, `_fields`, or wherever a user-provided value is expected to be written and returned. + +Implementing `Elastic.Transport.Serializer` is technically enough to create a custom source serializer. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=vanilla-serializer-using-directives] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=vanilla-serializer] +---- + +Registering up the serializer is performed in the `ConnectionSettings` constructor. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=register-vanilla-serializer] +---- +<1> If implementing `Serializer` is enough, why must we provide an instance wrapped in a factory `Func`? + +There are various cases where you might have a POCO type that contains an `Elastic.Clients.Elasticsearch` type as one of its properties. The `SourceSerializerFactory` delegate provides access to the default built-in serializer so you can access it when necessary. For example, consider if you want to use percolation; you need to store {es} queries as part of the `_source` of your document, which means you need to have a POCO that looks like this. + +[source,csharp] +---- +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=querydsl-using-directives] +include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=percolation-document] +---- + +A custom serializer would not know how to serialize `Query` or other `Elastic.Clients.Elasticsearch` types that could appear as part of +the `_source` of a document. Therefore, your custom `Serializer` would need to store a reference to our built-in serializer and delegate serialization of Elastic types back to it. \ No newline at end of file diff --git a/docs/client-concepts/serialization/modeling-documents-with-types.asciidoc b/docs/client-concepts/serialization/modeling-documents-with-types.asciidoc new file mode 100644 index 00000000000..d4e57b6d575 --- /dev/null +++ b/docs/client-concepts/serialization/modeling-documents-with-types.asciidoc @@ -0,0 +1,37 @@ +[[modeling-documents-with-types]] +==== Modeling documents with types + +{es} provides search and aggregation capabilities on the documents that it is sent and indexes. These documents are sent as +JSON objects within the request body of a HTTP request. It is natural to model documents within the {es} .NET client using +https://en.wikipedia.org/wiki/Plain_Old_CLR_Object[POCOs (__Plain Old CLR Objects__)]. + +This section provides an overview of how types and type hierarchies can be used to model documents. + +[[default-behaviour]] +===== Default behaviour + +The default behaviour is to serialize type property names as camelcase JSON object members. + +We can model documents using a regular class (POCO). + +[source,csharp] +---- +include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[my-document-poco] +---- + +We can then index the an instance of the document into {es}. + +[source,csharp] +---- +include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[usings] +include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[index-my-document] +---- + +The index request is serialized, with the source serializer handling the `MyDocument` type, serializing the POCO property named `StringProperty` to the JSON object member named `stringProperty`. + +[source,javascript] +---- +{ + "stringProperty": "value" +} +---- \ No newline at end of file diff --git a/docs/reference/troubleshoot/audit-trail.md b/docs/client-concepts/troubleshooting/audit-trail.asciidoc similarity index 71% rename from docs/reference/troubleshoot/audit-trail.md rename to docs/client-concepts/troubleshooting/audit-trail.asciidoc index 96d3008aaf2..7622c09e41c 100644 --- a/docs/reference/troubleshoot/audit-trail.md +++ b/docs/client-concepts/troubleshooting/audit-trail.asciidoc @@ -1,15 +1,22 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/audit-trail.html ---- -# Audit trail [audit-trail] -Elasticsearch.Net and NEST provide an audit trail for the events within the request pipeline that occur when a request is made. This audit trail is available on the response as demonstrated in the following example. +:github: https://github.com/elastic/elasticsearch-net -We’ll use a Sniffing connection pool here since it sniffs on startup and pings before first usage, so we can get an audit trail with a few events out +:nuget: https://www.nuget.org/packages -```csharp + +[[audit-trail]] +=== Audit trail + +Elasticsearch.Net and NEST provide an audit trail for the events within the request pipeline that +occur when a request is made. This audit trail is available on the response as demonstrated in the +following example. + +We'll use a Sniffing connection pool here since it sniffs on startup and pings before +first usage, so we can get an audit trail with a few events out + +[source,csharp] +---- var pool = new SniffingConnectionPool(new []{ TestConnectionSettings.CreateUri() }); var connectionSettings = new ConnectionSettings(pool) .DefaultMappingFor(i => i @@ -17,19 +24,21 @@ var connectionSettings = new ConnectionSettings(pool) ); var client = new ElasticClient(connectionSettings); -``` +---- After issuing the following request -```csharp +[source,csharp] +---- var response = client.Search(s => s .MatchAll() ); -``` +---- -The audit trail is provided in the [Debug information](debug-information.md) in a human readable fashion, similar to +The audit trail is provided in the <> in a human +readable fashion, similar to -``` +.... Valid NEST response built from a successful low level call on POST: /project/doc/_search # Audit trail of this API call: - [1] SniffOnStartup: Took: 00:00:00.0360264 @@ -40,27 +49,32 @@ Valid NEST response built from a successful low level call on POST: /project/doc # Response: -``` +.... + to help with troubleshootin -```csharp +[source,csharp] +---- var debug = response.DebugInformation; -``` +---- But can also be accessed manually: -```csharp +[source,csharp] +---- response.ApiCall.AuditTrail.Count.Should().Be(4, "{0}", debug); response.ApiCall.AuditTrail[0].Event.Should().Be(SniffOnStartup, "{0}", debug); response.ApiCall.AuditTrail[1].Event.Should().Be(SniffSuccess, "{0}", debug); response.ApiCall.AuditTrail[2].Event.Should().Be(PingSuccess, "{0}", debug); response.ApiCall.AuditTrail[3].Event.Should().Be(HealthyResponse, "{0}", debug); -``` +---- -Each audit has a started and ended `DateTime` on it that will provide some understanding of how long it took +Each audit has a started and ended `DateTime` on it that will provide +some understanding of how long it took -```csharp +[source,csharp] +---- response.ApiCall.AuditTrail .Should().OnlyContain(a => a.Ended - a.Started >= TimeSpan.Zero); -``` +---- diff --git a/docs/reference/images/elasticsearch-client-net-api-capture-requests-localhost.png b/docs/client-concepts/troubleshooting/capture-requests-localhost.png similarity index 100% rename from docs/reference/images/elasticsearch-client-net-api-capture-requests-localhost.png rename to docs/client-concepts/troubleshooting/capture-requests-localhost.png diff --git a/docs/reference/images/elasticsearch-client-net-api-capture-requests-remotehost.png b/docs/client-concepts/troubleshooting/capture-requests-remotehost.png similarity index 100% rename from docs/reference/images/elasticsearch-client-net-api-capture-requests-remotehost.png rename to docs/client-concepts/troubleshooting/capture-requests-remotehost.png diff --git a/docs/client-concepts/troubleshooting/debug-information.asciidoc b/docs/client-concepts/troubleshooting/debug-information.asciidoc new file mode 100644 index 00000000000..a7504312d2d --- /dev/null +++ b/docs/client-concepts/troubleshooting/debug-information.asciidoc @@ -0,0 +1,180 @@ + + +:github: https://github.com/elastic/elasticsearch-net + +:nuget: https://www.nuget.org/packages + +[[debug-information]] +=== Debug information + +Every response from Elasticsearch.Net and NEST contains a `DebugInformation` property +that provides a human readable description of what happened during the request for both successful and +failed requests + +[source,csharp] +---- +var response = client.Search(s => s + .Query(q => q + .MatchAll() + ) +); + +response.DebugInformation.Should().Contain("Valid NEST response"); +---- + +This can be useful in tracking down numerous problems and can also be useful when filing an +{github}/issues[issue] on the GitHub repository. + +==== Request and response bytes + +By default, the request and response bytes are not available within the debug information, but +can be enabled globally on Connection Settings by setting `DisableDirectStreaming`. This +disables direct streaming of + +. the serialized request type to the request stream + +. the response stream to a deserialized response type + +[source,csharp] +---- +var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); + +var settings = new ConnectionSettings(connectionPool) + .DisableDirectStreaming(); <1> + +var client = new ElasticClient(settings); +---- +<1> disable direct streaming for *all* requests + +or on a _per request_ basis + +[source,csharp] +---- +var response = client.Search(s => s + .RequestConfiguration(r => r + .DisableDirectStreaming() <1> + ) + .Query(q => q + .MatchAll() + ) +); +---- +<1> disable direct streaming for *this* request only + +Configuring `DisableDirectStreaming` on an individual request takes precedence over +any global configuration. + +There is typically a performance and allocation cost associated with disabling direct streaming +since both the request and response bytes must be buffered in memory, to allow them to be +exposed on the response call details. + +==== TCP statistics + +It can often be useful to see the statistics for active TCP connections, particularly when +trying to diagnose issues with the client. The client can collect the states of active TCP +connections just before making a request, and expose these on the response and in the debug +information. + +Similarly to `DisableDirectStreaming`, TCP statistics can be collected for every request +by configuring on `ConnectionSettings` + +[source,csharp] +---- +var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); + +var settings = new ConnectionSettings(connectionPool) + .EnableTcpStats(); <1> + +var client = new ElasticClient(settings); +---- +<1> collect TCP statistics for *all* requests + +or on a _per request_ basis + +[source,csharp] +---- +var response = client.Search(s => s + .RequestConfiguration(r => r + .EnableTcpStats() <1> + ) + .Query(q => q + .MatchAll() + ) +); + +var debugInformation = response.DebugInformation; +---- +<1> collect TCP statistics for *this* request only + +With `EnableTcpStats` set, the states of active TCP connections will now be included +on the response and in the debug information. + +The client includes a `TcpStats` +class to help with retrieving more detail about active TCP connections should it be +required + +[source,csharp] +---- +var tcpStatistics = TcpStats.GetActiveTcpConnections(); <1> +var ipv4Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv4); <2> +var ipv6Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv6); <3> + +var response = client.Search(s => s + .Query(q => q + .MatchAll() + ) +); +---- +<1> Retrieve details about active TCP connections, including local and remote addresses and ports +<2> Retrieve statistics about IPv4 +<3> Retrieve statistics about IPv6 + +[NOTE] +-- +Collecting TCP statistics may not be accessible in all environments, for example, Azure App Services. +When this is the case, `TcpStats.GetActiveTcpConnections()` returns `null`. + +-- + +==== ThreadPool statistics + +It can often be useful to see the statistics for thread pool threads, particularly when +trying to diagnose issues with the client. The client can collect statistics for both +worker threads and asynchronous I/O threads, and expose these on the response and +in debug information. + +Similar to collecting TCP statistics, ThreadPool statistics can be collected for all requests +by configuring `EnableThreadPoolStats` on `ConnectionSettings` + +[source,csharp] +---- +var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); + +var settings = new ConnectionSettings(connectionPool) + .EnableThreadPoolStats(); <1> + +var client = new ElasticClient(settings); +---- +<1> collect thread pool statistics for *all* requests + +or on a _per request_ basis + +[source,csharp] +---- +var response = client.Search(s => s + .RequestConfiguration(r => r + .EnableThreadPoolStats() <1> + ) + .Query(q => q + .MatchAll() + ) + ); + +var debugInformation = response.DebugInformation; <2> +---- +<1> collect thread pool statistics for *this* request only +<2> contains thread pool statistics + +With `EnableThreadPoolStats` set, the statistics of thread pool threads will now be included +on the response and in the debug information. + diff --git a/docs/client-concepts/troubleshooting/debug-mode.asciidoc b/docs/client-concepts/troubleshooting/debug-mode.asciidoc new file mode 100644 index 00000000000..05d84092f3e --- /dev/null +++ b/docs/client-concepts/troubleshooting/debug-mode.asciidoc @@ -0,0 +1,65 @@ + + +:github: https://github.com/elastic/elasticsearch-net + +:nuget: https://www.nuget.org/packages + +[[debug-mode]] +=== Debug mode + +The <> explains that every response from Elasticsearch.Net +and NEST contains a `DebugInformation` property, and properties on `ConnectionSettings` and +`RequestConfiguration` can control which additional information is included in debug information, +for all requests or on a per request basis, respectively. + +During development, it can be useful to enable the most verbose debug information, to help +identify and troubleshoot problems, or simply ensure that the client is behaving as expected. +The `EnableDebugMode` setting on `ConnectionSettings` is a convenient shorthand for enabling +verbose debug information, configuring a number of settings like + +* disabling direct streaming to capture request and response bytes + +* prettyfying JSON responses from Elasticsearch + +* collecting TCP statistics when a request is made + +* collecting thread pool statistics when a request is made + +* including the Elasticsearch stack trace in the response if there is a an error on the server side + +[source,csharp] +---- +IConnectionPool pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); + +var settings = new ConnectionSettings(pool) + .EnableDebugMode(); <1> + +var client = new ElasticClient(settings); + +var response = client.Search(s => s + .Query(q => q + .MatchAll() + ) +); + +var debugInformation = response.DebugInformation; <2> +---- +<1> configure debug mode +<2> verbose debug information + +In addition to exposing debug information on the response, debug mode will also cause the debug +information to be written to the trace listeners in the `System.Diagnostics.Debug.Listeners` collection +by default, when the request has completed. A delegate can be passed when enabling debug mode to perform +a different action when a request has completed, using <> + +[source,csharp] +---- +var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); +var client = new ElasticClient(new ConnectionSettings(pool) + .EnableDebugMode(apiCallDetails => + { + // do something with the call details e.g. send with logging framework + }) +); +---- + diff --git a/docs/client-concepts/troubleshooting/deprecation-logging.asciidoc b/docs/client-concepts/troubleshooting/deprecation-logging.asciidoc new file mode 100644 index 00000000000..c69a0fc1ee1 --- /dev/null +++ b/docs/client-concepts/troubleshooting/deprecation-logging.asciidoc @@ -0,0 +1,40 @@ + + +:github: https://github.com/elastic/elasticsearch-net + +:nuget: https://www.nuget.org/packages + +[[deprecation-logging]] +=== Deprecation logging + +Elasticsearch will send back `Warn` HTTP Headers when you are using an API feature that is +deprecated and will be removed or rewritten in a future release. + +Elasticsearch.NET and NEST report these back to you so you can log and watch out for them. + +[source,csharp] +---- +var request = new SearchRequest +{ + Size = 0, + Aggregations = new CompositeAggregation("test") + { + Sources = new [] + { + new DateHistogramCompositeAggregationSource("date") + { + Field = Field(f => f.LastActivity), + Interval = new Time("7d"), + Format = "yyyy-MM-dd" + } + } + } +}; +var response = this.Client.Search(request); + +response.ApiCall.DeprecationWarnings.Should().NotBeNullOrEmpty(); +response.ApiCall.DeprecationWarnings.Should().HaveCountGreaterOrEqualTo(1); +response.DebugInformation.Should().Contain("deprecated"); <1> +---- +<1> `DebugInformation` also contains the deprecation warnings + diff --git a/docs/client-concepts/troubleshooting/diagnostic-source.asciidoc b/docs/client-concepts/troubleshooting/diagnostic-source.asciidoc new file mode 100644 index 00000000000..2f8e189cd60 --- /dev/null +++ b/docs/client-concepts/troubleshooting/diagnostic-source.asciidoc @@ -0,0 +1,121 @@ + + +:github: https://github.com/elastic/elasticsearch-net + +:nuget: https://www.nuget.org/packages + +[[diagnostic-source]] +=== Diagnostic Source + +Elasticsearch.Net and NEST support capturing diagnostics information using `DiagnosticSource` and `Activity` from the +`System.Diagnostics` namespace. + +To aid with the discoverability of the topics you can subscribe to and the event names they emit, +both topics and event names are exposed as strongly typed strings under `Elasticsearch.Net.Diagnostics.DiagnosticSources` + +Subscribing to DiagnosticSources means implementing `IObserver` +or using `.Subscribe(observer, filter)` to opt in to the correct topic. + +Here we choose the more verbose `IObserver<>` implementation + +[source,csharp] +---- +public class ListenerObserver : IObserver, IDisposable +{ + private long _messagesWrittenToConsole = 0; + public long MessagesWrittenToConsole => _messagesWrittenToConsole; + + public Exception SeenException { get; private set; } + + public void OnError(Exception error) => SeenException = error; + public bool Completed { get; private set; } + public void OnCompleted() => Completed = true; + + private void WriteToConsole(string eventName, T data) + { + var a = Activity.Current; + Interlocked.Increment(ref _messagesWrittenToConsole); + } + + private List Disposables { get; } = new List(); + + public void OnNext(DiagnosticListener value) + { + void TrySubscribe(string sourceName, Func>> listener) <1> + { + if (value.Name != sourceName) return; + + var subscription = value.Subscribe(listener()); + Disposables.Add(subscription); + } + + TrySubscribe(DiagnosticSources.AuditTrailEvents.SourceName, + () => new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value))); + + TrySubscribe(DiagnosticSources.Serializer.SourceName, + () => new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value))); + + TrySubscribe(DiagnosticSources.RequestPipeline.SourceName, + () => new RequestPipelineDiagnosticObserver( + v => WriteToConsole(v.Key, v.Value), + v => WriteToConsole(v.Key, v.Value) + )); + + TrySubscribe(DiagnosticSources.HttpConnection.SourceName, + () => new HttpConnectionDiagnosticObserver( + v => WriteToConsole(v.Key, v.Value), + v => WriteToConsole(v.Key, v.Value) + )); + } + + public void Dispose() + { + foreach(var d in Disposables) d.Dispose(); + } +} +---- +<1> By inspecting the name, we can selectively subscribe only to the topics `Elasticsearch.Net` emit + +Thanks to `DiagnosticSources`, you do not have to guess the topics emitted. + +The `DiagnosticListener.Subscribe` method expects an `IObserver>` +which is a rather generic message contract. As a subscriber, it's useful to know what `object` is in each case. +To help with this, each topic within the client has a dedicated `Observer` implementation that +takes an `onNext` delegate typed to the context object actually emitted. + +The RequestPipeline diagnostic source emits a different context objects the start and end of the `Activity` +For this reason, `RequestPipelineDiagnosticObserver` accepts two `onNext` delegates, +one for the `.Start` events and one for the `.Stop` events. + +[[subscribing-to-topics]] +==== Subscribing to topics + +As a concrete example of subscribing to topics, let's hook into all diagnostic sources and use +`ListenerObserver` to only listen to the ones from `Elasticsearch.Net` + +[source,csharp] +---- +using(var listenerObserver = new ListenerObserver()) +using (var subscription = DiagnosticListener.AllListeners.Subscribe(listenerObserver)) +{ + var pool = new SniffingConnectionPool(new []{ TestConnectionSettings.CreateUri() }); <1> + var connectionSettings = new ConnectionSettings(pool) + .DefaultMappingFor(i => i + .IndexName("project") + ); + + var client = new ElasticClient(connectionSettings); + + var response = client.Search(s => s <2> + .MatchAll() + ); + + listenerObserver.SeenException.Should().BeNull(); <3> + listenerObserver.Completed.Should().BeFalse(); + listenerObserver.MessagesWrittenToConsole.Should().BeGreaterThan(0); +} +---- +<1> use a sniffing connection pool that sniffs on startup and pings before first usage, so our diagnostics will emit most topics. +<2> make a search API call +<3> verify that the listener is picking up events + diff --git a/docs/reference/images/elasticsearch-client-net-api-inspect-requests.png b/docs/client-concepts/troubleshooting/inspect-requests.png similarity index 100% rename from docs/reference/images/elasticsearch-client-net-api-inspect-requests.png rename to docs/client-concepts/troubleshooting/inspect-requests.png diff --git a/docs/client-concepts/troubleshooting/logging-with-fiddler.asciidoc b/docs/client-concepts/troubleshooting/logging-with-fiddler.asciidoc new file mode 100644 index 00000000000..527f4bc53db --- /dev/null +++ b/docs/client-concepts/troubleshooting/logging-with-fiddler.asciidoc @@ -0,0 +1,48 @@ + + +:github: https://github.com/elastic/elasticsearch-net + +:nuget: https://www.nuget.org/packages + +[[logging-with-fiddler]] +=== Logging with Fiddler + +A web debugging proxy such as http://www.telerik.com/fiddler[Fiddler] is a useful way to capture HTTP traffic +from a machine, particularly whilst developing against a local Elasticsearch cluster. + +==== Capturing traffic to a remote cluster + +To capture traffic against a remote cluster is as simple as launching Fiddler! You may want to also +filter traffic to only show requests to the remote cluster by using the filters tab + +image::capture-requests-remotehost.png[Capturing requests to a remote host] + +==== Capturing traffic to a local cluster + +The .NET Framework is hardcoded not to send requests for `localhost` through any proxies and as a proxy +Fiddler will not receive such traffic. + +This is easily circumvented by using `ipv4.fiddler` as the hostname instead of `localhost` + +[source,csharp] +---- +var isFiddlerRunning = Process.GetProcessesByName("fiddler").Any(); +var host = isFiddlerRunning ? "ipv4.fiddler" : "localhost"; + +var connectionSettings = new ConnectionSettings(new Uri($"http://{host}:9200")) + .PrettyJson(); <1> + +var client = new ElasticClient(connectionSettings); +---- +<1> prettify json requests and responses to make them easier to read in Fiddler + +With Fiddler running, the requests and responses will now be captured and can be inspected in the +Inspectors tab + +image::inspect-requests.png[Inspecting requests and responses] + +As before, you may also want to filter traffic to only show requests to `ipv4.fiddler` on the port +on which you are running Elasticsearch. + +image::capture-requests-localhost.png[Capturing requests to localhost] + diff --git a/docs/reference/troubleshoot/logging-with-onrequestcompleted.md b/docs/client-concepts/troubleshooting/logging-with-on-request-completed.asciidoc similarity index 65% rename from docs/reference/troubleshoot/logging-with-onrequestcompleted.md rename to docs/client-concepts/troubleshooting/logging-with-on-request-completed.asciidoc index f1c365ffaf6..a21c9ce67a0 100644 --- a/docs/reference/troubleshoot/logging-with-onrequestcompleted.md +++ b/docs/client-concepts/troubleshooting/logging-with-on-request-completed.asciidoc @@ -1,17 +1,24 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging-with-on-request-completed.html ---- -# Logging with OnRequestCompleted [logging-with-on-request-completed] -When constructing the connection settings to pass to the client, you can pass a callback of type `Action` to the `OnRequestCompleted` method that can eavesdrop every time a response(good or bad) is received. +:github: https://github.com/elastic/elasticsearch-net -If you have complex logging needs this is a good place to add that in since you have access to both the request and response details. +:nuget: https://www.nuget.org/packages -In this example, we’ll use `OnRequestCompleted` on connection settings to increment a counter each time it’s called. +[[logging-with-on-request-completed]] +=== Logging with OnRequestCompleted -```csharp +When constructing the connection settings to pass to the client, you can pass a callback of type +`Action` to the `OnRequestCompleted` method that can eavesdrop every time a +response(good or bad) is received. + +If you have complex logging needs this is a good place to add that in +since you have access to both the request and response details. + +In this example, we'll use `OnRequestCompleted` on connection settings to increment a counter each time +it's called. + +[source,csharp] +---- var counter = 0; var client = new ElasticClient(new AlwaysInMemoryConnectionSettings().OnRequestCompleted(r => counter++)); <1> @@ -20,16 +27,16 @@ counter.Should().Be(1); await client.RootNodeInfoAsync(); <3> counter.Should().Be(2); -``` - -1. Construct a client -2. Make a synchronous call and assert the counter is incremented -3. Make an asynchronous call and assert the counter is incremented +---- +<1> Construct a client +<2> Make a synchronous call and assert the counter is incremented +<3> Make an asynchronous call and assert the counter is incremented +`OnRequestCompleted` is called even when an exception is thrown, so it can be used even if the client is +configured to throw exceptions -`OnRequestCompleted` is called even when an exception is thrown, so it can be used even if the client is configured to throw exceptions - -```csharp +[source,csharp] +---- var counter = 0; var client = FixedResponseClient.Create( <1> new { }, @@ -44,24 +51,25 @@ counter.Should().Be(1); await Assert.ThrowsAsync(async () => await client.RootNodeInfoAsync()); counter.Should().Be(2); -``` - -1. Configure a client with a connection that **always returns a HTTP 500 response -2. Always throw exceptions when a call results in an exception -3. Assert an exception is thrown and the counter is incremented - - -Here’s an example using `OnRequestCompleted()` for more complex logging +---- +<1> Configure a client with a connection that **always returns a HTTP 500 response +<2> Always throw exceptions when a call results in an exception +<3> Assert an exception is thrown and the counter is incremented -::::{note} -By default, the client writes directly to the request stream and deserializes directly from the response stream. +Here's an example using `OnRequestCompleted()` for more complex logging -If you would also like to capture the request and/or response bytes, you also need to set `.DisableDirectStreaming()` to `true`. +[NOTE] +-- +By default, the client writes directly to the request stream and deserializes directly from the +response stream. -:::: +If you would also like to capture the request and/or response bytes, +you also need to set `.DisableDirectStreaming()` to `true`. +-- -```csharp +[source,csharp] +---- var list = new List(); var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); @@ -122,21 +130,25 @@ list.Should().BeEquivalentTo(new[] <6> @"POST http://localhost:9200/_all/_search?typed_keys=true&scroll=10m {""sort"":[{""_doc"":{""order"":""asc""}}]}", @"Status: 200" }); -``` - -1. Here we use `InMemoryConnection` but in a real application, you’d use an `IConnection` that *actually* sends the request, such as `HttpConnection` -2. Disable direct streaming so we can capture the request and response bytes -3. Perform some action when a request completes. Here, we’re just adding to a list, but in your application you may be logging to a file. -4. Make a synchronous call -5. Make an asynchronous call -6. Assert the list contains the contents written in the delegate passed to `OnRequestCompleted` - - -When running an application in production, you probably don’t want to disable direct streaming for *all* requests, since doing so will incur a performance overhead, due to buffering request and response bytes in memory. It can however be useful to capture requests and responses in an ad-hoc fashion, perhaps to troubleshoot an issue in production. - -`DisableDirectStreaming` can be enabled on a *per-request* basis for this purpose. In using this feature, it is possible to configure a general logging mechanism in `OnRequestCompleted` and log out request and responses only when necessary - -```csharp +---- +<1> Here we use `InMemoryConnection` but in a real application, you'd use an `IConnection` that _actually_ sends the request, such as `HttpConnection` +<2> Disable direct streaming so we can capture the request and response bytes +<3> Perform some action when a request completes. Here, we're just adding to a list, but in your application you may be logging to a file. +<4> Make a synchronous call +<5> Make an asynchronous call +<6> Assert the list contains the contents written in the delegate passed to `OnRequestCompleted` + +When running an application in production, you probably don't want to disable direct streaming for _all_ +requests, since doing so will incur a performance overhead, due to buffering request and +response bytes in memory. It can however be useful to capture requests and responses in an ad-hoc fashion, +perhaps to troubleshoot an issue in production. + +`DisableDirectStreaming` can be enabled on a _per-request_ basis for this purpose. In using this feature, +it is possible to configure a general logging mechanism in `OnRequestCompleted` and log out +request and responses only when necessary + +[source,csharp] +---- var list = new List(); var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); @@ -199,11 +211,9 @@ list.Should().BeEquivalentTo(new[] @"POST http://localhost:9200/_all/_search?typed_keys=true&scroll=10m {""sort"":[{""_doc"":{""order"":""asc""}}]}", <4> @"Status: 200" }); -``` - -1. Make a synchronous call where the request and response bytes will not be buffered -2. Make an asynchronous call where `DisableDirectStreaming()` is enabled -3. Only the method and url for the first request is captured -4. the body of the second request is captured - +---- +<1> Make a synchronous call where the request and response bytes will not be buffered +<2> Make an asynchronous call where `DisableDirectStreaming()` is enabled +<3> Only the method and url for the first request is captured +<4> the body of the second request is captured diff --git a/docs/configuration.asciidoc b/docs/configuration.asciidoc new file mode 100644 index 00000000000..a577b3e40b6 --- /dev/null +++ b/docs/configuration.asciidoc @@ -0,0 +1,247 @@ +[[configuration]] +== Configuration + +Connecting to {es} with the client is easy, but it's possible that you'd like to +change the default connection behaviour. There are a number of configuration +options available on `ElasticsearchClientSettings` that can be used to control how the +client interact with {es}. + +=== Options on ElasticsearchClientSettings + +The following is a list of available connection configuration options on +`ElasticsearchClientSettings`: + +`Authentication`:: + +An implementation of `IAuthenticationHeader` describing what http header to use +to authenticate with the product. ++ + `BasicAuthentication` for basic authentication ++ + `ApiKey` for simple secret token ++ + `Base64ApiKey` for Elastic Cloud style encoded api keys + +`ClientCertificate`:: + +Use the following certificates to authenticate all HTTP requests. You can also +set them on individual request using `ClientCertificates`. + +`ClientCertificates`:: + +Use the following certificates to authenticate all HTTP requests. You can also +set them on individual request using `ClientCertificates`. + +`ConnectionLimit`:: + +Limits the number of concurrent connections that can be opened to an endpoint. +Defaults to 80 (see `DefaultConnectionLimit`). ++ +For Desktop CLR, this setting applies to the `DefaultConnectionLimit` property +on the `ServicePointManager` object when creating `ServicePoint` objects, +affecting the default `IConnection` implementation. ++ +For Core CLR, this setting applies to the `MaxConnectionsPerServer` property on +the `HttpClientHandler` instances used by the `HttpClient` inside the default +`IConnection` implementation. + +`DeadTimeout`:: + +The time to put dead nodes out of rotation (this will be multiplied by the +number of times they've been dead). + +`DefaultDisableIdInference`:: + +Disables automatic Id inference for given CLR types. ++ +The client by default will use the value of a property named `Id` on a CLR type +as the `_id` to send to {es}. Adding a type will disable this behaviour for that +CLR type. If `Id` inference should be disabled for all CLR types, use +`DefaultDisableIdInference`. + +`DefaultFieldNameInferrer`:: + +Specifies how field names are inferred from CLR property names. ++ +By default, the client camel cases property names. For example, CLR property +`EmailAddress` will be inferred as "emailAddress" {es} document field name. + +`DefaultIndex`:: + +The default index to use for a request when no index has been explicitly +specified and no default indices are specified for the given CLR type specified +for the request. + +`DefaultMappingFor`:: + +Specify how the mapping is inferred for a given CLR type. The mapping can infer +the index, id and relation name for a given CLR type, as well as control +serialization behaviour for CLR properties. + +`DisableAutomaticProxyDetection`:: + +Disabled proxy detection on the webrequest, in some cases this may speed up the +first connection your appdomain makes, in other cases it will actually increase +the time for the first connection. No silver bullet! Use with care! + +`DisableDirectStreaming`:: + +When set to true will disable (de)serializing directly to the request and +response stream and return a byte[] copy of the raw request and response. +Defaults to false. + +`DisablePing`:: + +This signals that we do not want to send initial pings to unknown/previously +dead nodes and just send the call straightaway. + +`DnsRefreshTimeout`:: + +DnsRefreshTimeout for the connections. Defaults to 5 minutes. + +`EnableDebugMode`:: + +Turns on settings that aid in debugging like `DisableDirectStreaming()` and +`PrettyJson()` so that the original request and response JSON can be inspected. +It also always asks the server for the full stack trace on errors. + +`EnableHttpCompression`:: + +Enable gzip compressed requests and responses. + +`EnableHttpPipelining`:: + +Whether HTTP pipelining is enabled. The default is `true`. + +`EnableTcpKeepAlive`:: + +Sets the keep-alive option on a TCP connection. ++ +For Desktop CLR, sets `ServicePointManager`.`SetTcpKeepAlive`. + +`EnableTcpStats`:: + +Enable statistics about TCP connections to be collected when making a request. + +`GlobalHeaders`:: + +Try to send these headers for every request. + +`GlobalQueryStringParameters`:: + +Append these query string parameters automatically to every request. + +`MaxDeadTimeout`:: + +The maximum amount of time a node is allowed to marked dead. + +`MaximumRetries`:: + +When a retryable exception occurs or status code is returned this controls the +maximum amount of times we should retry the call to {es}. + +`MaxRetryTimeout`:: + +Limits the total runtime including retries separately from `RequestTimeout`. +When not specified defaults to `RequestTimeout` which itself defaults to 60 +seconds. + +`MemoryStreamFactory`:: + +Provides a memory stream factory. + +`NodePredicate`:: + +Register a predicate to select which nodes that you want to execute API calls +on. Note that sniffing requests omit this predicate and always execute on all +nodes. When using an `IConnectionPool` implementation that supports reseeding of +nodes, this will default to omitting master only node from regular API calls. +When using static or single node connection pooling it is assumed the list of +node you instantiate the client with should be taken verbatim. + +`OnRequestCompleted`:: + +Allows you to register a callback every time a an API call is returned. + +`OnRequestDataCreated`:: + +An action to run when the `RequestData` for a request has been created. + +`PingTimeout`:: + +The timeout in milliseconds to use for ping requests, which are issued to +determine whether a node is alive. + +`PrettyJson`:: + +Provide hints to serializer and products to produce pretty, non minified json. ++ +Note: this is not a guarantee you will always get prettified json. + +`Proxy`:: + +If your connection has to go through proxy, use this method to specify the +proxy url. + +`RequestTimeout`:: + +The timeout in milliseconds for each request to {es}. + +`ServerCertificateValidationCallback`:: + +Register a `ServerCertificateValidationCallback` per request. + +`SkipDeserializationForStatusCodes`:: + +Configure the client to skip deserialization of certain status codes, for +example, you run {es} behind a proxy that returns an unexpected json format. + +`SniffLifeSpan`:: + +Force a new sniff for the cluster when the cluster state information is older +than the specified timespan. + +`SniffOnConnectionFault`:: + +Force a new sniff for the cluster state every time a connection dies. + +`SniffOnStartup`:: + +Sniff the cluster state immediately on startup. + +`ThrowExceptions`:: + +Instead of following a c/go like error checking on response. `IsValid` do throw +an exception (except when `SuccessOrKnownError` is false) on the client when a +call resulted in an exception on either the client or the {es} server. ++ +Reasons for such exceptions could be search parser errors, index missing +exceptions, and so on. + +`TransferEncodingChunked`:: + +Whether the request should be sent with chunked Transfer-Encoding. + +`UserAgent`:: + +The user agent string to send with requests. Useful for debugging purposes to +understand client and framework versions that initiate requests to {es}. + + +==== ElasticsearchClientSettings with ElasticsearchClient + +Here's an example to demonstrate setting configuration options using the client. + +[source,csharp] +---- +var settings= new ElasticsearchClientSettings() + .DefaultMappingFor(i => i + .IndexName("my-projects") + .IdProperty(p => p.Name) + ) + .EnableDebugMode() + .PrettyJson() + .RequestTimeout(TimeSpan.FromMinutes(2)); + +var client = new ElasticsearchClient(settings); +---- diff --git a/docs/connecting.asciidoc b/docs/connecting.asciidoc new file mode 100644 index 00000000000..13d706ba04d --- /dev/null +++ b/docs/connecting.asciidoc @@ -0,0 +1,173 @@ +[[connecting]] +== Connecting + +This page contains the information you need to create an instance of the .NET +Client for {es} that connects to your {es} cluster. + +It's possible to connect to your {es} cluster via a single node, or by +specifying multiple nodes using a node pool. Using a node pool has a few +advantages over a single node, such as load balancing and cluster failover +support. The client provides convenient configuration options to connect to an +Elastic Cloud deployment. + +IMPORTANT: Client applications should create a single instance of +`ElasticsearchClient` that is used throughout your application for its entire +lifetime. Internally the client manages and maintains HTTP connections to nodes, +reusing them to optimize performance. If you use a dependency injection +container for your application, the client instance should be registered with a +singleton lifetime. + +[discrete] +[[cloud-deployment]] +=== Connecting to a cloud deployment + +https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html[Elastic Cloud] +is the easiest way to get started with {es}. When connecting to Elastic Cloud +with the .NET {es} client you should always use the Cloud ID. You can find this +value within the "Manage Deployment" page after you've created a cluster +(look in the top-left if you're in Kibana). + +We recommend using a Cloud ID whenever possible because your client will be +automatically configured for optimal use with Elastic Cloud, including HTTPS and +HTTP compression. + +Connecting to an Elasticsearch Service deployment is achieved by providing the +unique Cloud ID for your deployment when configuring the `ElasticsearchClient` +instance. You also require suitable credentials, either a username and password or +an API key that your application uses to authenticate with your deployment. + +As a security best practice, it is recommended to create a dedicated API key per +application, with permissions limited to only those required for any API calls +the application is authorized to make. + +The following snippet demonstrates how to create a client instance that connects to +an {es} deployment in the cloud. + +[source,csharp] +---- +using Elastic.Clients.Elasticsearch; +using Elastic.Transport; + +var client = new ElasticsearchClient("", new ApiKey("")); <1> +---- +<1> Replace the placeholder string values above with your cloud ID and the API key +configured for your application to access your deployment. + + +[discrete] +[[single-node]] +=== Connecting to a single node + +Single node configuration is best suited to connections to a multi-node cluster +running behind a load balancer or reverse proxy, which is exposed via a single +URL. It may also be convenient to use a single node during local application +development. If the URL represents a single {es} node, be aware that this offers +no resiliency should the server be unreachable or unresponsive. + +By default, security features such as authentication and TLS are enabled on {es} +clusters. When you start {es} for the first time, TLS is configured +automatically for the HTTP layer. A CA certificate is generated and stored on +disk which is used to sign the certificates for the HTTP layer of the {es} +cluster. + +In order for the client to establish a connection with the cluster over HTTPS, +the CA certificate must be trusted by the client application. The simplest +choice is to use the hex-encoded SHA-256 fingerprint of the CA certificate. The +CA fingerprint is output to the terminal when you start {es} for the first time. +You'll see a distinct block like the one below in the output from {es} (you may +have to scroll up if it's been a while): + +```sh +---------------------------------------------------------------- +-> Elasticsearch security features have been automatically configured! +-> Authentication is enabled and cluster connections are encrypted. + +-> Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`): + lhQpLELkjkrawaBoaz0Q + +-> HTTP CA certificate SHA-256 fingerprint: + a52dd93511e8c6045e21f16654b77c9ee0f34aea26d9f40320b531c474676228 +... +---------------------------------------------------------------- +``` + +Note down the `elastic` user password and HTTP CA fingerprint for the next +sections. + +The CA fingerprint can also be retrieved at any time from a running cluster using +the following command: + +[source,shell] +---- +openssl x509 -fingerprint -sha256 -in config/certs/http_ca.crt +---- + +The command returns the security certificate, including the fingerprint. The +`issuer` should be `Elasticsearch security auto-configuration HTTP CA`. + +[source,shell] +---- +issuer= /CN=Elasticsearch security auto-configuration HTTP CA +SHA256 Fingerprint= +---- + +Visit the +{ref}/configuring-stack-security.html[Start the Elastic Stack with security enabled automatically] +documentation for more information. + +The following snippet shows you how to create a client instance that connects to +your {es} cluster via a single node, using the CA fingerprint: + +[source,csharp] +---- +using Elastic.Clients.Elasticsearch; +using Elastic.Transport; + +var settings = new ElasticsearchClientSettings(new Uri("https://localhost:9200")) + .CertificateFingerprint("") + .Authentication(new BasicAuthentication("", "")); + +var client = new ElasticsearchClient(settings); +---- + +The preceding snippet demonstrates configuring the client to authenticate by +providing a username and password with basic authentication. If preferred, you +may also use `ApiKey` authentication as shown in the cloud connection example. + +[discrete] +[[multiple-nodes]] +=== Connecting to multiple nodes using a node pool + +To provide resiliency, you should configure multiple nodes for your cluster to +which the client attempts to communicate. By default, the client cycles through +nodes for each request in a round robin fashion. The client also tracks +unhealthy nodes and avoids sending requests to them until they become healthy. + +This configuration is best suited to connect to a known small sized cluster, +where you do not require sniffing to detect the cluster topology. + +The following snippet shows you how to connect to multiple nodes by using a +static node pool: + +[source,csharp] +---- +using Elastic.Clients.Elasticsearch; +using Elastic.Transport; + +var nodes = new Uri[] +{ + new Uri("https://myserver1:9200"), + new Uri("https://myserver2:9200"), + new Uri("https://myserver3:9200") +}; + +var pool = new StaticNodePool(nodes); + +var settings = new ElasticsearchClientSettings(pool) + .CertificateFingerprint("") + .Authentication(new ApiKey("")); + +var client = new ElasticsearchClient(settings); +---- + + diff --git a/docs/docset.yml b/docs/docset.yml deleted file mode 100644 index ba5567217f7..00000000000 --- a/docs/docset.yml +++ /dev/null @@ -1,12 +0,0 @@ -project: '.NET client' -products: - - id: elasticsearch-client -cross_links: - - apm-agent-dotnet - - docs-content - - elasticsearch -toc: - - toc: reference - - toc: release-notes -subs: - es: "Elasticsearch" diff --git a/docs/getting-started.asciidoc b/docs/getting-started.asciidoc new file mode 100644 index 00000000000..21d320236c0 --- /dev/null +++ b/docs/getting-started.asciidoc @@ -0,0 +1,160 @@ +[[getting-started-net]] +== Getting started + +This page guides you through the installation process of the .NET client, shows +you how to instantiate the client, and how to perform basic Elasticsearch +operations with it. + +[discrete] +=== Requirements + +* .NET Core, .NET 5+ or .NET Framework (4.6.1 and higher). + +[discrete] +=== Installation + +To install the latest version of the client for SDK style projects, run the following command: + +[source,shell] +-------------------------- +dotnet add package Elastic.Clients.Elasticsearch +-------------------------- + +Refer to the <> page to learn more. + + +[discrete] +=== Connecting + +You can connect to the Elastic Cloud using an API key and your Elasticsearch +Cloud ID. + +[source,net] +---- +var client = new ElasticsearchClient("", new ApiKey("")); +---- + +You can find your Elasticsearch Cloud ID on the deployment page: + +image::images/es-cloudid.jpg[alt="Cloud ID on the deployment page",align="center"] + +To generate an API key, use the +https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Elasticsearch Create API key API] +or https://www.elastic.co/guide/en/kibana/current/api-keys.html#create-api-key[Kibana Stack Management]. + +For other connection options, refer to the <> section. + + +[discrete] +=== Operations + +Time to use Elasticsearch! This section walks you through the basic, and most +important, operations of Elasticsearch. For more operations and more advanced +examples, refer to the <> page. + + +[discrete] +==== Creating an index + +This is how you create the `my_index` index: + +[source,net] +---- +var response = await client.Indices.CreateAsync("my_index"); +---- + + +[discrete] +==== Indexing documents + +This is a simple way of indexing a document: + +[source,net] +---- +var doc = new MyDoc +{ + Id = 1, + User = "flobernd", + Message = "Trying out the client, so far so good?" +}; + +var response = await client.IndexAsync(doc, "my_index"); +---- + + +[discrete] +==== Getting documents + +You can get documents by using the following code: + +[source,net] +---- +var response = await client.GetAsync(id, idx => idx.Index("my_index")); + +if (response.IsValidResponse) +{ + var doc = response.Source; +} +---- + + +[discrete] +==== Searching documents + +This is how you can create a single match query with the .NET client: + +[source,net] +---- +var response = await client.SearchAsync(s => s + .Index("my_index") + .From(0) + .Size(10) + .Query(q => q + .Term(t => t.User, "flobernd") + ) +); + +if (response.IsValidResponse) +{ + var doc = response.Documents.FirstOrDefault(); +} +---- + + +[discrete] +==== Updating documents + +This is how you can update a document, for example to add a new field: + +[source,net] +---- +doc.Message = "This is a new message"; + +var response = await client.UpdateAsync("my_index", 1, u => u + .Doc(doc)); +---- + + +[discrete] +==== Deleting documents + +[source,net] +---- +var response = await client.DeleteAsync("my_index", 1); +---- + + +[discrete] +==== Deleting an index + +[source,net] +---- +var response = await client.Indices.DeleteAsync("my_index"); +---- + + +[discrete] +== Further reading + +* Refer to the <> page to learn more about how to use the +client the most efficiently. diff --git a/docs/reference/images/create-api-key.png b/docs/images/create-api-key.png similarity index 100% rename from docs/reference/images/create-api-key.png rename to docs/images/create-api-key.png diff --git a/docs/images/es-cloudid.jpg b/docs/images/es-cloudid.jpg new file mode 100644 index 00000000000..face79aa148 Binary files /dev/null and b/docs/images/es-cloudid.jpg differ diff --git a/docs/reference/images/es-endpoint.jpg b/docs/images/es-endpoint.jpg similarity index 100% rename from docs/reference/images/es-endpoint.jpg rename to docs/images/es-endpoint.jpg diff --git a/docs/index.asciidoc b/docs/index.asciidoc new file mode 100644 index 00000000000..b1202f34de2 --- /dev/null +++ b/docs/index.asciidoc @@ -0,0 +1,33 @@ +[[elasticsearch-net-reference]] += Elasticsearch .NET Client + +include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[] +include::{asciidoc-dir}/../../shared/attributes.asciidoc[] + +:doc-tests-src: {docdir}/../tests/Tests/Documentation +:net-client: Elasticsearch .NET Client +:latest-version: 8.15.8 + +:es-docs: https://www.elastic.co/guide/en/elasticsearch/reference/{branch} + +include::intro.asciidoc[] + +include::getting-started.asciidoc[] + +include::install.asciidoc[] + +include::connecting.asciidoc[] + +include::configuration.asciidoc[] + +include::client-concepts/client-concepts.asciidoc[] + +include::usage/index.asciidoc[] + +include::migration-guide.asciidoc[] + +include::troubleshooting.asciidoc[] + +include::redirects.asciidoc[] + +include::release-notes/release-notes.asciidoc[] \ No newline at end of file diff --git a/docs/install.asciidoc b/docs/install.asciidoc new file mode 100644 index 00000000000..09a383cda5a --- /dev/null +++ b/docs/install.asciidoc @@ -0,0 +1,95 @@ +[[installation]] +== Installation + +This page shows you how to install the .NET client for {es}. + +IMPORTANT: The v8 client for .NET does not have complete feature parity with +the v7 `NEST` client. It may not be suitable for for all applications until +additional endpoints and features are supported. We therefore recommend you thoroughly +review our <> before attempting to migrate +existing applications to the `Elastic.Clients.Elasticsearch` library. +Until the new client supports all endpoints and features your application requires, +you may continue to use the 7.17.x https://www.nuget.org/packages/NEST[NEST] client +to communicate with v8 Elasticsearch servers using compatibility mode. Refer to the +https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html[Connecting to Elasticsearch v8.x using the v7.17.x client documentation] +for guidance on configuring the 7.17.x client. + +[discrete] +[[dot-net-client]] +=== Installing the .NET client + +For SDK style projects, you can install the {es} client by running the following +.NET CLI command in your terminal: + +[source,text] +---- +dotnet add package Elastic.Clients.Elasticsearch +---- + +This command adds a package reference to your project (csproj) file for the +latest stable version of the client. + +If you prefer, you may also manually add a package reference inside your project +file: + +[source,shell] +---- + +---- +_NOTE: The version number should reflect the latest published version from +https://www.nuget.org/packages/Elastic.Clients.Elasticsearch[NuGet.org]. To install +a different version, modify the version as necessary._ + +For Visual Studio users, the .NET client can also be installed from the Package +Manager Console inside Visual Studio using the following command: + +[source,shell] +---- +Install-Package Elastic.Clients.Elasticsearch +---- + +Alternatively, search for `Elastic.Clients.Elasticsearch` in the NuGet Package +Manager UI. + +To learn how to connect the {es} client, refer to the <> section. + +[discrete] +[[compatibility]] +=== Compatibility + +The {es} client is compatible with currently maintained .NET runtime versions. +Compatibility with End of Life (EOL) .NET runtimes is not guaranteed or +supported. + +Language clients are forward compatible; meaning that the clients support +communicating with greater or equal minor versions of {es} without breaking. It +does not mean that the clients automatically support new features of newer +{es} versions; it is only possible after a release of a new client version. For +example, a 8.12 client version won't automatically support the new features of +the 8.13 version of {es}, the 8.13 client version is required for that. {es} +language clients are only backwards compatible with default distributions and +without guarantees made. + +|=== +| Elasticsearch Version | Elasticsearch-NET Branch | Supported + +| main | main | +| 8.x | 8.x | 8.x +| 7.x | 7.x | 7.17 +|=== + +Refer to the https://www.elastic.co/support/eol[end-of-life policy] for more +information. + +[discrete] +[[ci-feed]] +=== CI feed + +We publish CI builds of our client packages, including the latest +unreleased features. If you want to experiment with the latest bits, you +can add the CI feed to your list of NuGet package sources. + +Feed URL: https://f.feedz.io/elastic/all/nuget/index.json + +We do not recommend using CI builds for production applications as they are not +formally supported until they are released. \ No newline at end of file diff --git a/docs/intro.asciidoc b/docs/intro.asciidoc new file mode 100644 index 00000000000..d638ad6acac --- /dev/null +++ b/docs/intro.asciidoc @@ -0,0 +1,54 @@ +:github: https://github.com/elastic/elasticsearch-net + +[[introduction]] +== Introduction + +*Rapidly develop applications with the .NET client for {es}.* + +Designed for .NET application developers, the .NET language client +library provides a strongly typed API and query DSL for interacting with {es}. +The .NET client includes higher-level abstractions, such as +helpers for coordinating bulk indexing and update operations. It also comes with +built-in, configurable cluster failover retry mechanisms. + +The {es} .NET client is available as a https://www.nuget.org/packages/Elastic.Clients.Elasticsearch[NuGet] +package for use with .NET Core, .NET 5+, and .NET Framework (4.6.1 and later) +applications. + +_NOTE: This documentation covers the v8 .NET client for {es}, for use +with {es} 8.x versions. To develop applications targeting {es} v7, use the +https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17[v7 (NEST) client]._ + +[discrete] +[[features]] +=== Features + +* One-to-one mapping with the REST API. +* Strongly typed requests and responses for {es} APIs. +* Fluent API for building requests. +* Query DSL to assist with constructing search queries. +* Helpers for common tasks such as bulk indexing of documents. +* Pluggable serialization of requests and responses based on `System.Text.Json`. +* Diagnostics, auditing, and .NET activity integration. + +The .NET {es} client is built on the Elastic Transport library, which provides: + +* Connection management and load balancing across all available nodes. +* Request retries and dead connections handling. + +[discrete] +=== {es} version compatibility + +Language clients are forward compatible: clients support communicating +with current and later minor versions of {es}. {es} language clients are +backward compatible with default distributions only and without guarantees. + +[discrete] +=== Questions, bugs, comments, feature requests + +To submit a bug report or feature request, use +{github}/issues[GitHub issues]. + +For more general questions and comments, try the community forum +on https://discuss.elastic.co/c/elasticsearch[discuss.elastic.co]. +Mention `.NET` in the title to indicate the discussion topic. \ No newline at end of file diff --git a/docs/migration-guide.asciidoc b/docs/migration-guide.asciidoc new file mode 100644 index 00000000000..1d3ae76032c --- /dev/null +++ b/docs/migration-guide.asciidoc @@ -0,0 +1,334 @@ +[[migration-guide]] +== Migration guide: From NEST v7 to .NET Client v8 + +The following migration guide explains the current state of the client, missing +features, breaking changes and our rationale for some of the design choices we have introduced. + +[discrete] +=== Version 8 is a refresh + +[IMPORTANT] +-- +It is important to highlight that v8 of the {net-client} represents +a new start for the client design. It is important to review how this may affect +your code and usage. +-- + +Mature code becomes increasingly hard to maintain over time. +Major releases allow us to simplify and better align our language clients with +each other in terms of design. It is crucial to find the right balance +between uniformity across programming languages and the idiomatic concerns of +each language. For .NET, we typically compare and contrast with https://github.com/elastic/elasticsearch-java[Java] and https://github.com/elastic/go-elasticsearch[Go] +to make sure that our approach is equivalent for each of these. We also take +heavy inspiration from Microsoft framework design guidelines and the conventions +of the wider .NET community. + +[discrete] +==== New Elastic.Clients.Elasticsearch NuGet package + +We have shipped the new code-generated client as a +https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/[NuGet package] +with a new root namespace, `Elastic.Clients.Elasticsearch`. +The v8 client is built upon the foundations of the v7 `NEST` client, but there +are changes. By shipping as a new package, the expectation is that migration can +be managed with a phased approach. + +While this is a new package, we have aligned the major version (v8.x.x) with the +supported {es} server version to clearly indicate the client/server compatibility. +The v8 client is designed to work with version 8 of {es}. + +The v7 `NEST` client continues to be supported but will not gain new features or +support for new {es} endpoints. It should be considered deprecated in favour of +the new client. + +[discrete] +==== Limited feature set + +[CAUTION] +-- +The version 8 {net-client} does not have feature parity with the previous v7 `NEST` +high-level client. +-- + +If a feature you depend on is missing (and not explicitly documented below as a +feature that we do not plan to reintroduce), open https://github.com/elastic/elasticsearch-net/issues/new/choose[an issue] +or comment on a relevant existing issue to highlight your need to us. This will +help us prioritise our roadmap. + +[discrete] +=== Code generation + +Given the size of the {es} API surface today, it is no longer practical +to maintain thousands of types (requests, responses, queries, aggregations, etc.) +by hand. To ensure consistent, accurate, and timely alignment between language +clients and {es}, the 8.x clients, and many of the associated types are now +automatically code-generated from a https://github.com/elastic/elasticsearch-specification[shared specification]. This is a common solution to maintaining alignment between +client and server among SDKs and libraries, such as those for Azure, AWS and the +Google Cloud Platform. + +Code-generation from a specification has inevitably led to some differences +between the existing v7 `NEST` types and those available in the new v7 {net-client}. +For version 8, we generate strictly from the specification, special +casing a few areas to improve usability or to align with language idioms. + +The base type hierarchy for concepts such as `Properties`, `Aggregations` and +`Queries` is no longer present in generated code, as these arbitrary groupings do +not align with concrete concepts of the public server API. These considerations +do not preclude adding syntactic sugar and usability enhancements to types in future +releases on a case-by-case basis. + +[discrete] +=== Elastic.Transport + +The .NET client includes a transport layer responsible for abstracting HTTP +concepts and to provide functionality such as our request pipeline. This +supports round-robin load-balancing of requests to nodes, pinging failed +nodes and sniffing the cluster for node roles. + +In v7, this layer shipped as `Elasticsearch.Net` and was considered our low-level +client which could be used to send and receive raw JSON bytes between the client +and server. + +As part of the work for 8.0.0, we have moved the transport layer out into +a https://www.nuget.org/packages/Elastic.Transport[new dedicated package] and +https://github.com/elastic/elastic-transport-net[repository], named +`Elastic.Transport`. This supports reuse across future clients and allows +consumers with extremely high-performance requirements to build upon this foundation. + +[discrete] +=== System.Text.Json for serialization + +The v7 `NEST` high-level client used an internalized and modified version of +https://github.com/neuecc/Utf8Json[Utf8Json] for request and response +serialization. This was introduced for its performance improvements +over https://www.newtonsoft.com/json[Json.NET], the more common JSON framework at +the time. + +While Utf8Json provides good value, we have identified minor bugs and +performance issues that have required maintenance over time. Some of these +are hard to change without more significant effort. This library is no longer +maintained, and any such changes cannot easily be contributed back to the +original project. + +With .NET Core 3.0, Microsoft shipped new https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis[System.Text.Json APIs] +that are included in-the-box with current versions of .NET. We have adopted +`System.Text.Json` for all serialization. Consumers can still define and register +their own `Serializer` implementation for their document types should they prefer +to use a different serialization library. + +By adopting `System.Text.Json`, we now depend on a well-maintained and supported +library from Microsoft. `System.Text.Json` is designed from the ground up to support +the latest performance optimizations in .NET and, as a result, provides both fast and low-allocation serialization. + +[discrete] +=== Mockability of ElasticsearchClient + +Testing code is an important part of software development. We recommend +that consumers prefer introducing an abstraction for their use of the {net-client} +as the prefered way to decouple consuming code from client types and support unit +testing. + +To support user testing scenarios, we have unsealed the `ElasticsearchClient` +type and made its methods virtual. This supports mocking the type directly for unit +testing. This is an improvement over the original `IElasticClient` interface from +`NEST` (v7) which only supported mocking of top-level client methods. + +We have also introduced a `TestableResponseFactory` in `Elastic.Transport` to +make it easier to create response instances with specific status codes and validity +that can be used during unit testing. + +These changes are in addition to our existing support for testing with an +`InMemoryConnection`, virtualized clusters and with our +https://github.com/elastic/elasticsearch-net-abstractions/blob/master/src/Elastic.Elasticsearch.Managed[`Elastic.Elasticsearch.Managed`] library for integration +testing against real {es} instances. + +[discrete] +=== Migrating to Elastic.Clients.Elasticsearch + +[WARNING] +-- +The version 8 client does not currently have full-feature parity with `NEST`. The +client primary use case is for application developers communicating with {es}. +-- + +The version 8 client focuses on core endpoints, more specifically for common CRUD +scenarios. The intention is to reduce the feature gap in subsequent versions. Review this documentation carefully to learn about the missing features and reduced API surface details before migrating from the v7 `NEST` client! + +The choice to code-generate a new evolution of the {net-client} introduces some +significant breaking changes. + +The v8 client is shipped as a new https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/[NuGet package] +which can be installed alongside v7 `NEST`. Some consumers may prefer a phased migration with both +packages side-by-side for a short period of time to manage complex migrations. In addition, `NEST` 7.17.x can continue to be used in +https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html[compatibility mode] +with {es} 8.x servers until the v8 {net-client} features +align with application requirements. + +[discrete] +=== Breaking Changes + +[WARNING] +-- +As a result of code-generating a majority of the client types, version 8 of +the client includes multiple breaking changes. +-- + +We have strived to keep the core foundation reasonably similar, but types emitted +through code-generation are subject to change between `NEST` (v7) and the new +`Elastic.Clients.Elasticsearch` (v8) package. + +[discrete] +==== Namespaces + +The package and top-level namespace for the v8 client have been renamed to +`Elastic.Clients.Elasticsearch`. All types belong to this namespace. When +necessary, to avoid potential conflicts, types are generated into suitable +sub-namespaces based on the https://github.com/elastic/elasticsearch-specification[{es} specification]. Additional `using` directives may be required to access such types +when using the {net-client}. + +Transport layer concepts have moved to the new `Elastic.Transport` NuGet package +and related types are defined under its namespace. Some configuration and low-level transport functionality may require a `using` directive for the `Elastic.Transport` +namespace. + +[discrete] +==== Type names + +Type names may have changed from previous versions. These are not listed explicitly due to the potentially vast number of subtle differences. +Type names will now more closely align to those used in the JSON and as documented +in the {es} documentation. + +[discrete] +==== Class members + +Types may include renamed properties based on the {es} specification, +which differ from the original `NEST` property names. The types used for properties +may also have changed due to code-generation. If you identify missing or +incorrectly-typed properties, please open https://github.com/elastic/elasticsearch-net/issues/new/choose[an issue] to alert us. + +[discrete] +==== Sealing classes + +Opinions on "sealing by default" within the .NET ecosystem tend to be quite +polarized. Microsoft seal all internal types for potential performance gains +and we see a benefit in starting with that approach for the {net-client}, +even for our public API surface. + +While it prevents inheritance and, therefore, may inhibit a few consumer scenarios, +sealing by default is intended to avoid the unexpected or invalid +extension of types that could inadvertently be broken in the future. + +[discrete] +==== Removed features + +As part of the clean-slate redesign of the new client, +certain features are removed from the v8.0 client. These are listed below: + +[discrete] +===== Attribute mappings + +In previous versions of the `NEST` client, attributes could be used to configure +the mapping behaviour and inference for user types. It is recommended that +mapping be completed via the fluent API when configuring client instances. +`System.Text.Json` attributes may be used to rename +and ignore properties during source serialization. + +[discrete] +===== CAT APIs + +The https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html[CAT APIs] +of {es} are intended for human-readable usage and will no longer be supported +via the v8 {net-client}. + +[discrete] +===== Interface removal + +Several interfaces are removed to simplify the library and avoid interfaces where only a +single implementation of that interface is expected to exist, such as +`IElasticClient` in `NEST`. Abstract base classes are preferred +over interfaces across the library, as this makes it easier to add enhancements +without introducing breaking changes for derived types. + +[discrete] +==== Missing features + +The following are some of the main features which +have not been re-implemented for the v8 client. +These might be reviewed and prioritized for inclusion in +future releases. + +* Query DSL operators for combining queries. +* Scroll Helper. +* Fluent API for union types. +* `AutoMap` for field datatype inference. +* Visitor pattern support for types such as `Properties`. +* Support for `JoinField` which affects `ChildrenAggregation`. +* Conditionless queries. +* DiagnosticSources have been removed in `Elastic.Transport` to provide a clean-slate +for an improved diagnostics story. The {net-client} emits https://opentelemetry.io/[OpenTelemetry] compatible `Activity` spans which can be consumed by APM agents such as the https://www.elastic.co/guide/en/apm/agent/dotnet/current/index.html[Elastic APM Agent for .NET]. +* Documentation is a work in progress, and we will expand on the documented scenarios +in future releases. + +[discrete] +=== Reduced API surface + +In the current versions of the code-generated .NET client, supporting commonly used +endpoints is critical. Some specific queries and aggregations need further work to generate code correctly, +hence they are not included yet. +Ensure that the features you are using are currently supported before migrating. + +An up to date list of all supported and unsupported endpoints can be found on https://github.com/elastic/elasticsearch-net/issues/7890[GitHub]. + +[discrete] +=== Workarounds for missing features + +If you encounter a missing feature with the v8 client, there are several ways to temporarily work around this issue until we officially reintroduce the feature. + +`NEST` 7.17.x can continue to be used in +https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html[compatibility mode] +with {es} 8.x servers until the v8 {net-client} features +align with application requirements. + +As a last resort, the low-level client `Elastic.Transport` can be used to create any desired request by hand: + +[source,csharp] +---- +public class MyRequestParameters : RequestParameters +{ + public bool Pretty + { + get => Q("pretty"); + init => Q("pretty", value); + } +} + +// ... + +var body = """ + { + "name": "my-api-key", + "expiration": "1d", + "...": "..." + } + """; + +MyRequestParameters requestParameters = new() +{ + Pretty = true +}; + +var pathAndQuery = requestParameters.CreatePathWithQueryStrings("/_security/api_key", + client.ElasticsearchClientSettings); +var endpointPath = new EndpointPath(Elastic.Transport.HttpMethod.POST, pathAndQuery); + +// Or, if the path does not contain query parameters: +// new EndpointPath(Elastic.Transport.HttpMethod.POST, "my_path") + +var response = await client.Transport + .RequestAsync( + endpointPath, + PostData.String(body), + null, + null, + cancellationToken: default) + .ConfigureAwait(false); +---- \ No newline at end of file diff --git a/docs/redirects.asciidoc b/docs/redirects.asciidoc new file mode 100644 index 00000000000..cb97e146ff1 --- /dev/null +++ b/docs/redirects.asciidoc @@ -0,0 +1,24 @@ +["appendix",role="exclude",id="redirects"] += Deleted pages + +The following pages have moved or been deleted. + +[role="exclude",id="configuration-options"] +== Configuration options + +This page has moved. See <>. + +[role="exclude",id="nest"] +== NEST - High level client + +This page has been deleted. + +[role="exclude",id="indexing-documents"] +== Indexing documents + +This page has been deleted. + +[role="exclude",id="bulkall-observable"] +== Multiple documents with `BulkAllObservable` helper + +This page has been deleted. \ No newline at end of file diff --git a/docs/reference/_options_on_elasticsearchclientsettings.md b/docs/reference/_options_on_elasticsearchclientsettings.md deleted file mode 100644 index 49d5f904bd6..00000000000 --- a/docs/reference/_options_on_elasticsearchclientsettings.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/_options_on_elasticsearchclientsettings.html ---- - -# Options on ElasticsearchClientSettings [_options_on_elasticsearchclientsettings] - -The following is a list of available connection configuration options on `ElasticsearchClientSettings`: - -`Authentication` -: An implementation of `IAuthenticationHeader` describing what http header to use to authenticate with the product. - - ``` - `BasicAuthentication` for basic authentication - ``` - ``` - `ApiKey` for simple secret token - ``` - ``` - `Base64ApiKey` for Elastic Cloud style encoded api keys - ``` - - -`ClientCertificate` -: Use the following certificates to authenticate all HTTP requests. You can also set them on individual request using `ClientCertificates`. - -`ClientCertificates` -: Use the following certificates to authenticate all HTTP requests. You can also set them on individual request using `ClientCertificates`. - -`ConnectionLimit` -: Limits the number of concurrent connections that can be opened to an endpoint. Defaults to 80 (see `DefaultConnectionLimit`). - - For Desktop CLR, this setting applies to the `DefaultConnectionLimit` property on the `ServicePointManager` object when creating `ServicePoint` objects, affecting the default `IConnection` implementation. - - For Core CLR, this setting applies to the `MaxConnectionsPerServer` property on the `HttpClientHandler` instances used by the `HttpClient` inside the default `IConnection` implementation. - - -`DeadTimeout` -: The time to put dead nodes out of rotation (this will be multiplied by the number of times they’ve been dead). - -`DefaultDisableIdInference` -: Disables automatic Id inference for given CLR types. - - The client by default will use the value of a property named `Id` on a CLR type as the `_id` to send to {{es}}. Adding a type will disable this behaviour for that CLR type. If `Id` inference should be disabled for all CLR types, use `DefaultDisableIdInference`. - - -`DefaultFieldNameInferrer` -: Specifies how field names are inferred from CLR property names. - - By default, the client camel cases property names. For example, CLR property `EmailAddress` will be inferred as "emailAddress" {{es}} document field name. - - -`DefaultIndex` -: The default index to use for a request when no index has been explicitly specified and no default indices are specified for the given CLR type specified for the request. - -`DefaultMappingFor` -: Specify how the mapping is inferred for a given CLR type. The mapping can infer the index, id and relation name for a given CLR type, as well as control serialization behaviour for CLR properties. - -`DisableAutomaticProxyDetection` -: Disabled proxy detection on the webrequest, in some cases this may speed up the first connection your appdomain makes, in other cases it will actually increase the time for the first connection. No silver bullet! Use with care! - -`DisableDirectStreaming` -: When set to true will disable (de)serializing directly to the request and response stream and return a byte[] copy of the raw request and response. Defaults to false. - -`DisablePing` -: This signals that we do not want to send initial pings to unknown/previously dead nodes and just send the call straightaway. - -`DnsRefreshTimeout` -: DnsRefreshTimeout for the connections. Defaults to 5 minutes. - -`EnableDebugMode` -: Turns on settings that aid in debugging like `DisableDirectStreaming()` and `PrettyJson()` so that the original request and response JSON can be inspected. It also always asks the server for the full stack trace on errors. - -`EnableHttpCompression` -: Enable gzip compressed requests and responses. - -`EnableHttpPipelining` -: Whether HTTP pipelining is enabled. The default is `true`. - -`EnableTcpKeepAlive` -: Sets the keep-alive option on a TCP connection. - - For Desktop CLR, sets `ServicePointManager`.`SetTcpKeepAlive`. - - -`EnableTcpStats` -: Enable statistics about TCP connections to be collected when making a request. - -`GlobalHeaders` -: Try to send these headers for every request. - -`GlobalQueryStringParameters` -: Append these query string parameters automatically to every request. - -`MaxDeadTimeout` -: The maximum amount of time a node is allowed to marked dead. - -`MaximumRetries` -: When a retryable exception occurs or status code is returned this controls the maximum amount of times we should retry the call to {{es}}. - -`MaxRetryTimeout` -: Limits the total runtime including retries separately from `RequestTimeout`. When not specified defaults to `RequestTimeout` which itself defaults to 60 seconds. - -`MemoryStreamFactory` -: Provides a memory stream factory. - -`NodePredicate` -: Register a predicate to select which nodes that you want to execute API calls on. Note that sniffing requests omit this predicate and always execute on all nodes. When using an `IConnectionPool` implementation that supports reseeding of nodes, this will default to omitting master only node from regular API calls. When using static or single node connection pooling it is assumed the list of node you instantiate the client with should be taken verbatim. - -`OnRequestCompleted` -: Allows you to register a callback every time a an API call is returned. - -`OnRequestDataCreated` -: An action to run when the `RequestData` for a request has been created. - -`PingTimeout` -: The timeout in milliseconds to use for ping requests, which are issued to determine whether a node is alive. - -`PrettyJson` -: Provide hints to serializer and products to produce pretty, non minified json. - - Note: this is not a guarantee you will always get prettified json. - - -`Proxy` -: If your connection has to go through proxy, use this method to specify the proxy url. - -`RequestTimeout` -: The timeout in milliseconds for each request to {{es}}. - -`ServerCertificateValidationCallback` -: Register a `ServerCertificateValidationCallback` per request. - -`SkipDeserializationForStatusCodes` -: Configure the client to skip deserialization of certain status codes, for example, you run {{es}} behind a proxy that returns an unexpected json format. - -`SniffLifeSpan` -: Force a new sniff for the cluster when the cluster state information is older than the specified timespan. - -`SniffOnConnectionFault` -: Force a new sniff for the cluster state every time a connection dies. - -`SniffOnStartup` -: Sniff the cluster state immediately on startup. - -`ThrowExceptions` -: Instead of following a c/go like error checking on response. `IsValid` do throw an exception (except when `SuccessOrKnownError` is false) on the client when a call resulted in an exception on either the client or the {{es}} server. - - Reasons for such exceptions could be search parser errors, index missing exceptions, and so on. - - -`TransferEncodingChunked` -: Whether the request should be sent with chunked Transfer-Encoding. - -`UserAgent` -: The user agent string to send with requests. Useful for debugging purposes to understand client and framework versions that initiate requests to {{es}}. - -## ElasticsearchClientSettings with ElasticsearchClient [_elasticsearchclientsettings_with_elasticsearchclient] - -Here’s an example to demonstrate setting configuration options using the client. - -```csharp -var settings= new ElasticsearchClientSettings() - .DefaultMappingFor(i => i - .IndexName("my-projects") - .IdProperty(p => p.Name) - ) - .EnableDebugMode() - .PrettyJson() - .RequestTimeout(TimeSpan.FromMinutes(2)); - -var client = new ElasticsearchClient(settings); -``` - - diff --git a/docs/reference/aggregations.md b/docs/reference/aggregations.md deleted file mode 100644 index 8a3ca8e8327..00000000000 --- a/docs/reference/aggregations.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/aggregations.html ---- - -# Aggregation examples [aggregations] - -This page demonstrates how to use aggregations. - - -## Top-level aggreggation [_top_level_aggreggation] - - -### Fluent API [_fluent_api] - -```csharp -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .MatchAll(_ => {}) - ) - .Aggregations(aggregations => aggregations - .Add("agg_name", aggregation => aggregation - .Max(max => max - .Field(x => x.Age) - ) - ) - ) - .Size(10) - ); -``` - - -### Object initializer API [_object_initializer_api] - -```csharp -var response = await client.SearchAsync(new SearchRequest("persons") -{ - Query = Query.MatchAll(new MatchAllQuery()), - Aggregations = new Dictionary - { - { "agg_name", Aggregation.Max(new MaxAggregation - { - Field = Infer.Field(x => x.Age) - })} - }, - Size = 10 -}); -``` - - -### Consume the response [_consume_the_response] - -```csharp -var max = response.Aggregations!.GetMax("agg_name")!; -Console.WriteLine(max.Value); -``` - - -## Sub-aggregation [_sub_aggregation] - - -### Fluent API [_fluent_api_2] - -```csharp -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .MatchAll(_ => {}) - ) - .Aggregations(aggregations => aggregations - .Add("firstnames", aggregation => aggregation - .Terms(terms => terms - .Field(x => x.FirstName) - ) - .Aggregations(aggregations => aggregations - .Add("avg_age", aggregation => aggregation - .Max(avg => avg - .Field(x => x.Age) - ) - ) - ) - ) - ) - .Size(10) - ); -``` - - -### Object initializer API [_object_initializer_api_2] - -```csharp -var topLevelAggregation = Aggregation.Terms(new TermsAggregation -{ - Field = Infer.Field(x => x.FirstName) -}); - -topLevelAggregation.Aggregations = new Dictionary -{ - { "avg_age", new MaxAggregation - { - Field = Infer.Field(x => x.Age) - }} -}; - -var response = await client.SearchAsync(new SearchRequest("persons") -{ - Query = Query.MatchAll(new MatchAllQuery()), - Aggregations = new Dictionary - { - { "firstnames", topLevelAggregation} - }, - Size = 10 -}); -``` - - -### Consume the response [_consume_the_response_2] - -```csharp -var firstnames = response.Aggregations!.GetStringTerms("firstnames")!; -foreach (var bucket in firstnames.Buckets) -{ - var avg = bucket.Aggregations.GetAverage("avg_age")!; - Console.WriteLine($"The average age for persons named '{bucket.Key}' is {avg}"); -} -``` - diff --git a/docs/reference/client-concepts.md b/docs/reference/client-concepts.md deleted file mode 100644 index 73655604f4a..00000000000 --- a/docs/reference/client-concepts.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/client-concepts.html ---- - -# Client concepts [client-concepts] - -The .NET client for {{es}} maps closely to the original {{es}} API. All -requests and responses are exposed through types, making it ideal for getting up and running quickly. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md deleted file mode 100644 index 0577876926e..00000000000 --- a/docs/reference/configuration.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/configuration.html ---- - -# Configuration [configuration] - -Connecting to {{es}} with the client is easy, but it’s possible that you’d like to change the default connection behaviour. There are a number of configuration options available on `ElasticsearchClientSettings` that can be used to control how the client interact with {{es}}. - - diff --git a/docs/reference/connecting.md b/docs/reference/connecting.md deleted file mode 100644 index 5c7fc326ca2..00000000000 --- a/docs/reference/connecting.md +++ /dev/null @@ -1,123 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/connecting.html ---- - -# Connecting [connecting] - -This page contains the information you need to create an instance of the .NET Client for {{es}} that connects to your {{es}} cluster. - -It’s possible to connect to your {{es}} cluster via a single node, or by specifying multiple nodes using a node pool. Using a node pool has a few advantages over a single node, such as load balancing and cluster failover support. The client provides convenient configuration options to connect to an Elastic Cloud deployment. - -::::{important} -Client applications should create a single instance of `ElasticsearchClient` that is used throughout your application for its entire lifetime. Internally the client manages and maintains HTTP connections to nodes, reusing them to optimize performance. If you use a dependency injection container for your application, the client instance should be registered with a singleton lifetime. -:::: - - - -## Connecting to a cloud deployment [cloud-deployment] - -[Elastic Cloud](docs-content://deploy-manage/deploy/elastic-cloud/cloud-hosted.md) is the easiest way to get started with {{es}}. When connecting to Elastic Cloud with the .NET {{es}} client you should always use the Cloud ID. You can find this value within the "Manage Deployment" page after you’ve created a cluster (look in the top-left if you’re in Kibana). - -We recommend using a Cloud ID whenever possible because your client will be automatically configured for optimal use with Elastic Cloud, including HTTPS and HTTP compression. - -Connecting to an Elasticsearch Service deployment is achieved by providing the unique Cloud ID for your deployment when configuring the `ElasticsearchClient` instance. You also require suitable credentials, either a username and password or an API key that your application uses to authenticate with your deployment. - -As a security best practice, it is recommended to create a dedicated API key per application, with permissions limited to only those required for any API calls the application is authorized to make. - -The following snippet demonstrates how to create a client instance that connects to an {{es}} deployment in the cloud. - -```csharp -using Elastic.Clients.Elasticsearch; -using Elastic.Transport; - -var client = new ElasticsearchClient("", new ApiKey("")); <1> -``` - -1. Replace the placeholder string values above with your cloud ID and the API key configured for your application to access your deployment. - - - -## Connecting to a single node [single-node] - -Single node configuration is best suited to connections to a multi-node cluster running behind a load balancer or reverse proxy, which is exposed via a single URL. It may also be convenient to use a single node during local application development. If the URL represents a single {{es}} node, be aware that this offers no resiliency should the server be unreachable or unresponsive. - -By default, security features such as authentication and TLS are enabled on {{es}} clusters. When you start {{es}} for the first time, TLS is configured automatically for the HTTP layer. A CA certificate is generated and stored on disk which is used to sign the certificates for the HTTP layer of the {{es}} cluster. - -In order for the client to establish a connection with the cluster over HTTPS, the CA certificate must be trusted by the client application. The simplest choice is to use the hex-encoded SHA-256 fingerprint of the CA certificate. The CA fingerprint is output to the terminal when you start {{es}} for the first time. You’ll see a distinct block like the one below in the output from {{es}} (you may have to scroll up if it’s been a while): - -```sh ----------------------------------------------------------------- --> Elasticsearch security features have been automatically configured! --> Authentication is enabled and cluster connections are encrypted. - --> Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`): - lhQpLELkjkrawaBoaz0Q - --> HTTP CA certificate SHA-256 fingerprint: - a52dd93511e8c6045e21f16654b77c9ee0f34aea26d9f40320b531c474676228 -... ----------------------------------------------------------------- -``` - -Note down the `elastic` user password and HTTP CA fingerprint for the next sections. - -The CA fingerprint can also be retrieved at any time from a running cluster using the following command: - -```shell -openssl x509 -fingerprint -sha256 -in config/certs/http_ca.crt -``` - -The command returns the security certificate, including the fingerprint. The `issuer` should be `Elasticsearch security auto-configuration HTTP CA`. - -```shell -issuer= /CN=Elasticsearch security auto-configuration HTTP CA -SHA256 Fingerprint= -``` - -Visit the [Start the Elastic Stack with security enabled automatically](docs-content://deploy-manage/deploy/self-managed/installing-elasticsearch.md) documentation for more information. - -The following snippet shows you how to create a client instance that connects to your {{es}} cluster via a single node, using the CA fingerprint: - -```csharp -using Elastic.Clients.Elasticsearch; -using Elastic.Transport; - -var settings = new ElasticsearchClientSettings(new Uri("https://localhost:9200")) - .CertificateFingerprint("") - .Authentication(new BasicAuthentication("", "")); - -var client = new ElasticsearchClient(settings); -``` - -The preceding snippet demonstrates configuring the client to authenticate by providing a username and password with basic authentication. If preferred, you may also use `ApiKey` authentication as shown in the cloud connection example. - - -## Connecting to multiple nodes using a node pool [multiple-nodes] - -To provide resiliency, you should configure multiple nodes for your cluster to which the client attempts to communicate. By default, the client cycles through nodes for each request in a round robin fashion. The client also tracks unhealthy nodes and avoids sending requests to them until they become healthy. - -This configuration is best suited to connect to a known small sized cluster, where you do not require sniffing to detect the cluster topology. - -The following snippet shows you how to connect to multiple nodes by using a static node pool: - -```csharp -using Elastic.Clients.Elasticsearch; -using Elastic.Transport; - -var nodes = new Uri[] -{ - new Uri("https://myserver1:9200"), - new Uri("https://myserver2:9200"), - new Uri("https://myserver3:9200") -}; - -var pool = new StaticNodePool(nodes); - -var settings = new ElasticsearchClientSettings(pool) - .CertificateFingerprint("") - .Authentication(new ApiKey("")); - -var client = new ElasticsearchClient(settings); -``` - diff --git a/docs/reference/esql.md b/docs/reference/esql.md deleted file mode 100644 index f1db14c16cc..00000000000 --- a/docs/reference/esql.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -navigation_title: "Using ES|QL" -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/esql.html ---- - -# ES|QL in the .NET client [esql] - - -This page helps you understand and use [ES|QL](docs-content://explore-analyze/query-filter/languages/esql.md) in the .NET client. - -There are two ways to use ES|QL in the .NET client: - -* Use the Elasticsearch [ES|QL API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) directly: This is the most flexible approach, but it’s also the most complex because you must handle results in their raw form. You can choose the precise format of results, such as JSON, CSV, or text. -* Use ES|QL high-level helpers: These helpers take care of parsing the raw response into something readily usable by the application. Several helpers are available for different use cases, such as object mapping, cursor traversal of results (in development), and dataframes (in development). - - -## How to use the ES|QL API [esql-how-to] - -The [ES|QL query API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) allows you to specify how results should be returned. You can choose a [response format](docs-content://explore-analyze/query-filter/languages/esql-rest.md#esql-rest-format) such as CSV, text, or JSON, then fine-tune it with parameters like column separators and locale. - -The following example gets ES|QL results as CSV and parses them: - -```csharp -var response = await client.Esql.QueryAsync(r => r - .Query("FROM index") - .Format("csv") -); -var csvContents = Encoding.UTF8.GetString(response.Data); -``` - - -## Consume ES|QL results [esql-consume-results] - -The previous example showed that although the raw ES|QL API offers maximum flexibility, additional work is required in order to make use of the result data. - -To simplify things, try working with these three main representations of ES|QL results (each with its own mapping helper): - -* **Objects**, where each row in the results is mapped to an object from your application domain. This is similar to what ORMs (object relational mappers) commonly do. -* **Cursors**, where you scan the results row by row and access the data using column names. This is similar to database access libraries. -* **Dataframes**, where results are organized in a column-oriented structure that allows efficient processing of column data. - -```csharp -// ObjectAPI example -var response = await client.Esql.QueryAsObjectsAsync(x => x.Query("FROM index")); -foreach (var person in response) -{ - // ... -} -``` diff --git a/docs/reference/examples.md b/docs/reference/examples.md deleted file mode 100644 index 67cc8034ae0..00000000000 --- a/docs/reference/examples.md +++ /dev/null @@ -1,164 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/examples.html ---- - -# CRUD usage examples [examples] - -This page helps you to understand how to perform various basic {{es}} CRUD (create, read, update, delete) operations using the .NET client. It demonstrates how to create a document by indexing an object into {{es}}, read a document back, retrieving it by ID or performing a search, update one of the fields in a document and delete a specific document. - -These examples assume you have an instance of the `ElasticsearchClient` accessible via a local variable named `client` and several using directives in your C# file. - -```csharp -using System; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.QueryDsl; -var client = new ElasticsearchClient(); <1> -``` - -1. The default constructor, assumes an unsecured {{es}} server is running and exposed on *http://localhost:9200*. See [connecting](/reference/connecting.md) for examples of connecting to secured servers and [Elastic Cloud](https://www.elastic.co/cloud) deployments. - - -The examples operate on data representing tweets. Tweets are modelled in the client application using a C# class named *Tweet* containing several properties that map to the document structure being stored in {{es}}. - -```csharp -public class Tweet -{ - public int Id { get; set; } <1> - public string User { get; set; } - public DateTime PostDate { get; set; } - public string Message { get; set; } -} -``` - -1. By default, the .NET client will try to find a property called `Id` on the class. When such a property is present it will index the document into {{es}} using the ID specified by the value of this property. - - - -## Indexing a document [indexing-net] - -Documents can be indexed by creating an instance representing a tweet and indexing it via the client. In these examples, we will work with an index named *my-tweet-index*. - -```csharp -var tweet = new Tweet <1> -{ - Id = 1, - User = "stevejgordon", - PostDate = new DateTime(2009, 11, 15), - Message = "Trying out the client, so far so good?" -}; - -var response = await client.IndexAsync(tweet, "my-tweet-index"); <2> - -if (response.IsValidResponse) <3> -{ - Console.WriteLine($"Index document with ID {response.Id} succeeded."); <4> -} -``` - -1. Create an instance of the `Tweet` class with relevant properties set. -2. Prefer the async APIs, which require awaiting the response. -3. Check the `IsValid` property on the response to confirm that the request and operation succeeded. -4. Access the `IndexResponse` properties, such as the ID, if necessary. - - - -## Getting a document [getting-net] - -```csharp -var response = await client.GetAsync(1, idx => idx.Index("my-tweet-index")); <1> - -if (response.IsValidResponse) -{ - var tweet = response.Source; <2> -} -``` - -1. The `GetResponse` is mapped 1-to-1 with the Elasticsearch JSON response. -2. The original document is deserialized as an instance of the Tweet class, accessible on the response via the `Source` property. - - - -## Searching for documents [searching-net] - -The client exposes a fluent interface and a powerful query DSL for searching. - -```csharp -var response = await client.SearchAsync(s => s <1> - .Index("my-tweet-index") <2> - .From(0) - .Size(10) - .Query(q => q - .Term(t => t.User, "stevejgordon") <3> - ) -); - -if (response.IsValidResponse) -{ - var tweet = response.Documents.FirstOrDefault(); <4> -} -``` - -1. The generic type argument specifies the `Tweet` class, which is used when deserialising the hits from the response. -2. The index can be omitted if a `DefaultIndex` has been configured on `ElasticsearchClientSettings`, or a specific index was configured when mapping this type. -3. Execute a term query against the `user` field, searching for tweets authored by the user *stevejgordon*. -4. Documents matched by the query are accessible via the `Documents` collection property on the `SearchResponse`. - - -You may prefer using the object initializer syntax for requests if lambdas aren’t your thing. - -```csharp -var request = new SearchRequest("my-tweet-index") <1> -{ - From = 0, - Size = 10, - Query = new TermQuery("user") { Value = "stevejgordon" } -}; - -var response = await client.SearchAsync(request); <2> - -if (response.IsValidResponse) -{ - var tweet = response.Documents.FirstOrDefault(); -} -``` - -1. Create an instance of `SearchRequest`, setting properties to control the search operation. -2. Pass the request to the `SearchAsync` method on the client. - - - -## Updating documents [updating-net] - -Documents can be updated in several ways, including by providing a complete replacement for an existing document ID. - -```csharp -tweet.Message = "This is a new message"; <1> - -var response = await client.UpdateAsync("my-tweet-index", 1, u => u - .Doc(tweet)); <2> - -if (response.IsValidResponse) -{ - Console.WriteLine("Update document succeeded."); -} -``` - -1. Update a property on the existing tweet instance. -2. Send the updated tweet object in the update request. - - - -## Deleting documents [deleting-net] - -Documents can be deleted by providing the ID of the document to remove. - -```csharp -var response = await client.DeleteAsync("my-tweet-index", 1); - -if (response.IsValidResponse) -{ - Console.WriteLine("Delete document succeeded."); -} -``` - diff --git a/docs/reference/getting-started.md b/docs/reference/getting-started.md deleted file mode 100644 index 39f6b5f0484..00000000000 --- a/docs/reference/getting-started.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/getting-started-net.html - - https://www.elastic.co/guide/en/serverless/current/elasticsearch-dot-net-client-getting-started.html ---- - -# Getting started [getting-started-net] - -This page guides you through the installation process of the .NET client, shows you how to instantiate the client, and how to perform basic Elasticsearch operations with it. - - -### Requirements [_requirements] - -* .NET Core, .NET 5+ or .NET Framework (4.6.1 and higher). - - -### Installation [_installation] - -To install the latest version of the client for SDK style projects, run the following command: - -```shell -dotnet add package Elastic.Clients.Elasticsearch -``` - -Refer to the [*Installation*](/reference/installation.md) page to learn more. - - -### Connecting [_connecting] - -You can connect to the Elastic Cloud using an API key and the Elasticsearch endpoint. - -```csharp -var client = new ElasticsearchClient("", new ApiKey("")); -``` - -Your Elasticsearch endpoint can be found on the **My deployment** page of your deployment: - -![Finding Elasticsearch endpoint](images/es-endpoint.jpg) - -You can generate an API key on the **Management** page under Security. - -![Create API key](images/create-api-key.png) - -For other connection options, refer to the [*Connecting*](/reference/connecting.md) section. - - -### Operations [_operations] - -Time to use Elasticsearch! This section walks you through the basic, and most important, operations of Elasticsearch. For more operations and more advanced examples, refer to the [*CRUD usage examples*](/reference/examples.md) page. - - -#### Creating an index [_creating_an_index] - -This is how you create the `my_index` index: - -```csharp -var response = await client.Indices.CreateAsync("my_index"); -``` - - -#### Indexing documents [_indexing_documents] - -This is a simple way of indexing a document: - -```csharp -var doc = new MyDoc -{ - Id = 1, - User = "flobernd", - Message = "Trying out the client, so far so good?" -}; - -var response = await client.IndexAsync(doc, "my_index"); -``` - - -#### Getting documents [_getting_documents] - -You can get documents by using the following code: - -```csharp -var response = await client.GetAsync(id, idx => idx.Index("my_index")); - -if (response.IsValidResponse) -{ - var doc = response.Source; -} -``` - - -#### Searching documents [_searching_documents] - -This is how you can create a single match query with the .NET client: - -```csharp -var response = await client.SearchAsync(s => s - .Index("my_index") - .From(0) - .Size(10) - .Query(q => q - .Term(t => t.User, "flobernd") - ) -); - -if (response.IsValidResponse) -{ - var doc = response.Documents.FirstOrDefault(); -} -``` - - -#### Updating documents [_updating_documents] - -This is how you can update a document, for example to add a new field: - -```csharp -doc.Message = "This is a new message"; - -var response = await client.UpdateAsync("my_index", 1, u => u - .Doc(doc)); -``` - - -#### Deleting documents [_deleting_documents] - -```csharp -var response = await client.DeleteAsync("my_index", 1); -``` - - -#### Deleting an index [_deleting_an_index] - -```csharp -var response = await client.Indices.DeleteAsync("my_index"); -``` - - -## Further reading [_further_reading] - -* Refer to the [*Usage recommendations*](/reference/recommendations.md) page to learn more about how to use the client the most efficiently. diff --git a/docs/reference/index.md b/docs/reference/index.md deleted file mode 100644 index 90ef20d616f..00000000000 --- a/docs/reference/index.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/index.html - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/introduction.html ---- - -# .NET [introduction] - -**Rapidly develop applications with the .NET client for {{es}}.** - -Designed for .NET application developers, the .NET language client library provides a strongly typed API and query DSL for interacting with {{es}}. The .NET client includes higher-level abstractions, such as helpers for coordinating bulk indexing and update operations. It also comes with built-in, configurable cluster failover retry mechanisms. - -The {{es}} .NET client is available as a [NuGet](https://www.nuget.org/packages/Elastic.Clients.Elasticsearch) package for use with .NET Core, .NET 5+, and .NET Framework (4.6.1 and later) applications. - -*NOTE: This documentation covers the v8 .NET client for {{es}}, for use with {{es}} 8.x versions. To develop applications targeting {{es}} v7, use the [v7 (NEST) client](https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17).* - - -## Features [features] - -* One-to-one mapping with the REST API. -* Strongly typed requests and responses for {{es}} APIs. -* Fluent API for building requests. -* Query DSL to assist with constructing search queries. -* Helpers for common tasks such as bulk indexing of documents. -* Pluggable serialization of requests and responses based on `System.Text.Json`. -* Diagnostics, auditing, and .NET activity integration. - -The .NET {{es}} client is built on the Elastic Transport library, which provides: - -* Connection management and load balancing across all available nodes. -* Request retries and dead connections handling. - - -## {{es}} version compatibility [_es_version_compatibility] - -Language clients are forward compatible: clients support communicating with current and later minor versions of {{es}}. {{es}} language clients are backward compatible with default distributions only and without guarantees. - - -## Questions, bugs, comments, feature requests [_questions_bugs_comments_feature_requests] - -To submit a bug report or feature request, use [GitHub issues](https://github.com/elastic/elasticsearch-net/issues). - -For more general questions and comments, try the community forum on [discuss.elastic.co](https://discuss.elastic.co/c/elasticsearch). Mention `.NET` in the title to indicate the discussion topic. - diff --git a/docs/reference/installation.md b/docs/reference/installation.md deleted file mode 100644 index 8fbb2dcd2e9..00000000000 --- a/docs/reference/installation.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/installation.html ---- - -# Installation [installation] - -This page shows you how to install the .NET client for {{es}}. - -::::{important} -The v8 client for .NET does not have complete feature parity with the v7 `NEST` client. It may not be suitable for for all applications until additional endpoints and features are supported. We therefore recommend you thoroughly review our [release notes](/release-notes/index.md) before attempting to migrate existing applications to the `Elastic.Clients.Elasticsearch` library. Until the new client supports all endpoints and features your application requires, you may continue to use the 7.17.x [NEST](https://www.nuget.org/packages/NEST) client to communicate with v8 Elasticsearch servers using compatibility mode. Refer to the [Connecting to Elasticsearch v8.x using the v7.17.x client documentation](https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html) for guidance on configuring the 7.17.x client. -:::: - - - -## Installing the .NET client [dot-net-client] - -For SDK style projects, you can install the {{es}} client by running the following .NET CLI command in your terminal: - -```text -dotnet add package Elastic.Clients.Elasticsearch -``` - -This command adds a package reference to your project (csproj) file for the latest stable version of the client. - -If you prefer, you may also manually add a package reference inside your project file: - -```shell - -``` - -*NOTE: The version number should reflect the latest published version from [NuGet.org](https://www.nuget.org/packages/Elastic.Clients.Elasticsearch). To install a different version, modify the version as necessary.* - -For Visual Studio users, the .NET client can also be installed from the Package Manager Console inside Visual Studio using the following command: - -```shell -Install-Package Elastic.Clients.Elasticsearch -``` - -Alternatively, search for `Elastic.Clients.Elasticsearch` in the NuGet Package Manager UI. - -To learn how to connect the {{es}} client, refer to the [Connecting](/reference/connecting.md) section. - - -## Compatibility [compatibility] - -The {{es}} client is compatible with currently maintained .NET runtime versions. Compatibility with End of Life (EOL) .NET runtimes is not guaranteed or supported. - -Language clients are forward compatible; meaning that the clients support communicating with greater or equal minor versions of {{es}} without breaking. It does not mean that the clients automatically support new features of newer {{es}} versions; it is only possible after a release of a new client version. For example, a 8.12 client version won’t automatically support the new features of the 8.13 version of {{es}}, the 8.13 client version is required for that. {{es}} language clients are only backwards compatible with default distributions and without guarantees made. - -| Elasticsearch Version | Elasticsearch-NET Branch | Supported | -| --- | --- | --- | -| main | main | | -| 8.x | 8.x | 8.x | -| 7.x | 7.x | 7.17 | - -Refer to the [end-of-life policy](https://www.elastic.co/support/eol) for more information. - - -## CI feed [ci-feed] - -We publish CI builds of our client packages, including the latest unreleased features. If you want to experiment with the latest bits, you can add the CI feed to your list of NuGet package sources. - -Feed URL: [https://f.feedz.io/elastic/all/nuget/index.json](https://f.feedz.io/elastic/all/nuget/index.json) - -We do not recommend using CI builds for production applications as they are not formally supported until they are released. - diff --git a/docs/reference/mappings.md b/docs/reference/mappings.md deleted file mode 100644 index b28b8ba4a5e..00000000000 --- a/docs/reference/mappings.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/mappings.html ---- - -# Custom mapping examples [mappings] - -This page demonstrates how to configure custom mappings on an index. - - -## Configure mappings during index creation [_configure_mappings_during_index_creation] - -```csharp -await client.Indices.CreateAsync(index => index - .Index("index") - .Mappings(mappings => mappings - .Properties(properties => properties - .IntegerNumber(x => x.Age!) - .Keyword(x => x.FirstName!, keyword => keyword.Index(false)) - ) - ) -); -``` - - -## Configure mappings after index creation [_configure_mappings_after_index_creation] - -```csharp -await client.Indices.PutMappingAsync(mappings => mappings - .Indices("index") - .Properties(properties => properties - .IntegerNumber(x => x.Age!) - .Keyword(x => x.FirstName!, keyword => keyword.Index(false)) - ) -); -``` - diff --git a/docs/reference/migration-guide.md b/docs/reference/migration-guide.md deleted file mode 100644 index 43a4bcec67d..00000000000 --- a/docs/reference/migration-guide.md +++ /dev/null @@ -1,222 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/migration-guide.html ---- - -# Migration guide: From NEST v7 to .NET Client v8 [migration-guide] - -The following migration guide explains the current state of the client, missing features, breaking changes and our rationale for some of the design choices we have introduced. - - -## Version 8 is a refresh [_version_8_is_a_refresh] - -::::{important} -It is important to highlight that v8 of the Elasticsearch .NET Client represents a new start for the client design. It is important to review how this may affect your code and usage. - -:::: - - -Mature code becomes increasingly hard to maintain over time. Major releases allow us to simplify and better align our language clients with each other in terms of design. It is crucial to find the right balance between uniformity across programming languages and the idiomatic concerns of each language. For .NET, we typically compare and contrast with [Java](https://github.com/elastic/elasticsearch-java) and [Go](https://github.com/elastic/go-elasticsearch) to make sure that our approach is equivalent for each of these. We also take heavy inspiration from Microsoft framework design guidelines and the conventions of the wider .NET community. - - -### New Elastic.Clients.Elasticsearch NuGet package [_new_elastic_clients_elasticsearch_nuget_package] - -We have shipped the new code-generated client as a [NuGet package](https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/) with a new root namespace, `Elastic.Clients.Elasticsearch`. The v8 client is built upon the foundations of the v7 `NEST` client, but there are changes. By shipping as a new package, the expectation is that migration can be managed with a phased approach. - -While this is a new package, we have aligned the major version (v8.x.x) with the supported {{es}} server version to clearly indicate the client/server compatibility. The v8 client is designed to work with version 8 of {{es}}. - -The v7 `NEST` client continues to be supported but will not gain new features or support for new {{es}} endpoints. It should be considered deprecated in favour of the new client. - - -### Limited feature set [_limited_feature_set] - -::::{warning} -The version 8 Elasticsearch .NET Client does not have feature parity with the previous v7 `NEST` high-level client. - -:::: - - -If a feature you depend on is missing (and not explicitly documented below as a feature that we do not plan to reintroduce), open [an issue](https://github.com/elastic/elasticsearch-net/issues/new/choose) or comment on a relevant existing issue to highlight your need to us. This will help us prioritise our roadmap. - - -## Code generation [_code_generation] - -Given the size of the {{es}} API surface today, it is no longer practical to maintain thousands of types (requests, responses, queries, aggregations, etc.) by hand. To ensure consistent, accurate, and timely alignment between language clients and {{es}}, the 8.x clients, and many of the associated types are now automatically code-generated from a [shared specification](https://github.com/elastic/elasticsearch-specification). This is a common solution to maintaining alignment between client and server among SDKs and libraries, such as those for Azure, AWS and the Google Cloud Platform. - -Code-generation from a specification has inevitably led to some differences between the existing v7 `NEST` types and those available in the new v7 Elasticsearch .NET Client. For version 8, we generate strictly from the specification, special casing a few areas to improve usability or to align with language idioms. - -The base type hierarchy for concepts such as `Properties`, `Aggregations` and `Queries` is no longer present in generated code, as these arbitrary groupings do not align with concrete concepts of the public server API. These considerations do not preclude adding syntactic sugar and usability enhancements to types in future releases on a case-by-case basis. - - -## Elastic.Transport [_elastic_transport] - -The .NET client includes a transport layer responsible for abstracting HTTP concepts and to provide functionality such as our request pipeline. This supports round-robin load-balancing of requests to nodes, pinging failed nodes and sniffing the cluster for node roles. - -In v7, this layer shipped as `Elasticsearch.Net` and was considered our low-level client which could be used to send and receive raw JSON bytes between the client and server. - -As part of the work for 8.0.0, we have moved the transport layer out into a [new dedicated package](https://www.nuget.org/packages/Elastic.Transport) and [repository](https://github.com/elastic/elastic-transport-net), named `Elastic.Transport`. This supports reuse across future clients and allows consumers with extremely high-performance requirements to build upon this foundation. - - -## System.Text.Json for serialization [_system_text_json_for_serialization] - -The v7 `NEST` high-level client used an internalized and modified version of [Utf8Json](https://github.com/neuecc/Utf8Json) for request and response serialization. This was introduced for its performance improvements over [Json.NET](https://www.newtonsoft.com/json), the more common JSON framework at the time. - -While Utf8Json provides good value, we have identified minor bugs and performance issues that have required maintenance over time. Some of these are hard to change without more significant effort. This library is no longer maintained, and any such changes cannot easily be contributed back to the original project. - -With .NET Core 3.0, Microsoft shipped new [System.Text.Json APIs](https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis) that are included in-the-box with current versions of .NET. We have adopted `System.Text.Json` for all serialization. Consumers can still define and register their own `Serializer` implementation for their document types should they prefer to use a different serialization library. - -By adopting `System.Text.Json`, we now depend on a well-maintained and supported library from Microsoft. `System.Text.Json` is designed from the ground up to support the latest performance optimizations in .NET and, as a result, provides both fast and low-allocation serialization. - - -## Mockability of ElasticsearchClient [_mockability_of_elasticsearchclient] - -Testing code is an important part of software development. We recommend that consumers prefer introducing an abstraction for their use of the Elasticsearch .NET Client as the prefered way to decouple consuming code from client types and support unit testing. - -To support user testing scenarios, we have unsealed the `ElasticsearchClient` type and made its methods virtual. This supports mocking the type directly for unit testing. This is an improvement over the original `IElasticClient` interface from `NEST` (v7) which only supported mocking of top-level client methods. - -We have also introduced a `TestableResponseFactory` in `Elastic.Transport` to make it easier to create response instances with specific status codes and validity that can be used during unit testing. - -These changes are in addition to our existing support for testing with an `InMemoryConnection`, virtualized clusters and with our [`Elastic.Elasticsearch.Managed`](https://github.com/elastic/elasticsearch-net-abstractions/blob/master/src/Elastic.Elasticsearch.Managed) library for integration testing against real {{es}} instances. - - -## Migrating to Elastic.Clients.Elasticsearch [_migrating_to_elastic_clients_elasticsearch] - -::::{warning} -The version 8 client does not currently have full-feature parity with `NEST`. The client primary use case is for application developers communicating with {{es}}. - -:::: - - -The version 8 client focuses on core endpoints, more specifically for common CRUD scenarios. The intention is to reduce the feature gap in subsequent versions. Review this documentation carefully to learn about the missing features and reduced API surface details before migrating from the v7 `NEST` client! - -The choice to code-generate a new evolution of the Elasticsearch .NET Client introduces some significant breaking changes. - -The v8 client is shipped as a new [NuGet package](https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/) which can be installed alongside v7 `NEST`. Some consumers may prefer a phased migration with both packages side-by-side for a short period of time to manage complex migrations. In addition, `NEST` 7.17.x can continue to be used in [compatibility mode](https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html) with {{es}} 8.x servers until the v8 Elasticsearch .NET Client features align with application requirements. - - -## Breaking Changes [_breaking_changes] - -::::{warning} -As a result of code-generating a majority of the client types, version 8 of the client includes multiple breaking changes. - -:::: - - -We have strived to keep the core foundation reasonably similar, but types emitted through code-generation are subject to change between `NEST` (v7) and the new `Elastic.Clients.Elasticsearch` (v8) package. - - -### Namespaces [_namespaces] - -The package and top-level namespace for the v8 client have been renamed to `Elastic.Clients.Elasticsearch`. All types belong to this namespace. When necessary, to avoid potential conflicts, types are generated into suitable sub-namespaces based on the [{{es}} specification](https://github.com/elastic/elasticsearch-specification). Additional `using` directives may be required to access such types when using the Elasticsearch .NET Client. - -Transport layer concepts have moved to the new `Elastic.Transport` NuGet package and related types are defined under its namespace. Some configuration and low-level transport functionality may require a `using` directive for the `Elastic.Transport` namespace. - - -### Type names [_type_names] - -Type names may have changed from previous versions. These are not listed explicitly due to the potentially vast number of subtle differences. Type names will now more closely align to those used in the JSON and as documented in the {{es}} documentation. - - -### Class members [_class_members] - -Types may include renamed properties based on the {{es}} specification, which differ from the original `NEST` property names. The types used for properties may also have changed due to code-generation. If you identify missing or incorrectly-typed properties, please open [an issue](https://github.com/elastic/elasticsearch-net/issues/new/choose) to alert us. - - -### Sealing classes [_sealing_classes] - -Opinions on "sealing by default" within the .NET ecosystem tend to be quite polarized. Microsoft seal all internal types for potential performance gains and we see a benefit in starting with that approach for the Elasticsearch .NET Client, even for our public API surface. - -While it prevents inheritance and, therefore, may inhibit a few consumer scenarios, sealing by default is intended to avoid the unexpected or invalid extension of types that could inadvertently be broken in the future. - - -### Removed features [_removed_features] - -As part of the clean-slate redesign of the new client, certain features are removed from the v8.0 client. These are listed below: - - -#### Attribute mappings [_attribute_mappings] - -In previous versions of the `NEST` client, attributes could be used to configure the mapping behaviour and inference for user types. It is recommended that mapping be completed via the fluent API when configuring client instances. `System.Text.Json` attributes may be used to rename and ignore properties during source serialization. - - -#### CAT APIs [_cat_apis] - -The [CAT APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-cat) of {{es}} are intended for human-readable usage and will no longer be supported via the v8 Elasticsearch .NET Client. - - -#### Interface removal [_interface_removal] - -Several interfaces are removed to simplify the library and avoid interfaces where only a single implementation of that interface is expected to exist, such as `IElasticClient` in `NEST`. Abstract base classes are preferred over interfaces across the library, as this makes it easier to add enhancements without introducing breaking changes for derived types. - - -### Missing features [_missing_features] - -The following are some of the main features which have not been re-implemented for the v8 client. These might be reviewed and prioritized for inclusion in future releases. - -* Query DSL operators for combining queries. -* Scroll Helper. -* Fluent API for union types. -* `AutoMap` for field datatype inference. -* Visitor pattern support for types such as `Properties`. -* Support for `JoinField` which affects `ChildrenAggregation`. -* Conditionless queries. -* DiagnosticSources have been removed in `Elastic.Transport` to provide a clean-slate for an improved diagnostics story. The Elasticsearch .NET Client emits [OpenTelemetry](https://opentelemetry.io/) compatible `Activity` spans which can be consumed by APM agents such as the [Elastic APM Agent for .NET](apm-agent-dotnet://reference/index.md). -* Documentation is a work in progress, and we will expand on the documented scenarios in future releases. - - -## Reduced API surface [_reduced_api_surface] - -In the current versions of the code-generated .NET client, supporting commonly used endpoints is critical. Some specific queries and aggregations need further work to generate code correctly, hence they are not included yet. Ensure that the features you are using are currently supported before migrating. - -An up to date list of all supported and unsupported endpoints can be found on [GitHub](https://github.com/elastic/elasticsearch-net/issues/7890). - - -## Workarounds for missing features [_workarounds_for_missing_features] - -If you encounter a missing feature with the v8 client, there are several ways to temporarily work around this issue until we officially reintroduce the feature. - -`NEST` 7.17.x can continue to be used in [compatibility mode](https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html) with {{es}} 8.x servers until the v8 Elasticsearch .NET Client features align with application requirements. - -As a last resort, the low-level client `Elastic.Transport` can be used to create any desired request by hand: - -```csharp -public class MyRequestParameters : RequestParameters -{ - public bool Pretty - { - get => Q("pretty"); - init => Q("pretty", value); - } -} - -// ... - -var body = """ - { - "name": "my-api-key", - "expiration": "1d", - "...": "..." - } - """; - -MyRequestParameters requestParameters = new() -{ - Pretty = true -}; - -var pathAndQuery = requestParameters.CreatePathWithQueryStrings("/_security/api_key", - client.ElasticsearchClientSettings); -var endpointPath = new EndpointPath(Elastic.Transport.HttpMethod.POST, pathAndQuery); - -// Or, if the path does not contain query parameters: -// new EndpointPath(Elastic.Transport.HttpMethod.POST, "my_path") - -var response = await client.Transport - .RequestAsync( - endpointPath, - PostData.String(body), - null, - null, - cancellationToken: default) - .ConfigureAwait(false); -``` diff --git a/docs/reference/query.md b/docs/reference/query.md deleted file mode 100644 index 11f0e1d1452..00000000000 --- a/docs/reference/query.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/query.html ---- - -# Query examples [query] - -This page demonstrates how to perform a search request. - - -## Fluent API [_fluent_api_3] - -```csharp -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .Term(term => term - .Field(x => x.FirstName) - .Value("Florian") - ) - ) - .Size(10) - ); -``` - - -## Object initializer API [_object_initializer_api_3] - -```csharp -var response = await client - .SearchAsync(new SearchRequest("persons") - { - Query = Query.Term(new TermQuery(Infer.Field(x => x.FirstName)) - { - Value = "Florian" - }), - Size = 10 - }); -``` - - -## Consume the response [_consume_the_response_3] - -```csharp -foreach (var person in response.Documents) -{ - Console.WriteLine(person.FirstName); -} -``` - diff --git a/docs/reference/recommendations.md b/docs/reference/recommendations.md deleted file mode 100644 index 91a26544621..00000000000 --- a/docs/reference/recommendations.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/recommendations.html ---- - -# Usage recommendations [recommendations] - -To achieve the most efficient use of the Elasticsearch .NET Client, we recommend following the guidance defined in this article. - - -## Reuse the same client instance [_reuse_the_same_client_instance] - -When working with the Elasticsearch .NET Client we recommend that consumers reuse a single instance of `ElasticsearchClient` for the entire lifetime of the application. When reusing the same instance: - -* initialization overhead is limited to the first usage. -* resources such as TCP connections can be pooled and reused to improve efficiency. -* serialization overhead is reduced, improving performance. - -The `ElasticsearchClient` type is thread-safe and can be shared and reused across multiple threads in consuming applications. Client reuse can be achieved by creating a singleton static instance or by registering the type with a singleton lifetime when using dependency injection containers. - - -## Prefer asynchronous methods [_prefer_asynchronous_methods] - -The Elasticsearch .NET Client exposes synchronous and asynchronous methods on the `ElasticsearchClient`. We recommend always preferring the asynchronous methods, which have the `Async` suffix. Using the Elasticsearch .NET Client requires sending HTTP requests to {{es}} servers. Access to {{es}} is sometimes slow or delayed, and some complex queries may take several seconds to return. If such operations are blocked by calling the synchronous methods, the thread must wait until the HTTP request is complete. In high-load scenarios, this can cause significant thread usage, potentially affecting the throughput and performance of consuming applications. By preferring the asynchronous methods, application threads can continue with other work that doesn’t depend on the web resource until the potentially blocking task completes. - diff --git a/docs/reference/serialization.md b/docs/reference/serialization.md deleted file mode 100644 index 70b9bef8866..00000000000 --- a/docs/reference/serialization.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/serialization.html ---- - -# Serialization [serialization] - -By default, the .NET client for {{es}} uses the Microsoft System.Text.Json library for serialization. The client understands how to serialize and deserialize the request and response types correctly. It also handles (de)serialization of user POCO types representing documents read or written to {{es}}. - -The client has two distinct serialization responsibilities - serialization of the types owned by the `Elastic.Clients.Elasticsearch` library and serialization of source documents, modeled in application code. The first responsibility is entirely internal; the second is configurable. - - diff --git a/docs/reference/source-serialization.md b/docs/reference/source-serialization.md deleted file mode 100644 index e6b110b039f..00000000000 --- a/docs/reference/source-serialization.md +++ /dev/null @@ -1,524 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/source-serialization.html ---- - -# Source serialization [source-serialization] - -Source serialization refers to the process of (de)serializing POCO types in consumer applications as source documents indexed and retrieved from {{es}}. A source serializer implementation handles serialization, with the default implementation using the `System.Text.Json` library. As a result, you may use `System.Text.Json` attributes and converters to control the serialization behavior. - -* [Modelling documents with types](#modeling-documents-with-types) -* [Customizing source serialization](#customizing-source-serialization) - -## Modeling documents with types [modeling-documents-with-types] - -{{es}} provides search and aggregation capabilities on the documents that it is sent and indexes. These documents are sent as JSON objects within the request body of a HTTP request. It is natural to model documents within the {{es}} .NET client using [POCOs (*Plain Old CLR Objects*)](https://en.wikipedia.org/wiki/Plain_Old_CLR_Object). - -This section provides an overview of how types and type hierarchies can be used to model documents. - -### Default behaviour [default-behaviour] - -The default behaviour is to serialize type property names as camelcase JSON object members. - -We can model documents using a regular class (POCO). - -```csharp -public class MyDocument -{ - public string StringProperty { get; set; } -} -``` - -We can then index the an instance of the document into {{es}}. - -```csharp -using System.Threading.Tasks; -using Elastic.Clients.Elasticsearch; - -var document = new MyDocument -{ - StringProperty = "value" -}; - -var indexResponse = await Client - .IndexAsync(document, "my-index-name"); -``` - -The index request is serialized, with the source serializer handling the `MyDocument` type, serializing the POCO property named `StringProperty` to the JSON object member named `stringProperty`. - -```javascript -{ - "stringProperty": "value" -} -``` - - - -## Customizing source serialization [customizing-source-serialization] - -The built-in source serializer handles most POCO document models correctly. Sometimes, you may need further control over how your types are serialized. - -::::{note} -The built-in source serializer uses the [Microsoft `System.Text.Json` library](https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview) internally. You can apply `System.Text.Json` attributes and converters to control the serialization of your document types. -:::: - - - -#### Using `System.Text.Json` attributes [system-text-json-attributes] - -`System.Text.Json` includes attributes that can be applied to types and properties to control their serialization. These can be applied to your POCO document types to perform actions such as controlling the name of a property or ignoring a property entirely. Visit the [Microsoft documentation for further examples](https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview). - -We can model a document to represent data about a person using a regular class (POCO), applying `System.Text.Json` attributes as necessary. - -```csharp -using System.Text.Json.Serialization; - -public class Person -{ - [JsonPropertyName("forename")] <1> - public string FirstName { get; set; } - - [JsonIgnore] <2> - public int Age { get; set; } -} -``` - -1. The `JsonPropertyName` attribute ensures the `FirstName` property uses the JSON name `forename` when serialized. -2. The `JsonIgnore` attribute prevents the `Age` property from appearing in the serialized JSON. - - -We can then index an instance of the document into {{es}}. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; -using System.Threading.Tasks; -using Elastic.Transport; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; - -var person = new Person { FirstName = "Steve", Age = 35 }; -var indexResponse = await Client.IndexAsync(person, "my-index-name"); -``` - -The index request is serialized, with the source serializer handling the `Person` type, serializing the POCO property named `FirstName` to the JSON object member named `forename`. The `Age` property is ignored and does not appear in the JSON. - -```javascript -{ - "forename": "Steve" -} -``` - - -#### Configuring custom `JsonSerializerOptions` [configuring-custom-jsonserializeroptions] - -The default source serializer applies a set of standard `JsonSerializerOptions` when serializing source document types. In some circumstances, you may need to override some of our defaults. This is achievable by creating an instance of `DefaultSourceSerializer` and passing an `Action`, which is applied after our defaults have been set. This mechanism allows you to apply additional settings or change the value of our defaults. - -The `DefaultSourceSerializer` includes a constructor that accepts the current `IElasticsearchClientSettings` and a `configureOptions` `Action`. - -```csharp -public DefaultSourceSerializer(IElasticsearchClientSettings settings, Action configureOptions); -``` - -Our application defines the following `Person` class, which models a document we will index to {{es}}. - -```csharp -public class Person -{ - public string FirstName { get; set; } -} -``` - -We want to serialize our source document using Pascal Casing for the JSON properties. Since the options applied in the `DefaultSouceSerializer` set the `PropertyNamingPolicy` to `JsonNamingPolicy.CamelCase`, we must override this setting. After configuring the `ElasticsearchClientSettings`, we index our document to {{es}}. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; -using System.Threading.Tasks; -using Elastic.Transport; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; - -static void ConfigureOptions(JsonSerializerOptions o) => <1> - o.PropertyNamingPolicy = null; - -var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); -var settings = new ElasticsearchClientSettings( - nodePool, - sourceSerializer: (defaultSerializer, settings) => - new DefaultSourceSerializer(settings, ConfigureOptions)); <2> -var client = new ElasticsearchClient(settings); - -var person = new Person { FirstName = "Steve" }; -var indexResponse = await client.IndexAsync(person, "my-index-name"); -``` - -1. A local function can be defined, accepting a `JsonSerializerOptions` parameter. Here, we set `PropertyNamingPolicy` to `null`. This returns to the default behavior for `System.Text.Json`, which uses Pascal Case. -2. When creating the `ElasticsearchClientSettings`, we supply a `SourceSerializerFactory` using a lambda. The factory function creates a new instance of `DefaultSourceSerializer`, passing in the `settings` and our `ConfigureOptions` local function. We have now configured the settings with a custom instance of the source serializer. - - -The `Person` instance is serialized, with the source serializer serializing the POCO property named `FirstName` using Pascal Case. - -```javascript -{ - "FirstName": "Steve" -} -``` - -As an alternative to using a local function, we could store an `Action` into a variable instead, which can be passed to the `DefaultSouceSerializer` constructor. - -```csharp -Action configureOptions = o => o.PropertyNamingPolicy = null; -``` - - -#### Registering custom `System.Text.Json` converters [registering-custom-converters] - -In certain more advanced situations, you may have types which require further customization during serialization than is possible using `System.Text.Json` property attributes. In these cases, the recommendation from Microsoft is to leverage a custom `JsonConverter`. Source document types serialized using the `DefaultSourceSerializer` can leverage the power of custom converters. - -For this example, our application has a document class that should use a legacy JSON structure to continue operating with existing indexed documents. Several options are available, but we’ll apply a custom converter in this case. - -Our class is defined, and the `JsonConverter` attribute is applied to the class type, specifying the type of a custom converter. - -```csharp -using System.Text.Json.Serialization; - -[JsonConverter(typeof(CustomerConverter))] <1> -public class Customer -{ - public string CustomerName { get; set; } - public CustomerType CustomerType { get; set; } -} - -public enum CustomerType -{ - Standard, - Enhanced -} -``` - -1. The `JsonConverter` attribute signals to `System.Text.Json` that it should use a converter of type `CustomerConverter` when serializing instances of this class. - - -When serializing this class, rather than include a string value representing the value of the `CustomerType` property, we must send a boolean property named `isStandard`. This requirement can be achieved with a custom JsonConverter implementation. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; - -public class CustomerConverter : JsonConverter -{ - public override Customer Read(ref Utf8JsonReader reader, - Type typeToConvert, JsonSerializerOptions options) - { - var customer = new Customer(); - - while (reader.Read() && reader.TokenType != JsonTokenType.EndObject) - { - if (reader.TokenType == JsonTokenType.PropertyName) - { - if (reader.ValueTextEquals("customerName")) - { - reader.Read(); - customer.CustomerName = reader.GetString(); - continue; - } - - if (reader.ValueTextEquals("isStandard")) <1> - { - reader.Read(); - var isStandard = reader.GetBoolean(); - - if (isStandard) - { - customer.CustomerType = CustomerType.Standard; - } - else - { - customer.CustomerType = CustomerType.Enhanced; - } - - continue; - } - } - } - - return customer; - } - - public override void Write(Utf8JsonWriter writer, - Customer value, JsonSerializerOptions options) - { - if (value is null) - { - writer.WriteNullValue(); - return; - } - - writer.WriteStartObject(); - - if (!string.IsNullOrEmpty(value.CustomerName)) - { - writer.WritePropertyName("customerName"); - writer.WriteStringValue(value.CustomerName); - } - - writer.WritePropertyName("isStandard"); - - if (value.CustomerType == CustomerType.Standard) <2> - { - writer.WriteBooleanValue(true); - } - else - { - writer.WriteBooleanValue(false); - } - - writer.WriteEndObject(); - } -} -``` - -1. When reading, this converter reads the `isStandard` boolean and translate this to the correct `CustomerType` enum value. -2. When writing, this converter translates the `CustomerType` enum value to an `isStandard` boolean property. - - -We can then index a customer document into {{es}}. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; -using System.Threading.Tasks; -using Elastic.Transport; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; - -var customer = new Customer -{ - CustomerName = "Customer Ltd", - CustomerType = CustomerType.Enhanced -}; -var indexResponse = await Client.IndexAsync(customer, "my-index-name"); -``` - -The `Customer` instance is serialized using the custom converter, creating the following JSON document. - -```javascript -{ - "customerName": "Customer Ltd", - "isStandard": false -} -``` - - -#### Creating a custom `SystemTextJsonSerializer` [creating-custom-system-text-json-serializer] - -The built-in `DefaultSourceSerializer` includes the registration of `JsonConverter` instances which apply during source serialization. In most cases, these provide the proper behavior for serializing source documents, including those which use `Elastic.Clients.Elasticsearch` types on their properties. - -An example of a situation where you may require more control over the converter registration order is for serializing `enum` types. The `DefaultSourceSerializer` registers the `System.Text.Json.Serialization.JsonStringEnumConverter`, so enum values are serialized using their string representation. Generally, this is the preferred option for types used to index documents to {{es}}. - -In some scenarios, you may need to control the string value sent for an enumeration value. That is not directly supported in `System.Text.Json` but can be achieved by creating a custom `JsonConverter` for the `enum` type you wish to customize. In this situation, it is not sufficient to use the `JsonConverterAttribute` on the `enum` type to register the converter. `System.Text.Json` will prefer the converters added to the `Converters` collection on the `JsonSerializerOptions` over an attribute applied to an `enum` type. It is, therefore, necessary to either remove the `JsonStringEnumConverter` from the `Converters` collection or register a specialized converter for your `enum` type before the `JsonStringEnumConverter`. - -The latter is possible via several techniques. When using the {{es}} .NET library, we can achieve this by deriving from the abstract `SystemTextJsonSerializer` class. - -Here we have a POCO which uses the `CustomerType` enum as the type for a property. - -```csharp -using System.Text.Json.Serialization; - -public class Customer -{ - public string CustomerName { get; set; } - public CustomerType CustomerType { get; set; } -} - -public enum CustomerType -{ - Standard, - Enhanced -} -``` - -To customize the strings used during serialization of the `CustomerType`, we define a custom `JsonConverter` specific to our `enum` type. - -```csharp -using System.Text.Json.Serialization; - -public class CustomerTypeConverter : JsonConverter -{ - public override CustomerType Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) - { - return reader.GetString() switch <1> - { - "basic" => CustomerType.Standard, - "premium" => CustomerType.Enhanced, - _ => throw new JsonException( - $"Unknown value read when deserializing {nameof(CustomerType)}."), - }; - } - - public override void Write(Utf8JsonWriter writer, CustomerType value, JsonSerializerOptions options) - { - switch (value) <2> - { - case CustomerType.Standard: - writer.WriteStringValue("basic"); - return; - case CustomerType.Enhanced: - writer.WriteStringValue("premium"); - return; - } - - writer.WriteNullValue(); - } -} -``` - -1. When reading, this converter translates the string used in the JSON to the matching enum value. -2. When writing, this converter translates the `CustomerType` enum value to a custom string value written to the JSON. - - -We create a serializer derived from `SystemTextJsonSerializer` to give us complete control of converter registration order. - -```csharp -using System.Text.Json; -using Elastic.Clients.Elasticsearch.Serialization; - -public class MyCustomSerializer : SystemTextJsonSerializer <1> -{ - private readonly JsonSerializerOptions _options; - - public MyCustomSerializer(IElasticsearchClientSettings settings) : base(settings) - { - var options = DefaultSourceSerializer.CreateDefaultJsonSerializerOptions(false); <2> - - options.Converters.Add(new CustomerTypeConverter()); <3> - - _options = DefaultSourceSerializer.AddDefaultConverters(options); <4> - } - - protected override JsonSerializerOptions CreateJsonSerializerOptions() => _options; <5> -} -``` - -1. Inherit from `SystemTextJsonSerializer`. -2. In the constructor, use the factory method `DefaultSourceSerializer.CreateDefaultJsonSerializerOptions` to create default options for serialization. No default converters are registered at this stage because we pass `false` as an argument. -3. Register our `CustomerTypeConverter` as the first converter. -4. To apply any default converters, call the `DefaultSourceSerializer.AddDefaultConverters` helper method, passing the options to modify. -5. Implement the `CreateJsonSerializerOptions` method returning the stored `JsonSerializerOptions`. - - -Because we have registered our `CustomerTypeConverter` before the default converters (which include the `JsonStringEnumConverter`), our converter takes precedence when serializing `CustomerType` instances on source documents. - -The base `SystemTextJsonSerializer` class handles the implementation details of binding, which is required to ensure that the built-in converters can access the `IElasticsearchClientSettings` where needed. - -We can then index a customer document into {{es}}. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; -using System.Threading.Tasks; -using Elastic.Transport; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; - -var customer = new Customer -{ - CustomerName = "Customer Ltd", - CustomerType = CustomerType.Enhanced -}; - -var indexResponse = await client.IndexAsync(customer, "my-index-name"); -``` - -The `Customer` instance is serialized using the custom `enum` converter, creating the following JSON document. - -```javascript -{ - "customerName": "Customer Ltd", - "customerType": "premium" <1> -} -``` - -1. The string value applied during serialization is provided by our custom converter. - - - -#### Creating a custom `Serializer` [creating-custom-serializers] - -Suppose you prefer using an alternative JSON serialization library for your source types. In that case, you can inject an isolated serializer only to be called for the serialization of `_source`, `_fields`, or wherever a user-provided value is expected to be written and returned. - -Implementing `Elastic.Transport.Serializer` is technically enough to create a custom source serializer. - -```csharp -using System; -using System.IO; -using System.Threading; -using System.Threading.Tasks; -using Elastic.Transport; - -public class VanillaSerializer : Serializer -{ - public override object Deserialize(Type type, Stream stream) => - throw new NotImplementedException(); - - public override T Deserialize(Stream stream) => - throw new NotImplementedException(); - - public override ValueTask DeserializeAsync(Type type, Stream stream, CancellationToken cancellationToken = default) => - throw new NotImplementedException(); - - public override ValueTask DeserializeAsync(Stream stream, CancellationToken cancellationToken = default) => - throw new NotImplementedException(); - - public override void Serialize(T data, Stream stream, SerializationFormatting formatting = SerializationFormatting.None) => - throw new NotImplementedException(); - - public override Task SerializeAsync(T data, Stream stream, - SerializationFormatting formatting = SerializationFormatting.None, CancellationToken cancellationToken = default) => - throw new NotImplementedException(); -} -``` - -Registering up the serializer is performed in the `ConnectionSettings` constructor. - -```csharp -using System; -using System.Text.Json; -using System.Text.Json.Serialization; -using System.Threading.Tasks; -using Elastic.Transport; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; - -var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); -var settings = new ElasticsearchClientSettings( - nodePool, - sourceSerializer: (defaultSerializer, settings) => - new VanillaSerializer()); <1> -var client = new ElasticsearchClient(settings); -``` - -1. If implementing `Serializer` is enough, why must we provide an instance wrapped in a factory `Func`? - - -There are various cases where you might have a POCO type that contains an `Elastic.Clients.Elasticsearch` type as one of its properties. The `SourceSerializerFactory` delegate provides access to the default built-in serializer so you can access it when necessary. For example, consider if you want to use percolation; you need to store {{es}} queries as part of the `_source` of your document, which means you need to have a POCO that looks like this. - -```csharp -using Elastic.Clients.Elasticsearch.QueryDsl; - -public class MyPercolationDocument -{ - public Query Query { get; set; } - public string Category { get; set; } -} -``` - -A custom serializer would not know how to serialize `Query` or other `Elastic.Clients.Elasticsearch` types that could appear as part of the `_source` of a document. Therefore, your custom `Serializer` would need to store a reference to our built-in serializer and delegate serialization of Elastic types back to it. - - diff --git a/docs/reference/toc.yml b/docs/reference/toc.yml deleted file mode 100644 index 828400800c0..00000000000 --- a/docs/reference/toc.yml +++ /dev/null @@ -1,34 +0,0 @@ -toc: - - file: index.md - - file: getting-started.md - - file: installation.md - - file: connecting.md - - file: configuration.md - children: - - file: _options_on_elasticsearchclientsettings.md - - file: client-concepts.md - children: - - file: serialization.md - children: - - file: source-serialization.md - - file: using-net-client.md - children: - - file: aggregations.md - - file: esql.md - - file: examples.md - - file: mappings.md - - file: query.md - - file: recommendations.md - - file: transport.md - - file: migration-guide.md - - file: troubleshoot/index.md - children: - - file: troubleshoot/logging.md - children: - - file: troubleshoot/logging-with-onrequestcompleted.md - - file: troubleshoot/logging-with-fiddler.md - - file: troubleshoot/debugging.md - children: - - file: troubleshoot/audit-trail.md - - file: troubleshoot/debug-information.md - - file: troubleshoot/debug-mode.md \ No newline at end of file diff --git a/docs/reference/troubleshoot/debug-information.md b/docs/reference/troubleshoot/debug-information.md deleted file mode 100644 index f682179a044..00000000000 --- a/docs/reference/troubleshoot/debug-information.md +++ /dev/null @@ -1,165 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debug-information.html ---- - -# Debug information [debug-information] - -Every response from Elasticsearch.Net and NEST contains a `DebugInformation` property that provides a human readable description of what happened during the request for both successful and failed requests - -```csharp -var response = client.Search(s => s - .Query(q => q - .MatchAll() - ) -); - -response.DebugInformation.Should().Contain("Valid NEST response"); -``` - -This can be useful in tracking down numerous problems and can also be useful when filing an [issue](https://github.com/elastic/elasticsearch-net/issues) on the GitHub repository. - -## Request and response bytes [_request_and_response_bytes] - -By default, the request and response bytes are not available within the debug information, but can be enabled globally on Connection Settings by setting `DisableDirectStreaming`. This disables direct streaming of - -1. the serialized request type to the request stream -2. the response stream to a deserialized response type - -```csharp -var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); - -var settings = new ConnectionSettings(connectionPool) - .DisableDirectStreaming(); <1> - -var client = new ElasticClient(settings); -``` - -1. disable direct streaming for **all** requests - - -or on a *per request* basis - -```csharp -var response = client.Search(s => s - .RequestConfiguration(r => r - .DisableDirectStreaming() <1> - ) - .Query(q => q - .MatchAll() - ) -); -``` - -1. disable direct streaming for **this** request only - - -Configuring `DisableDirectStreaming` on an individual request takes precedence over any global configuration. - -There is typically a performance and allocation cost associated with disabling direct streaming since both the request and response bytes must be buffered in memory, to allow them to be exposed on the response call details. - - -## TCP statistics [_tcp_statistics] - -It can often be useful to see the statistics for active TCP connections, particularly when trying to diagnose issues with the client. The client can collect the states of active TCP connections just before making a request, and expose these on the response and in the debug information. - -Similarly to `DisableDirectStreaming`, TCP statistics can be collected for every request by configuring on `ConnectionSettings` - -```csharp -var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); - -var settings = new ConnectionSettings(connectionPool) - .EnableTcpStats(); <1> - -var client = new ElasticClient(settings); -``` - -1. collect TCP statistics for **all** requests - - -or on a *per request* basis - -```csharp -var response = client.Search(s => s - .RequestConfiguration(r => r - .EnableTcpStats() <1> - ) - .Query(q => q - .MatchAll() - ) -); - -var debugInformation = response.DebugInformation; -``` - -1. collect TCP statistics for **this** request only - - -With `EnableTcpStats` set, the states of active TCP connections will now be included on the response and in the debug information. - -The client includes a `TcpStats` class to help with retrieving more detail about active TCP connections should it be required - -```csharp -var tcpStatistics = TcpStats.GetActiveTcpConnections(); <1> -var ipv4Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv4); <2> -var ipv6Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv6); <3> - -var response = client.Search(s => s - .Query(q => q - .MatchAll() - ) -); -``` - -1. Retrieve details about active TCP connections, including local and remote addresses and ports -2. Retrieve statistics about IPv4 -3. Retrieve statistics about IPv6 - - -::::{note} -Collecting TCP statistics may not be accessible in all environments, for example, Azure App Services. When this is the case, `TcpStats.GetActiveTcpConnections()` returns `null`. - -:::: - - - -## ThreadPool statistics [_threadpool_statistics] - -It can often be useful to see the statistics for thread pool threads, particularly when trying to diagnose issues with the client. The client can collect statistics for both worker threads and asynchronous I/O threads, and expose these on the response and in debug information. - -Similar to collecting TCP statistics, ThreadPool statistics can be collected for all requests by configuring `EnableThreadPoolStats` on `ConnectionSettings` - -```csharp -var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); - -var settings = new ConnectionSettings(connectionPool) - .EnableThreadPoolStats(); <1> - -var client = new ElasticClient(settings); -``` - -1. collect thread pool statistics for **all** requests - - -or on a *per request* basis - -```csharp -var response = client.Search(s => s - .RequestConfiguration(r => r - .EnableThreadPoolStats() <1> - ) - .Query(q => q - .MatchAll() - ) - ); - -var debugInformation = response.DebugInformation; <2> -``` - -1. collect thread pool statistics for **this** request only -2. contains thread pool statistics - - -With `EnableThreadPoolStats` set, the statistics of thread pool threads will now be included on the response and in the debug information. - - diff --git a/docs/reference/troubleshoot/debug-mode.md b/docs/reference/troubleshoot/debug-mode.md deleted file mode 100644 index fce96b34be4..00000000000 --- a/docs/reference/troubleshoot/debug-mode.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debug-mode.html ---- - -# Debug mode [debug-mode] - -The [Debug information](debug-information.md) explains that every response from Elasticsearch.Net and NEST contains a `DebugInformation` property, and properties on `ConnectionSettings` and `RequestConfiguration` can control which additional information is included in debug information, for all requests or on a per request basis, respectively. - -During development, it can be useful to enable the most verbose debug information, to help identify and troubleshoot problems, or simply ensure that the client is behaving as expected. The `EnableDebugMode` setting on `ConnectionSettings` is a convenient shorthand for enabling verbose debug information, configuring a number of settings like - -* disabling direct streaming to capture request and response bytes -* prettyfying JSON responses from Elasticsearch -* collecting TCP statistics when a request is made -* collecting thread pool statistics when a request is made -* including the Elasticsearch stack trace in the response if there is a an error on the server side - -```csharp -IConnectionPool pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); - -var settings = new ConnectionSettings(pool) - .EnableDebugMode(); <1> - -var client = new ElasticClient(settings); - -var response = client.Search(s => s - .Query(q => q - .MatchAll() - ) -); - -var debugInformation = response.DebugInformation; <2> -``` - -1. configure debug mode -2. verbose debug information - - -In addition to exposing debug information on the response, debug mode will also cause the debug information to be written to the trace listeners in the `System.Diagnostics.Debug.Listeners` collection by default, when the request has completed. A delegate can be passed when enabling debug mode to perform a different action when a request has completed, using [`OnRequestCompleted`](logging-with-onrequestcompleted.md) - -```csharp -var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); -var client = new ElasticClient(new ConnectionSettings(pool) - .EnableDebugMode(apiCallDetails => - { - // do something with the call details e.g. send with logging framework - }) -); -``` - diff --git a/docs/reference/troubleshoot/debugging.md b/docs/reference/troubleshoot/debugging.md deleted file mode 100644 index 2514e064b48..00000000000 --- a/docs/reference/troubleshoot/debugging.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debugging.html ---- - -# Debugging [debugging] - -Elasticsearch.Net and NEST provide an audit trail and debug information to help you resolve issues: - -* [](audit-trail.md) -* [](debug-information.md) -* [](debug-mode.md) - - diff --git a/docs/reference/troubleshoot/index.md b/docs/reference/troubleshoot/index.md deleted file mode 100644 index b1163f89ecc..00000000000 --- a/docs/reference/troubleshoot/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -navigation_title: Troubleshoot -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/troubleshooting.html ---- - -# Troubleshoot: {{es}} .NET client [troubleshooting] - -The client can provide rich details about what occurred in the request pipeline during the process of making a request, and can also provide the raw request and response JSON. - -* [Logging](logging.md) -* [Debugging](debugging.md) - diff --git a/docs/reference/troubleshoot/logging-with-fiddler.md b/docs/reference/troubleshoot/logging-with-fiddler.md deleted file mode 100644 index 04e8c590501..00000000000 --- a/docs/reference/troubleshoot/logging-with-fiddler.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging-with-fiddler.html ---- - -# Logging with Fiddler [logging-with-fiddler] - -A web debugging proxy such as [Fiddler](http://www.telerik.com/fiddler) is a useful way to capture HTTP traffic from a machine, particularly whilst developing against a local Elasticsearch cluster. - -## Capturing traffic to a remote cluster [_capturing_traffic_to_a_remote_cluster] - -To capture traffic against a remote cluster is as simple as launching Fiddler! You may want to also filter traffic to only show requests to the remote cluster by using the filters tab - -![Capturing requests to a remote host](../images/elasticsearch-client-net-api-capture-requests-remotehost.png) - - -## Capturing traffic to a local cluster [_capturing_traffic_to_a_local_cluster] - -The .NET Framework is hardcoded not to send requests for `localhost` through any proxies and as a proxy Fiddler will not receive such traffic. - -This is easily circumvented by using `ipv4.fiddler` as the hostname instead of `localhost` - -```csharp -var isFiddlerRunning = Process.GetProcessesByName("fiddler").Any(); -var host = isFiddlerRunning ? "ipv4.fiddler" : "localhost"; - -var connectionSettings = new ConnectionSettings(new Uri($"http://{host}:9200")) - .PrettyJson(); <1> - -var client = new ElasticClient(connectionSettings); -``` - -1. prettify json requests and responses to make them easier to read in Fiddler - - -With Fiddler running, the requests and responses will now be captured and can be inspected in the Inspectors tab - -![Inspecting requests and responses](../images/elasticsearch-client-net-api-inspect-requests.png) - -As before, you may also want to filter traffic to only show requests to `ipv4.fiddler` on the port on which you are running Elasticsearch. - -![Capturing requests to localhost](../images/elasticsearch-client-net-api-capture-requests-localhost.png) - - diff --git a/docs/reference/troubleshoot/logging.md b/docs/reference/troubleshoot/logging.md deleted file mode 100644 index 8efdbc997d7..00000000000 --- a/docs/reference/troubleshoot/logging.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging.html ---- - -# Logging [logging] - -While developing with Elasticsearch using NEST, it can be extremely valuable to see the requests that NEST generates and sends to Elasticsearch, as well as the responses returned. - -* [](logging-with-onrequestcompleted.md) -* [](logging-with-fiddler.md) - - - diff --git a/docs/reference/using-net-client.md b/docs/reference/using-net-client.md deleted file mode 100644 index ace92b95d1a..00000000000 --- a/docs/reference/using-net-client.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/usage.html ---- - -# Using the .NET Client [usage] - -The sections below provide tutorials on the most frequently used and some less obvious features of {{es}}. - -For a full reference, see the [Elasticsearch documentation](docs-content://get-started/index.md) and in particular the [REST APIs](elasticsearch://reference/elasticsearch/rest-apis/index.md) section. The Elasticsearch .NET Client follows closely the JSON structures described there. - -A .NET API reference documentation for the Elasticsearch client package is available [here](https://elastic.github.io/elasticsearch-net). - -If you’re new to {{es}}, make sure also to read [Elasticsearch’s quick start](docs-content://solutions/search/get-started.md) that provides a good introduction. - -* [Usage recommendations](/reference/recommendations.md) -* [CRUD usage examples](/reference/examples.md) -* [Using ES|QL](/reference/esql.md) - -::::{note} -This is still a work in progress, more sections will be added in the near future. -:::: - - diff --git a/docs/release-notes/breaking-change-policy.asciidoc b/docs/release-notes/breaking-change-policy.asciidoc new file mode 100644 index 00000000000..74138bc005f --- /dev/null +++ b/docs/release-notes/breaking-change-policy.asciidoc @@ -0,0 +1,32 @@ +[[breaking-changes-policy]] +== Breaking changes policy + +The {net-client} source code is generated from a https://github.com/elastic/elasticsearch-specification[formal specification of the Elasticsearch API]. This API specification is large, and although it is tested against hundreds of Elasticsearch test files, it may have discrepancies with the actual API that result in issues in the {net-client}. + +Fixing these discrepancies in the API specification results in code changes in the {net-client}, and some of these changes can require code updates in your applications. + +This section explains how these breaking changes are considered for inclusion in {net-client} releases. + +[discrete] +==== Breaking changes in patch releases + +Some issues in the API specification are properties that have an incorrect type, such as a `long` that should be a `string`, or a required property that is actually optional. These issues can cause the {net-client} to not work properly or even throw exceptions. + +When a specification issue is discovered and resolved, it may require code updates in applications using the {net-client}. Such breaking changes are considered acceptable, _even in patch releases_ (e.g. 8.0.0 -> 8.0.1), as they introduce stability to APIs that may otherwise be unusable. + +We may also make breaking changes in patch releases to correct design flaws and code-generation issues that we deem beneficial to resolve at the earliest oppotunity. We will detail these in the relevant release notes and limit these as the client matures. + +[discrete] +==== Breaking changes in minor releases + +Along with these bug fixes, the API specification is constantly refined, more precise type definitions are introduced to improve developer comfort and remove ambiguities. The specification of often-used APIs is fairly mature, so these changes happen generally on less often used APIs. These changes can also cause breaking changes requiring code updates which are considered _acceptable in minor releases_ (e.g. 8.0 -> 8.1). + +[discrete] +==== Breaking changes in major releases + +Major releases (e.g. 7.x -> 8.x) can include larger refactorings of the API specification and the framework underlying the {net-client}. These refactorings are considered carefully and done only when they unlock new important features or new developments. + +[discrete] +==== Elasticsearch API stability guarantees + +All Elasticsearch APIs have stability indicators, which imply potential changes. If an API is `stable` only additional non-breaking changes are added. In case of `experimental` APIs, breaking changes can be introduced any time, which means that these changes, will also be reflected in the {net-client}. \ No newline at end of file diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md deleted file mode 100644 index 3ff9b442359..00000000000 --- a/docs/release-notes/breaking-changes.md +++ /dev/null @@ -1,235 +0,0 @@ ---- -navigation_title: "Breaking changes" -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/breaking-changes-policy.html ---- - -# Elasticsearch .NET Client breaking changes [elasticsearch-net-client-breaking-changes] - -Breaking changes can impact your Elastic applications, potentially disrupting normal operations. Before you upgrade, carefully review the Elasticsearch .NET Client breaking changes and take the necessary steps to mitigate any issues. To learn how to upgrade, check [Upgrade](docs-content://deploy-manage/upgrade.md). - -% ## Next version [elasticsearch-net-client-nextversion-breaking-changes] - -% ::::{dropdown} Title of breaking change -% -% **Impact**: High/Low. -% -% Description of the breaking change. -% For more information, check [PR #](PR link). -% -% :::: - -## 9.0.0 [elasticsearch-net-client-900-breaking-changes] - -### Overview - -- [1. Container types](#1-container-types) -- [2. Removal of certain generic request descriptors](#2-removal-of-certain-generic-request-descriptors) -- [3. Removal of certain descriptor constructors and related request APIs](#3-removal-of-certain-descriptor-constructors-and-related-request-apis) -- [4. Date / Time / Duration values](#4-date-time-duration-values) -- [5. `ExtendedBounds`](#5-extendedbounds) -- [6. `Field.Format`](#6-fieldformat) -- [7. `Field`/`Fields` semantics](#7-fieldfields-semantics) -- [8. `FieldValue`](#8-fieldvalue) -- [9. `FieldSort`](#9-fieldsort) -- [10. Descriptor types `class` -\> `struct`](#10-descriptor-types-class-struct) - -### Breaking changes - -#### 1. Container types [1-container-types] - -**Impact**: High. - -Container types now use regular properties for their variants instead of static factory methods ([read more](index.md#7-improved-container-design)). - -This change primarily affects the `Query` and `Aggregation` types. - -```csharp -// 8.x -new SearchRequest -{ - Query = Query.MatchAll( - new MatchAllQuery - { - } - ) -}; - -// 9.0 -new SearchRequest -{ - Query = new Query - { - MatchAll = new MatchAllQuery - { - } - } -}; -``` - -Previously required methods like e.g. `TryGet(out)` have been removed. - -The new recommended way of inspecting container types is to use simple pattern matching: - -```csharp -var query = new Query(); - -if (query.Nested is { } nested) -{ - // We have a nested query. -} -``` - -#### 2. Removal of certain generic request descriptors [2-removal-of-certain-generic-request-descriptors] - -**Impact**: High. - -Removed the generic version of some request descriptors for which the corresponding requests do not contain inferrable properties. - -These descriptors were generated unintentionally. - -When migrating, the generic type parameter must be removed from the type, e.g., `AsyncSearchStatusRequestDescriptor` should become just `AsyncSearchStatusRequestDescriptor`. - -List of affected descriptors: - -- `AsyncQueryDeleteRequestDescriptor` -- `AsyncQueryGetRequestDescriptor` -- `AsyncSearchStatusRequestDescriptor` -- `DatabaseConfigurationDescriptor` -- `DatabaseConfigurationFullDescriptor` -- `DeleteAsyncRequestDescriptor` -- `DeleteAsyncSearchRequestDescriptor` -- `DeleteDataFrameAnalyticsRequestDescriptor` -- `DeleteGeoipDatabaseRequestDescriptor` -- `DeleteIpLocationDatabaseRequestDescriptor` -- `DeleteJobRequestDescriptor` -- `DeletePipelineRequestDescriptor` -- `DeleteScriptRequestDescriptor` -- `DeleteSynonymRequestDescriptor` -- `EqlDeleteRequestDescriptor` -- `EqlGetRequestDescriptor` -- `GetAsyncRequestDescriptor` -- `GetAsyncSearchRequestDescriptor` -- `GetAsyncStatusRequestDescriptor` -- `GetDataFrameAnalyticsRequestDescriptor` -- `GetDataFrameAnalyticsStatsRequestDescriptor` -- `GetEqlStatusRequestDescriptor` -- `GetGeoipDatabaseRequestDescriptor` -- `GetIpLocationDatabaseRequestDescriptor` -- `GetJobsRequestDescriptor` -- `GetPipelineRequestDescriptor` -- `GetRollupCapsRequestDescriptor` -- `GetRollupIndexCapsRequestDescriptor` -- `GetScriptRequestDescriptor` -- `GetSynonymRequestDescriptor` -- `IndexModifyDataStreamActionDescriptor` -- `PreprocessorDescriptor` -- `PutGeoipDatabaseRequestDescriptor` -- `PutIpLocationDatabaseRequestDescriptor` -- `PutScriptRequestDescriptor` -- `PutSynonymRequestDescriptor` -- `QueryVectorBuilderDescriptor` -- `RankDescriptor` -- `RenderSearchTemplateRequestDescriptor` -- `SmoothingModelDescriptor` -- `StartDataFrameAnalyticsRequestDescriptor` -- `StartJobRequestDescriptor` -- `StopDataFrameAnalyticsRequestDescriptor` -- `StopJobRequestDescriptor` -- `TokenizationConfigDescriptor` -- `UpdateDataFrameAnalyticsRequestDescriptor` - -#### 3. Removal of certain descriptor constructors and related request APIs [3-removal-of-certain-descriptor-constructors-and-related-request-apis] - -**Impact**: High. - -Removed `(TDocument, IndexName)` descriptor constructors and related request APIs for all requests with `IndexName` and `Id` path parameters. - -For example: - -```csharp -// 8.x -public IndexRequestDescriptor(TDocument document, IndexName index, Id? id) { } -public IndexRequestDescriptor(TDocument document, IndexName index) { } -public IndexRequestDescriptor(TDocument document, Id? id) { } -public IndexRequestDescriptor(TDocument document) { } - -// 9.0 -public IndexRequestDescriptor(TDocument document, IndexName index, Id? id) { } -public IndexRequestDescriptor(TDocument document, Id? id) { } -public IndexRequestDescriptor(TDocument document) { } -``` - -These overloads caused invocation ambiguities since both, `IndexName` and `Id` implement implicit conversion operators from `string`. - -Alternative with same semantics: - -```csharp -// Descriptor constructor. -new IndexRequestDescriptor(document, "my_index", Id.From(document)); - -// Request API method. -await client.IndexAsync(document, "my_index", Id.From(document), ...); -``` - -#### 4. Date / Time / Duration values [4-date-time-duration-values] - -**Impact**: High. - -In places where previously `long` or `double` was used to represent a date/time/duration value, `DateTimeOffset` or `TimeSpan` is now used instead. - -#### 5. `ExtendedBounds` [5-extendedbounds] - -**Impact**: High. - -Removed `ExtendedBoundsDate`/`ExtendedBoundsDateDescriptor`, `ExtendedBoundsFloat`/`ExtendedBoundsFloatDescriptor`. - -Replaced by `ExtendedBounds`, `ExtendedBoundsOfFieldDateMathDescriptor`, and `ExtendedBoundsOfDoubleDescriptor`. - -#### 6. `Field.Format` [6-fieldformat] - -**Impact**: Low. - -Removed `Field.Format` property and corresponding constructor and inferrer overloads. - -This property has not been used for some time (replaced by the `FieldAndFormat` type). - -#### 7. `Field`/`Fields` semantics [7-fieldfields-semantics] - -**Impact**: Low. - -`Field`/`Fields` static factory methods and conversion operators no longer return nullable references but throw exceptions instead (`Field`) if the input `string`/`Expression`/`PropertyInfo` argument is `null`. - -This makes implicit conversions to `Field` more user-friendly without requiring the null-forgiveness operator (`!`) ([read more](index.md#5-field-name-inference)). - -#### 8. `FieldValue` [8-fieldvalue] - -**Impact**: Low. - -Removed `FieldValue.IsLazyDocument`, `FieldValue.IsComposite`, and the corresponding members in the `FieldValue.ValueKind` enum. - -These values have not been used for some time. - -#### 9. `FieldSort` [9-fieldsort] - -**Impact**: High. - -Removed `FieldSort` parameterless constructor. - -Please use the new constructor instead: - -```csharp -public FieldSort(Elastic.Clients.Elasticsearch.Field field) -``` - -**Impact**: Low. - -Removed static `FieldSort.Empty` member. - -Sorting got reworked which makes this member obsolete ([read more](index.md#8-sorting)). - -#### 10. Descriptor types `class` -> `struct` [10-descriptor-types-class-struct] - -**Impact**: Low. - -All descriptor types are now implemented as `struct` instead of `class`. diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md deleted file mode 100644 index 664ef25cbe8..00000000000 --- a/docs/release-notes/deprecations.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -navigation_title: "Deprecations" ---- - -# Elasticsearch .NET Client deprecations [elasticsearch-net-client-deprecations] -Over time, certain Elastic functionality becomes outdated and is replaced or removed. To help with the transition, Elastic deprecates functionality for a period before removal, giving you time to update your applications. - -Review the deprecated functionality for Elasticsearch .NET Client. While deprecations have no immediate impact, we strongly encourage you update your implementation after you upgrade. To learn how to upgrade, check out [Upgrade](docs-content://deploy-manage/upgrade.md). - -% ## Next version [elasticsearch-net-client-versionnext-deprecations] - -% ::::{dropdown} Deprecation title -% Description of the deprecation. -% For more information, check [PR #](PR link). -% **Impact**
Impact of deprecation. -% **Action**
Steps for mitigating deprecation impact. -% :::: - -## 9.0.0 [elasticsearch-net-client-900-deprecations] - -_No deprecations_ \ No newline at end of file diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md deleted file mode 100644 index 7bfcc1e88b0..00000000000 --- a/docs/release-notes/index.md +++ /dev/null @@ -1,406 +0,0 @@ ---- -navigation_title: "Elasticsearch .NET Client" -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/release-notes.html ---- - -# Elasticsearch .NET Client release notes [elasticsearch-net-client-release-notes] - -Review the changes, fixes, and more in each version of Elasticsearch .NET Client. - -To check for security updates, go to [Security announcements for the Elastic stack](https://discuss.elastic.co/c/announcements/security-announcements/31). - -% Release notes include only features, enhancements, and fixes. Add breaking changes, deprecations, and known issues to the applicable release notes sections. - -% ## version.next [felasticsearch-net-client-next-release-notes] - -% ### Features and enhancements [elasticsearch-net-client-next-features-enhancements] -% * - -% ### Fixes [elasticsearch-net-client-next-fixes] -% * - -## 9.0.0 [elasticsearch-net-client-900-release-notes] - -### Overview - -- [1. Request Method/API Changes](#1-request-methodapi-changes) - - [1.1. Synchronous Request APIs](#11-synchronous-request-apis) - - [1.2. Separate Type Arguments for Request/Response](#12-separate-type-arguments-for-requestresponse) -- [2. Improved Fluent API](#2-improved-fluent-api) - - [2.1. `ICollection`](#21-icollectione) - - [2.2. `IDictionary`](#22-idictionaryk-v) - - [2.3. `ICollection>`](#23-icollectionkeyvaluepairk-v) - - [2.4. Union Types](#24-union-types) -- [3. Improved Descriptor Design](#3-improved-descriptor-design) - - [3.1. Wrap](#31-wrap) - - [3.2. Unwrap / Inspect](#32-unwrap-inspect) - - [3.3. Removal of Side Effects](#33-removal-of-side-effects) -- [4. Request Path Parameter Properties](#4-request-path-parameter-properties) -- [5. Field Name Inference](#5-field-name-inference) -- [6. Uniform Date/Time/Duration Types](#6-uniform-datetimeduration-types) -- [7. Improved Container Design](#7-improved-container-design) -- [8. Sorting](#8-sorting) -- [9. Safer Object Creation](#9-safer-object-creation) -- [10. Serialization](#10-serialization) - -### Features and enhancements - -#### 1. Request Method/API Changes [1-request-methodapi-changes] - -##### 1.1. Synchronous Request APIs [11-synchronous-request-apis] - -Synchronous request APIs are no longer marked as `obsolete`. We received some feedback about this deprecation and decided to revert it. - -##### 1.2. Separate Type Arguments for Request/Response [12-separate-type-arguments-for-requestresponse] - -It is now possible to specify separate type arguments for requests/responses when executing request methods: - -```csharp -var response = await client.SearchAsync(x => x - .Query(x => x.Term(x => x.Field(x => x.FirstName).Value("Florian"))) -); - -var documents = response.Documents; <1> -``` - -1. `IReadOnlyCollection` - -The regular APIs with merged type arguments are still available. - -#### 2. Improved Fluent API [2-improved-fluent-api] - -The enhanced fluent API generation is likely the most notable change in the 9.0 client. - -This section describes the main syntax constructs generated based on the type of the property in the corresponding object. - -##### 2.1. `ICollection` [21-icollectione] - -Note: This syntax already existed in 8.x. - -```csharp -new SearchRequestDescriptor() - .Query(q => q - .Bool(b => b - .Must(new Query()) // Scalar: Single element. - .Must(new Query(), new Query()) // Scalar: Multiple elements (params). - .Must(m => m.MatchAll()) // Fluent: Single element. - .Must(m => m.MatchAll(), m => m.MatchNone()) // Fluent: Multiple elements (params). - ) - ); -``` - -##### 2.2. `IDictionary` [22-idictionaryk-v] - -The 9.0 client introduces full fluent API support for dictionary types. - -```csharp -new SearchRequestDescriptor() - .Aggregations(new Dictionary()) // Scalar. - .Aggregations(aggs => aggs // Fluent: Nested. - .Add("key", new MaxAggregation()) // Scalar: Key + Value. - .Add("key", x => x.Max()) // Fluent: Key + Value. - ) - .AddAggregation("key", new MaxAggregation()) // Scalar. - .AddAggregation("key", x => x.Max()); // Fluent. -``` - -:::{warning} - -The `Add{Element}` methods have different semantics compared to the standard setter methods. - -Standard fluent setters set or **replace** a value. - -In contrast, the new additive methods append new elements to the dictionary. - -::: - -For dictionaries where the value type does not contain required properties that must be initialized, another syntax is generated that allows easy addition of new entries by just specifying the key: - -```csharp -// Dictionary() - -new CreateIndexRequestDescriptor("index") - // ... all previous overloads ... - .Aliases(aliases => aliases // Fluent: Nested. - .Add("key") // Key only. - ) - .Aliases("key") // Key only: Single element. - .Aliases("first", "second") // Key only: Multiple elements (params). -``` - -If the value type in the dictionary is a collection, additional `params` overloads are generated: - -```csharp -// Dictionary> - -new CompletionSuggesterDescriptor() - // ... all previous overloads ... - .AddContext("key", - new CompletionContext{ Context = new Context("first") }, - new CompletionContext{ Context = new Context("second") } - ) - .AddContext("key", - x => x.Context(x => x.Category("first")), - x => x.Context(x => x.Category("second")) - ); -``` - -##### 2.3. `ICollection>` [23-icollectionkeyvaluepairk-v] - -Elasticsearch often uses `ICollection>` types for ordered dictionaries. - -The 9.0 client abstracts this implementation detail by providing a fluent API that can be used exactly like the one for `IDictionary` types: - -```csharp -new PutMappingRequestDescriptor("index") - .DynamicTemplates(new List>()) // Scalar. - .DynamicTemplates(x => x // Fluent: Nested. - .Add("key", new DynamicTemplate()) // Scalar: Key + Value. - .Add("key", x => x.Mapping(new TextProperty())) // Fluent: Key + Value. - ) - .AddDynamicTemplate("key", new DynamicTemplate()) // Scalar: Key + Value. - .AddDynamicTemplate("key", x => x.Runtime(x => x.Format("123"))); // Fluent: Key + Value. -``` - -##### 2.4. Union Types [24-union-types] - -Fluent syntax is now as well available for all auto-generated union- and variant-types. - -```csharp -// TermsQueryField : Union, TermsLookup> - -new TermsQueryDescriptor() - .Terms(x => x.Value("a", "b", "c")) <1> - .Terms(x => x.Lookup(x => x.Index("index").Id("id"))); <2> -``` - -1. `ICollection` -2. `TermsLookup` - -#### 3. Improved Descriptor Design [3-improved-descriptor-design] - -The 9.0 release features a completely overhauled descriptor design. - -Descriptors now wrap the object representation. This brings several internal quality-of-life improvements as well as noticeable benefits to end-users. - -##### 3.1. Wrap [31-wrap] - -Use the wrap constructor to create a new descriptor for an existing object: - -```csharp -var request = new SearchRequest(); - -// Wrap. -var descriptor = new SearchRequestDescriptor(request); -``` - -All fluent methods of the descriptor will mutate the existing `request` passed to the wrap constructor. - -:::{note} - -Descriptors are now implemented as `struct` instead of `class`, reducing allocation overhead as much as possible. - -::: - -##### 3.2. Unwrap / Inspect [32-unwrap-inspect] - -Descriptor values can now be inspected by unwrapping the object using an implicit conversion operator: - -```csharp -var descriptor = new SearchRequestDescriptor(); - -// Unwrap. -SearchRequest request = descriptor; -``` - -Unwrapping does not allocate or copy. - -##### 3.3. Removal of Side Effects [33-removal-of-side-effects] - -In 8.x, execution of (most but not all) lambda actions passed to descriptors was deferred until the actual request was made. It was never clear to the user when, and how often an action would be executed. - -In 9.0, descriptor actions are always executed immediately. This ensures no unforeseen side effects occur if the user-provided lambda action mutates external state (it is still recommended to exclusively use pure/invariant actions). Consequently, the effects of all changes performed by a descriptor method are immediately applied to the wrapped object. - -#### 4. Request Path Parameter Properties [4-request-path-parameter-properties] - -In 8.x, request path parameters like `Index`, `Id`, etc. could only be set by calling the corresponding constructor of the request. Afterwards, there was no way to read or change the current value. - -In the 9.0 client, all request path parameters are exposed as `get/set` properties, allowing for easy access: - -```csharp -// 8.x and 9.0 -var request = new SearchRequest(Indices.All); - -// 9.0 -var request = new SearchRequest { Indices = Indices.All }; -var indices = request.Indices; -request.Indices = "my_index"; -``` - -#### 5. Field Name Inference [5-field-name-inference] - -The `Field` type and especially its implicit conversion operations allowed for `null` return values. This led to a poor developer experience, as the null-forgiveness operator (`!`) had to be used frequently without good reason. - -This is no longer required in 9.0: - -```csharp -// 8.x -Field field = "field"!; - -// 9.0 -Field field = "field"; -``` - -#### 6. Uniform Date/Time/Duration Types [6-uniform-datetimeduration-types] - -The encoding of date, time and duration values in Elasticsearch often varies depending on the context. In addition to string representations in ISO 8601 and RFC 3339 format (always UTC), also Unix timestamps (in seconds, milliseconds, nanoseconds) or simply seconds, milliseconds, nanoseconds are frequently used. - -In 8.x, some date/time values are already mapped as `DateTimeOffset`, but the various non-ISO/RFC representations were not. - -9.0 now represents all date/time values uniformly as `DateTimeOffset` and also uses the native `TimeSpan` type for all durations. - -:::{note} - -There are some places where the Elasticsearch custom date/time/duration types are continued to be used. This is always the case when the type has special semantics and/or offers functionality that goes beyond that of the native date/time/duration types (e.g. `Duration`, `DateMath`). - -::: - -#### 7. Improved Container Design [7-improved-container-design] - -In 8.x, container types like `Query` or `Aggregation` had to be initialized using static factory methods. - -```csharp -// 8.x -var agg = Aggregation.Max(new MaxAggregation { Field = "my_field" }); -``` - -This made it mandatory to assign the created container to a temporary variable if additional properties of the container (not the contained variant) needed to be set: - -```csharp -// 8.x -var agg = Aggregation.Max(new MaxAggregation { Field = "my_field" }); -agg.Aggregations ??= new Dictionary(); -agg.Aggregations.Add("my_sub_agg", Aggregation.Terms(new TermsAggregation())); -``` - -Additionally, it was not possible to inspect the contained variant. - -In 9.0, each possible container variant is represented as a regular property of the container. This allows for determining and inspecting the contained variant and initializing container properties in one go when using an object initializer: - -```csharp -// 9.0 -var agg = new Aggregation -{ - Max = new MaxAggregation { Field = "my_field" }, - Aggregations = new Dictionary - { - { "my_sub_agg", new Aggregation{ Terms = new TermsAggregation() } } - } -}; -``` - -Previously required methods like e.g. `TryGet(out)` have been removed. - -The new recommended way of inspecting container types is to use simple pattern matching: - -```csharp -var query = new Query(); - -if (query.Nested is { } nested) -{ - // We have a nested query. -} -``` - -:::{warning} - -A container can still only contain a single variant. Setting multiple variants at once is invalid. - -Consecutive assignments of variant properties (e.g., first setting `Max`, then `Min`) will cause the previous variant to be replaced. - -::: - -#### 8. Sorting [8-sorting] - -Applying a sort order to a search request using the fluent API is now more convenient: - -```csharp -var search = new SearchRequestDescriptor() - .Sort( - x => x.Score(), - x => x.Score(x => x.Order(SortOrder.Desc)), - x => x.Field(x => x.FirstName), - x => x.Field(x => x.Age, x => x.Order(SortOrder.Desc)), - x => x.Field(x => x.Age, SortOrder.Desc) - // 7.x syntax - x => x.Field(x => x.Field(x => x.FirstName).Order(SortOrder.Desc)) - ); -``` - -The improvements are even more evident when specifying a sort order for aggregations: - -```csharp -new SearchRequestDescriptor() - .Aggregations(aggs => aggs - .Add("my_terms", agg => agg - .Terms(terms => terms - // 8.x syntax. - .Order(new List> - { - new KeyValuePair("_key", SortOrder.Desc) - }) - // 9.0 fluent syntax. - .Order(x => x - .Add(x => x.Age, SortOrder.Asc) - .Add("_key", SortOrder.Desc) - ) - // 9.0 fluent add syntax (valid for all dictionary-like values). - .AddOrder("_key", SortOrder.Desc) - ) - ) - ); -``` - -#### 9. Safer Object Creation [9-safer-object-creation] - -In version 9.0, users are better guided to correctly initialize objects and thus prevent invalid requests. - -For this purpose, at least one constructor is now created that enforces the initialization of all required properties. Existing parameterless constructors or constructor variants that allow the creation of incomplete objects are preserved for backwards compatibility reasons, but are marked as obsolete. - -For NET7+ TFMs, required properties are marked with the `required` keyword, and a non-deprecated parameterless constructor is unconditionally generated. - -:::{note} - -Please note that the use of descriptors still provides the chance to create incomplete objects/requests, as descriptors do not enforce the initialization of all required properties for usability reasons. - -::: - -#### 10. Serialization [10-serialization] - -Serialization in version 9.0 has been completely overhauled, with a primary focus on robustness and performance. Additionally, initial milestones have been set for future support of native AOT. - -In 9.0, round-trip serialization is now supported for all types (limited to all JSON serializable types). - -```csharp -var request = new SearchRequest{ /* ... */ }; - -var json = client.ElasticsearchClientSettings.RequestResponseSerializer.SerializeToString( - request, - SerializationFormatting.Indented -); - -var searchRequestBody = client.ElasticsearchClientSettings.RequestResponseSerializer.Deserialize(json)!; -``` - -:::{warning} - -Note that only the body is serialized for request types. Path- and query properties must be handled manually. - -::: - -:::{note} - -It is important to use the `RequestResponseSerializer` when (de-)serializing client internal types. Direct use of `JsonSerializer` will not work. - -::: diff --git a/docs/release-notes/known-issues.md b/docs/release-notes/known-issues.md deleted file mode 100644 index 004e370a72f..00000000000 --- a/docs/release-notes/known-issues.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -navigation_title: "Known issues" - ---- - -# Elasticsearch .NET Client known issues [elasticsearch-net-client-known-issues] - -Known issues are significant defects or limitations that may impact your implementation. These issues are actively being worked on and will be addressed in a future release. Review the Elasticsearch .NET Client known issues to help you make informed decisions, such as upgrading to a new version. - -% Use the following template to add entries to this page. - -% :::{dropdown} Title of known issue -% **Details** -% On [Month/Day/Year], a known issue was discovered that [description of known issue]. - -% **Workaround** -% Workaround description. - -% **Resolved** -% On [Month/Day/Year], this issue was resolved. - -::: - -Known issues are tracked on [GitHub](https://github.com/elastic/elasticsearch-net/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22Category%3A%20Bug%22). diff --git a/docs/release-notes/release-notes-8.0.0.asciidoc b/docs/release-notes/release-notes-8.0.0.asciidoc new file mode 100644 index 00000000000..5ab8ff266fc --- /dev/null +++ b/docs/release-notes/release-notes-8.0.0.asciidoc @@ -0,0 +1,511 @@ +[[release-notes-8.0.0]] +== Release notes v8.0.0 + +[TIP] +-- +Due to the extensive changes in the new {net-client}, we highly recommend +reviewing this documentation in full, before proceeding with migration to v8. +-- + +After many months of work, eleven alphas, six betas and two release candidates, +we are pleased to announce the GA release of the {net-client} v8.0.0. + +The overall themes of this release have been based around redesigning the client +for the future, standardizing serialization, performance improvements, codebase +simplification, and code-generation. + +The following release notes explain the current state of the client, missing +features, breaking changes and our rationale for some of the design choices we have introduced. + +[discrete] +=== Version 8 is a refresh + +[IMPORTANT] +-- +It is important to highlight that v8 of the {net-client} represents +a new start for the client design. It is important to review how this may affect +your code and usage. +-- + +Mature code becomes increasingly hard to maintain over time, and +our ability to make timely releases has diminished as code complexity has increased. +Major releases allow us to simplify and better align our language clients with +each other in terms of design. Here, it is crucial to find the right balance +between uniformity across programming languages and the idiomatic concerns of +each language. For .NET, we will typically compare and contrast with https://github.com/elastic/elasticsearch-java[Java] and https://github.com/elastic/go-elasticsearch[Go] +to make sure that our approach is equivalent for each of these. We also take +heavy inspiration from Microsoft framework design guidelines and the conventions +of the wider .NET community. + +[discrete] +==== New Elastic.Clients.Elasticsearch NuGet package + +We have shipped the new code-generated client as a +https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/[new NuGet package] +with a new root namespace, `Elastic.Clients.Elasticsearch`. +The new v8 client is built upon the foundations of the v7 `NEST` client, but there +are changes. By shipping as a new package, the expectation is that migration can +be managed with a phased approach. + +While this is a new package, we have aligned the major version (v8.x.x) with the +supported {es} server version to clearly indicate the client/server compatibility. +The v8 client is designed to work with version 8 of {es}. + +The v7 `NEST` client continues to be supported but will not gain new features or +support for new {es} endpoints. It should be considered deprecated in favour of +the new client. + +[discrete] +==== Limited feature set + +[CAUTION] +-- +The 8.0.0 {net-client} does not have feature parity with the previous v7 `NEST` +high-level client. +-- + +For the initial 8.0.0 release we have limited the features we are shipping. +Over the course of the 8.x releases, we will reintroduce features. Therefore, +if something is missing, it may not be permanently omitted. We will endeavour to communicate our plans as and when they become available. + +If a feature you depend on is missing (and not explicitly documented below as a +feature that we do not plan to reintroduce), please open https://github.com/elastic/elasticsearch-net/issues/new/choose[an issue] +or comment on a relevant existing issue to highlight your need to us. This will +help us prioritise our roadmap. + +[discrete] +=== Code generation + +Given the size of the Elasticsearch API surface today, it is no longer practical +to maintain thousands of types (requests, responses, queries, aggregations, etc.) +by hand. To ensure consistent, accurate and timely alignment between language +clients and {es}, the 8.x clients, and many of the associated types are now +automatically code-generated from a https://github.com/elastic/elasticsearch-specification[shared specification]. This is a common solution to maintaining alignment between +client and server among SDKs and libraries, such as those for Azure, AWS and the +Google Cloud Platform. + +Code-generation from a specification has inevitably led to some differences +between the existing v7 `NEST` types and those available in the new v7 {net-client}. +For the 8.0.0 release, we generate strictly from the specification, special +casing a few areas to improve usability or to align with language idioms. + +The base type hierarchy for concepts such as `Properties`, `Aggregations` and +`Queries` is no longer present in generated code, as these arbitrary groupings do +not align with concrete concepts of the public server API. These considerations +do not preclude adding syntactic sugar and usability enhancements to types in future +releases on a case-by-case basis. + +[discrete] +=== Elastic.Transport + +The .NET client includes a transport layer responsible for abstracting HTTP +concepts and to provide functionality such as our request pipeline. This +supports round-robin load-balancing of requests to nodes, pinging failed +nodes and sniffing the cluster for node roles. + +In v7, this layer shipped as `Elasticsearch.Net` and was considered our low-level +client which could be used to send and receive raw JSON bytes between the client +and server. + +As part of the work for 8.0.0, we have moved the transport layer out into +a https://www.nuget.org/packages/Elastic.Transport[new dedicated package] and +https://github.com/elastic/elastic-transport-net[repository], named +`Elastic.Transport`. This supports reuse across future clients and allows +consumers with extremely high-performance requirements to build upon this foundation. + +[discrete] +=== System.Text.Json for serialization + +The v7 `NEST` high-level client used an internalized and modified version of +https://github.com/neuecc/Utf8Json[Utf8Json] for request and response +serialization. This was introduced for its performance improvements +over https://www.newtonsoft.com/json[Json.NET], the more common JSON framework at +the time. + +While Utf8Json provides good value, we have identified minor bugs and +performance issues that have required maintenance over time. Some of these +are hard to change without more significant effort. This library is no longer +maintained, and any such changes cannot easily be contributed back to the +original project. + +With .NET Core 3.0, Microsoft shipped new https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis[System.Text.Json APIs] +that are included in-the-box with current versions of .NET. We have adopted +`System.Text.Json` for all serialization. Consumers can still define and register +their own `Serializer` implementation for their document types should they prefer +to use a different serialization library. + +By adopting `System.Text.Json`, we now depend on a well-maintained and supported +library from Microsoft. `System.Text.Json` is designed from the ground up to support +the latest performance optimizations in .NET and, as a result, provides both fast and low-allocation serialization. + +[discrete] +=== Mockability of ElasticsearchClient + +Testing code is an important part of software development. We recommend +that consumers prefer introducing an abstraction for their use of the {net-client} +as the prefered way to decouple consuming code from client types and support unit +testing. + +In order to support user testing scenarios, we have unsealed the `ElasticsearchClient` +type and made its methods virtual. This supports mocking the type directly for unit +testing. This is an improvement over the original `IElasticClient` interface from +`NEST` (v7) which only supported mocking of top-level client methods. + +We have also introduced a `TestableResponseFactory` in `Elastic.Transport` to +make it easier to create response instances with specific status codes and validity +that can be used during unit testing. + +These changes are in addition to our existing support for testing with an +`InMemoryConnection`, virtualized clusters and with our +https://github.com/elastic/elasticsearch-net-abstractions/blob/master/src/Elastic.Elasticsearch.Managed[`Elastic.Elasticsearch.Managed`] library for integration +testing against real {es} instances. We will introduce more documentation on testing methodologies in the near future. + +[discrete] +=== Migrating to Elastic.Clients.Elasticsearch + +[WARNING] +-- +The 8.0.0 release does not currently have full-feature parity with `NEST`. The +client primary use case is for application developers communitating with {es}. +-- + +The 8.0.0 release focuses on core endpoints, more specifically for common CRUD +scenarios. We intend to reduce the feature gap in subsequent versions. We anticipate +that this initial release will best suit new applications and may not yet be migration-ready for all existing consumers. We recommend reviewing this documentation carefully to learn about the missing features and reduced API surface details before migrating from the v7 `NEST` client. + +The choice to code-generate a new evolution of the {net-client} introduces some +significant breaking changes. We consciously took the opportunity to refactor +and reconsider historic design choices as part of this major release, intending +to limit future breaking changes. + +The v8 client is shipped as a new https://www.nuget.org/packages/Elastic.Clients.Elasticsearch/[NuGet package] +which can be installed alongside v7 `NEST`. We +anticipate that some consumers may prefer a phased migration with both +packages side-by-side for a short period of time to manage complex migrations. In addition, `NEST` 7.17.x can continue to be used in +https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/connecting-to-elasticsearch-v8.html[compatibility mode] +with {es} 8.x servers until the v8 {net-client} features +align with application requirements. + +We will continue to prioritize the feature roadmap and code-generation work +based on https://github.com/elastic/elasticsearch-net/issues[feedback] from consumers who may rely on features that are initially unavailable. + +[discrete] +=== Breaking Changes + +[WARNING] +-- +As a result of code-generating a majority of the client types, this version of +the client includes multiple breaking changes. +-- + +We have strived to keep the core foundation reasonably similar, but types emitted +through code-generation are subject to change between `NEST` (v7) and the new +`Elastic.Clients.Elasticsearch` (v8) package. + +[discrete] +==== Namespaces + +We have renamed the package and top-level namespace for the v8 client to +`Elastic.Clients.Elasticsearch`. All types belong to this namespace. When +necessary, to avoid potential conflicts, types are generated into suitable +sub-namespaces based on the https://github.com/elastic/elasticsearch-specification[{es} specification]. Additional `using` directives may be required to access such types +when using the {net-client}. + +Transport layer concepts have moved to the new `Elastic.Transport` NuGet package +and related types are defined under its namespace. Some configuration and low-level transport functionality may require a `using` directive for the `Elastic.Transport` +namespace. + +[discrete] +==== Type names + +Type names may have changed from previous versions. We are not listing these +explicitly due to the potentially vast number of subtle differences. +Type names will now more closely align to those used in the JSON and as documented +in the {es} documentation. + +[discrete] +==== Class members + +Types may include renamed properties based on the {es} specification, +which differ from the original `NEST` property names. The types used for properties +may also have changed due to code-generation. If you identify missing or +incorrectly-typed properties, please open https://github.com/elastic/elasticsearch-net/issues/new/choose[an issue] to alert us. + +[discrete] +==== Sealing classes + +Opinions on "sealing by default" within the .NET ecosystem tend to be quite +polarized. Microsoft seal all internal types for potential performance gains +and we see a benefit in starting with that approach for the {net-client}, +even for our public API surface. + +While it prevents inheritance and, therefore, may inhibit a few consumer scenarios, +sealing by default is intended to avoid the unexpected or invalid +extension of types that could inadvertently be broken in the future. That said, +sealing is not necessarily a final choice for all types; but it is clearly +easier for a future release to unseal a previously-sealed class than +vice versa. We can therefore choose to unseal as valid scenarios arise, +should we determine that doing so is the best solution for those scenarios, such +as with mockability of the `ElasticsearchClient`. This goes back to our clean-slate concept for this new client. + +[discrete] +==== Removed features + +As part of the clean-slate redesign of the new client, we have opted to remove +certain features from the v8.0 client. These are listed below: + +[discrete] +===== Attribute mappings + +In previous versions of the `NEST` client, attributes could be used to configure +the mapping behaviour and inference for user types. We have removed support for +these attributes and recommend that mapping be completed via the fluent API when +configuring client instances. `System.Text.Json` attributes may be used to rename +and ignore properties during source serialization. + +[discrete] +===== CAT APIs + +The https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html[CAT APIs] +of {es} are intended for human-readable usage and will no longer be supported +via the v8 {net-client}. + +[discrete] +===== Interface removal + +We have removed several interfaces that previously shipped as part of `NEST`. This +is a design decision to simplify the library and avoid interfaces where only a +single implementation of that interface is expected to exist, such as +`IElasticClient` in `NEST`. We have also switched to prefer abstract base classes +over interfaces across the library, as this allows us to add enhancements more +easily without introducing breaking changes for derived types. + +[discrete] +==== Missing features + +While not an exhaustive list, the following are some of the main features which +have not been re-implemented for this initial 8.0.0 GA release. +These remain on our roadmap and will be reviewed and prioritized for inclusion in +future releases. + +* Query DSL operators for combining queries. +* Scroll Helper. +* Fluent API for union types. +* `AutoMap` for field datatype inference. +* Visitor pattern support for types such as `Properties`. +* Support for `JoinField` which affects `ChildrenAggregation`. +* Conditionless queries. +* DiagnosticSources have been removed in `Elastic.Transport` to provide a clean-slate +for an improved diagnostics story. The {net-client} emits https://opentelemetry.io/[OpenTelemetry] compatible `Activity` spans which can be consumed by APM agents such as the https://www.elastic.co/guide/en/apm/agent/dotnet/current/index.html[Elastic APM Agent for .NET]. +* Documentation is a work in progress, and we will expand on the documented scenarios +in future releases. + +[discrete] +=== Reduced API surface + +In this first release of the code-generated .NET client, we have specifically +focused on supporting commonly used endpoints. We have also skipped specific +queries and aggregations which need further work to generate code correctly. +Before migrating, please refer to the lists below for the endpoints, +queries and aggregations currently generated and available +in the 8.0.0 GA release to ensure that the features you are using are currently +supported. + +[discrete] +==== Supported {es} endpoints + +The following are {es} endpoints currently generated and available +in the 8.0.0 {net-client}. + +* AsyncSearch.Delete +* AsyncSearch.Get +* AsyncSearch.Status +* AsyncSearch.Submit +* Bulk +* ClearScroll +* ClosePointInTime +* Cluster.Health +* Count +* Create +* Delete +* DeleteByQuery +* DeleteByQueryRethrottle +* DeleteScript +* EQL.Delete +* EQL.Get +* EQL.Search +* EQL.Status +* Exists +* ExistsSource +* Explain +* FieldCaps +* Get +* GetScript +* GetScriptContext +* GetScriptLanguages +* GetSource +* Index +* Indices.Clone +* Indices.Close +* Indices.Create +* Indices.CreateDataStream +* Indices.Delete +* Indices.DeleteAlias +* Indices.DeleteDataStream +* Indices.DeleteIndexTemplate +* Indices.DeleteTemplate +* Indices.Exists +* Indices.ExistsIndexTemplate +* Indices.ExistsTemplate +* Indices.Flush +* Indices.ForceMerge +* Indices.Get +* Indices.GetAlias +* Indices.GetDataStream +* Indices.GetFieldMapping +* Indices.GetIndexTemplate +* Indices.GetMapping +* Indices.GetTemplate +* Indices.Indices.SimulateTemplate +* Indices.MigrateToDataStream +* Indices.Open +* Indices.PutAlias +* Indices.PutIndexTemplate +* Indices.PutMapping +* Indices.PutTemplate +* Indices.Refresh +* Indices.Rollover +* Indices.Shrink +* Indices.SimulateIndexTemplate +* Indices.Split +* Indices.Unfreeze +* Info +* MGet +* MSearch +* MSearchTemplate +* OpenPointInTime +* Ping +* PutScript +* RankEval +* Reindex +* ReindexRethrottle +* Scroll +* Search +* SearchShards +* SQL.ClearCursor +* SQL.DeleteAsync +* SQL.GetAsync +* SQL.GetAsyncStatus +* SQL.Query +* TermsEnum +* Update +* UpdateByQuery +* UpdateByQueryRethrottle + +[discrete] +==== Supported queries + +The following are query types currently generated and available +in the 8.0.0 {net-client}. + +* Bool +* Boosting +* CombinedFields +* ConstantScore +* DisMax +* Exists +* FunctionScore +* Fuzzy +* HasChild +* HasParent +* Ids +* Intervals +* Match +* MatchAll +* MatchBool +* MatchNone +* MatchPhrase +* MatchPhrasePrefix +* MoreLikeThis +* MultiMatch +* Nested +* ParentId +* Percolate +* Prefix +* QueryString +* RankFeature +* Regexp +* Script +* ScriptScore +* Shape +* SimpleQueryString +* SpanContaining +* SpanFirst +* SpanMultiTerm +* SpanNear +* SpanNot +* SpanOr +* SpanTerm +* SpanWithin +* Term +* Terms +* TermsSet +* Wildcard +* Wrapper + +[discrete] +==== Supported aggregations + +The following are aggregation types currently generated and available +in the 8.0.0 {net-client}. + +* AdjacencyMatrix +* AutoDateHistogram +* Avg +* Boxplot +* Cardinality +* Children +* Composite +* CumulativeCardinality +* DateHistogram +* DateRange +* Derivative +* ExtendedStats +* Filters +* Global +* Histogram +* Inference +* IpRange +* MatrixStats +* Max +* MedianAbsoluteDeviation +* Min +* Missing +* MultiTerms +* Nested +* Parent +* PercentilesBucket +* Range +* Rate +* ReverseNested +* Sampler +* ScriptedMetric +* Stats +* StatsBucket +* StringStats +* Sum +* Terms +* TopHits +* TopMetrics +* TTest +* ValueCount +* VariableWidthHistogram +* WeightedAvg + +[discrete] +=== In closing + +Please give the new `Elastic.Clients.Elasticsearch` client a try in your .NET +applications. If you run into any problems, please open https://github.com/elastic/elasticsearch-net/issues/new/choose[an issue] to raise those +with us. Please let us know how you get on and if you have any questions, +reach out on the https://discuss.elastic.co[Discuss forums]. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.1.asciidoc b/docs/release-notes/release-notes-8.0.1.asciidoc new file mode 100644 index 00000000000..93d2dd8d376 --- /dev/null +++ b/docs/release-notes/release-notes-8.0.1.asciidoc @@ -0,0 +1,71 @@ +[[release-notes-8.0.1]] +== Release notes v8.0.1 + +[discrete] +=== Bug fixes + +- Fix MultiSearchTemplateRequest body serialization (issue: +https://github.com/elastic/elasticsearch-net/issues/7006[#7006]) + +[discrete] +=== Enhancements + +- Seal union types for consistency + +[discrete] +=== Breaking changes + +This release includes the following breaking changes: + +[discrete] +==== MultiSearchTemplate type changes + +The `Core.MSearchTemplate.RequestItem` type has been renamed to +`Core.MSearchTemplate.SearchTemplateRequestItem`. It no longer derives from the +`Union` type. It has been manually designed to support serialization to +NDJSON, as required by the MSearchTemplate endpoint. + +The `MultiSearchTemplateRequest.SearchTemplates` property has been updated to +use this newly defined type. + +This breaking change has been included in this patch release due to the +original code-generated type functioning incorrectly, and therefore, we have +determined that this should ship ASAP. + +[discrete] +==== MultiSearch type changes + +The `Core.MSearch.SearchRequestItem` type has been sealed for consistency with +the design choices of the rest of the client. While technically breaking, we +have decided that this should be included in this release before any potentially +derived types may exist in consuming applications. + +[discrete] +==== Sealing union types + +Code-generated types derived from `Union` were incorrectly unsealed. +While technically breaking, we have decided that these should be sealed in this +patch release before any potential derived types may exist in consuming +applications. Sealing types by default aligns with our broader design choices +and this decision is described in the <>. + +Affected types: +- `Aggregations.Buckets` +- `Aggregations.FieldDateMatch` +- `Aggregations.Percentiles` +- `Analysis.CharFilter` +- `Analysis.TokenFilter` +- `Analysis.Tokenizer` +- `ByteSize` +- `Fuzziness` +- `GeoHashPrecision` +- `MultiGetResponseItem` +- `MultiSearchResponseItem` +- `QueryDsl.Like` +- `QueryDsl.TermsQueryField` +- `Script` +- `Slices` +- `SourceConfig` +- `SourceConfigParam` +- `Tasks.TaskInfos` +- `TrackHits` \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.10.asciidoc b/docs/release-notes/release-notes-8.0.10.asciidoc new file mode 100644 index 00000000000..8bd7bfd896e --- /dev/null +++ b/docs/release-notes/release-notes-8.0.10.asciidoc @@ -0,0 +1,11 @@ +[[release-notes-8.0.10]] +== Release notes v8.0.10 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7549[#7549] Update to latest +transport to ensure ActivitySource is static. (issue: https://github.com/elastic/elasticsearch-net/issues/7540[#7540]) + +This avoids undue and potentially high volume allocations of `ActivitySource` across +consuming applications and is therefore a recommended upgrade. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.2.asciidoc b/docs/release-notes/release-notes-8.0.2.asciidoc new file mode 100644 index 00000000000..5b53dcf4773 --- /dev/null +++ b/docs/release-notes/release-notes-8.0.2.asciidoc @@ -0,0 +1,82 @@ +[[release-notes-8.0.2]] +== Release notes v8.0.2 + +[discrete] +=== Bug fixes + +- Add missing accessor properties for dictionary responses (issue: +https://github.com/elastic/elasticsearch-net/issues/7048[#7048]) +- Fix to ensure dynamic HTTP methods are used when available (issue: +https://github.com/elastic/elasticsearch-net/issues/7057[#7057]) +- Fix resolvable dictionary properties (issue: +https://github.com/elastic/elasticsearch-net/issues/7075[#7075]) + +[discrete] +=== Breaking changes + +Some low-impact changes were made to existing types to fix the resolvable +dictionary properties. We determined it worthwhile to retype the properties to +prefer the interfaces over concrete types. + +[discrete] +==== Changes to dictionary properties on generated types + +As part of fixing the resolvable dictionary properties some low-impact changes +were made to the generated types. We determined it worthwhile to retype the +properties to prefer the interfaces over concrete types. + +Types that are immutable and only apply to server responses now use +`IReadOnlyDictionary` for relevant properties. For mutable types, we prefer +`IDictionary`. + +`HealthResponse.Indices` has changed from a bespoke `ReadOnlyIndexNameDictionary` +property to prefer `IReadOnlyDictionary` to improve ease of use and familiarity. + +[discrete] +==== Internalise ReadOnlyIndexNameDictionary + +After changes for resolvable dictionaries, the `ReadOnlyIndexNameDictionary` type +was made internal and is no longer part of the public API. Properties that +previously used this type are now typed as `IReadOnlyDictionary`. This brings +advantages in being more familiar for developers. + +[discrete] +==== Remove IndexName.GetString(ITransportConfiguration settings) method + +This method is used internally by the client and should not be exposed to +consuming applications. Instead, we prefer explicit interface implementation for +`IUrlParameter.GetString`. + +[discrete] +==== Remove Metric.GetString(ITransportConfiguration settings) method + +This method is used internally by the client and should not be exposed to +consuming applications. Instead, we prefer explicit interface implementation for +`IUrlParameter.GetString`. + +[discrete] +==== Remove TimeStamp.GetString(ITransportConfiguration settings) method + +This method is used internally by the client and should not be exposed to +consuming applications. Instead, we prefer explicit interface implementation for +`IUrlParameter.GetString`. + +[discrete] +==== Remove IndexUuid.GetString(ITransportConfiguration settings) method + +This method is used internally by the client and should not be exposed to +consuming applications. Instead, we prefer explicit interface implementation for +`IUrlParameter.GetString`. + +[discrete] +==== Remove TaskId.GetString(ITransportConfiguration settings) method + +This method is used internally by the client and should not be exposed to +consuming applications. Instead, we prefer explicit interface implementation for +`IUrlParameter.GetString`. + +[discrete] +==== The Metric type is now sealed + +This type has been sealed to align with other types for consistency. We don’t +expect consumers to derive from this type. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.3.asciidoc b/docs/release-notes/release-notes-8.0.3.asciidoc new file mode 100644 index 00000000000..64c8dccfedf --- /dev/null +++ b/docs/release-notes/release-notes-8.0.3.asciidoc @@ -0,0 +1,17 @@ +[[release-notes-8.0.3]] +== Release notes v8.0.3 + +[discrete] +=== Bug fixes + +- Fix field sort serialization (issue: +https://github.com/elastic/elasticsearch-net/issues/7074[#7074]) + +[discrete] +=== Enhancements + +[discrete] +==== Update to Elastic.Transport 0.4.5 + +Upgrades the client to depend on the 0.4.5 release of Elastic.Transport which +includes automatic sending of https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-api-compatibility.html#rest-api-compatibility[REST API compatibility] headers for Elasticsearch requests. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.4.asciidoc b/docs/release-notes/release-notes-8.0.4.asciidoc new file mode 100644 index 00000000000..ac61771ebde --- /dev/null +++ b/docs/release-notes/release-notes-8.0.4.asciidoc @@ -0,0 +1,138 @@ +[[release-notes-8.0.4]] +== Release notes v8.0.4 + +[discrete] +=== Bug fixes + +- Fix code-gen for IndexSettingsAnalysis (issue: +https://github.com/elastic/elasticsearch-net/issues/7118[#7118]) +- Complete implementation of Metrics type +- Update generated code with fixes from 8.6 specification (issue: +https://github.com/elastic/elasticsearch-net/issues/7119[#7119]). Adds `Missing` +property to `MultiTermLookup`. + +[discrete] +=== Breaking changes + +In the course of fixing the code-generation of types used on `IndexSettingsAnalysis`, +several breaking changes were introduced. Some of these were necessary to make the +types usable, while others fixed the consistency of the generated code. + +[discrete] +==== IndexSettingsAnalysis + +Code-generation has been updated to apply transforms to fix the specification +of the `IndexSettingsAnalysis` type. As a result, all properties have been renamed, +and some property types have been changed. + +* The `Analyzer` property is now pluralized and renamed to `Analyzers` to align with +NEST and make it clearer that this can contain more than one analyzer definition. +* The `CharFilter` property is now pluralized and renamed to `CharFilters` to align with +NEST and make it clearer that this can contain more than one char filter definition. +Its type has changes from a `IDictionary` +to `CharFilters`, a tagged union type deriving from IsADictionary`. +* The `Filter` property is now pluralized and renamed to `TokenFilters` to align with +NEST and make it clearer that this can contain more than one token filter definition. +Its type has changes from a `IDictionary` +to `TokenFilters`, a tagged union type deriving from IsADictionary`. +* The `Normalizer` property is now pluralized and renamed to `Normalizers` to align with +NEST and make it clearer that this can contain more than one normalizer definition. +* The `Tokenizer` property is now pluralized and renamed to `Tokenizers` to align with +NEST and make it clearer that this can contain more than one tokenizer definition. +Its type has changes from a `IDictionary` +to `TokenFilters`, a tagged union type deriving from IsADictionary`. + +*_Before_* + +[source,csharp] +---- +public sealed partial class IndexSettingsAnalysis +{ + public Elastic.Clients.Elasticsearch.Analysis.Analyzers? Analyzer { get; set; } + public IDictionary? CharFilter { get; set; } + public IDictionary? Filter { get; set; } + public Elastic.Clients.Elasticsearch.Analysis.Normalizers? Normalizer { get; set; } + public IDictionary? Tokenizer { get; set; } +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class IndexSettingsAnalysis +{ + public Elastic.Clients.Elasticsearch.Analysis.Analyzers? Analyzers { get; set; } + public Elastic.Clients.Elasticsearch.Analysis.CharFilters? CharFilters { get; set; } + public Elastic.Clients.Elasticsearch.Analysis.TokenFilters? TokenFilters { get; set; } + public Elastic.Clients.Elasticsearch.Analysis.Normalizers? Normalizers { get; set; } + public Elastic.Clients.Elasticsearch.Analysis.Tokenizers? Tokenizers { get; set; } +} +---- + +The `IndexSettingsAnalysisDescriptor` type has been updated accordingly to apply +the above changes. It now supports a more convenient syntax to easily define +the filters, normalizers and tokenizers that apply to the settings for indices. + +[discrete] +===== Example usage of updated fluent syntax: + +[source,csharp] +---- +var descriptor = new CreateIndexRequestDescriptor("test") + .Settings(s => s + .Analysis(a => a + .Analyzers(a => a + .Stop("stop-name", stop => stop.StopwordsPath("analysis/path.txt")) + .Pattern("pattern-name", pattern => pattern.Version("version")) + .Custom("my-custom-analyzer", c => c + .Filter(new[] { "stop", "synonym" }) + .Tokenizer("standard"))) + .TokenFilters(f => f + .Synonym("synonym", synonym => synonym + .SynonymsPath("analysis/synonym.txt"))))); +---- + +[discrete] +==== Token Filters + +Token filter types now implement the `ITokenFilter` interface, rather than +`ITokenFilterDefinition`. + +The `TokenFilter` union type has been renamed to `CategorizationTokenFilter` to +clearly signify it's use only within ML categorization contexts. + +A `TokenFilters` type has been introduced, which derives from `IsADictionary` and +supports convenient addition of known token filters via the fluent API. + +[discrete] +==== Character Filters + +Character filter types now implement the `ICharFilter` interface, rather than +`ICharFilterDefinition`. + +The `CharFilter` union type has been renamed to `CategorizationCharFilter` to +clearly signify it's use only within ML categorization contexts. + +A `CharFilters` type has been introduced, which derives from `IsADictionary` and +supports convenient addition of known character filters via the fluent API. + +[discrete] +==== Tokenizers + +Tokenizer types now implement the `ITokenizer` interface, rather than +`ITokenizerDefinition`. + +The `Tokenizer` union type has been renamed to `CategorizationTokenizer` to +clearly signify it's use only within ML categorization contexts. + +A `Tokenizers` type has been introduced, which derives from `IsADictionary` and +supports convenient addition of known tokenizers via the fluent API. + +[discrete] +==== IndexManagement.StorageType + +The 8.6 specification fixed this type to mark is as a non-exhaustive enum, since +it supports additional values besides those coded into the specification. As a +result the code-generation for this type causes some breaking changes. The type +is no longer generated as an `enum` and is not a custom `readonly struct`. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.5.asciidoc b/docs/release-notes/release-notes-8.0.5.asciidoc new file mode 100644 index 00000000000..15961356b15 --- /dev/null +++ b/docs/release-notes/release-notes-8.0.5.asciidoc @@ -0,0 +1,98 @@ +[[release-notes-8.0.5]] +== Release notes v8.0.5 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7171[#7171] Fix code-gen for IndexTemplate (issue: https://github.com/elastic/elasticsearch-net/issues/7161[#7161]) +- https://github.com/elastic/elasticsearch-net/pull/7181[#7181] Fix MultiGet response deserialization for non-matched IDs (issue: https://github.com/elastic/elasticsearch-net/issues/7169[#7169]) +- https://github.com/elastic/elasticsearch-net/pull/7182[#7182] Implement Write method on SourceConfigConverter (issue: https://github.com/elastic/elasticsearch-net/issues/7170[#7170]) +- https://github.com/elastic/elasticsearch-net/pull/7205[#7205] Update to Elastic.Transport to 0.4.6 which improves the version detection used by the REST API compatibility Accept header + +[discrete] +=== Breaking changes + +In the course of fixing the code-generation for index templates to avoid serialization failures, some breaking changes were introduced. + +[discrete] +==== IndexTemplate + +`IndexTemplate` forms part of the `IndexTemplateItem` included on `GetIndexTemplateResponse`. + +* The type for the `ComposedOf` property has changed from `IReadOnlyCollection` to `IReadOnlyCollection` +* The type for the `IndexPatterns` property has changed from `Elastic.Clients.Elasticsearch.Names` to `IReadOnlyCollection` + +*_Before_* + +[source,csharp] +---- +public sealed partial class IndexTemplate +{ + ... + public IReadOnlyCollection ComposedOf { get; init; } + public Elastic.Clients.Elasticsearch.Names IndexPatterns { get; init; } + ... +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class IndexTemplate +{ + ... + public IReadOnlyCollection ComposedOf { get; init; } + public IReadOnlyCollection IndexPatterns { get; init; } + ... +} +---- + +[discrete] +==== SimulateIndexTemplateRequest + +* The type for the `ComposedOf` property has changed from `IReadOnlyCollection` to `IReadOnlyCollection` + +*_Before_* + +[source,csharp] +---- +public sealed partial class SimulateIndexTemplateRequest +{ + ... + public IReadOnlyCollection? ComposedOf { get; set; } + ... +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class SimulateIndexTemplateRequest +{ + ... + public IReadOnlyCollection? ComposedOf { get; set; } + ... +} +---- + +[discrete] +==== SimulateIndexTemplateRequestDescriptor and SimulateIndexTemplateRequestDescriptor + +The `ComposedOf` method signature has changed to accept a parameter of `ICollection?` instead of +`ICollection?`. + +*_Before_* + +[source,csharp] +---- +public SimulateIndexTemplateRequestDescriptor ComposedOf(ICollection? composedOf) +---- + +*_After_* + +[source,csharp] +---- +public SimulateIndexTemplateRequestDescriptor ComposedOf(ICollection? composedOf) +---- \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.6.asciidoc b/docs/release-notes/release-notes-8.0.6.asciidoc new file mode 100644 index 00000000000..301f18470fd --- /dev/null +++ b/docs/release-notes/release-notes-8.0.6.asciidoc @@ -0,0 +1,110 @@ +[[release-notes-8.0.6]] +== Release notes v8.0.6 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7244[#7244] Fix code-gen for +single or many types. Includes support for deserializing numbers represented as +strings in the JSON payload. (issues: https://github.com/elastic/elasticsearch-net/issues/7221[#7221], +https://github.com/elastic/elasticsearch-net/issues/7234[#7234], +https://github.com/elastic/elasticsearch-net/issues/7240[#7240]). +- https://github.com/elastic/elasticsearch-net/pull/7253[#7253] Fix code-gen for +enums with aliases (issue: https://github.com/elastic/elasticsearch-net/issues/7236[#7236]) +- https://github.com/elastic/elasticsearch-net/pull/7262[#7262] Update to +`Elastic.Transport` 0.4.7 which includes fixes for helpers used during application +testing. + +[discrete] +=== Features + +- https://github.com/elastic/elasticsearch-net/pull/7272[#7272] Support custom JsonSerializerOptions. + +[discrete] +=== Breaking changes + +[discrete] +==== DynamicTemplate + +`DynamicTemplate` forms part of the `TypeMapping` object, included on `GetIndexRespone`. + +* The type for the `Mapping` property has changed from `Elastic.Clients.Elasticsearch.Properties` +to `Elastic.Clients.Elasticsearch.IProperty`. This breaking change fixes an error +introduced by the code-generator. Before introducing this fix, the type could +not correctly deserialize responses for GET index requests and prevented dynamic +templates from being configured for indices via PUT index. + +*_Before_* + +[source,csharp] +---- +public sealed partial class DynamicTemplate +{ + ... + public Elastic.Clients.Elasticsearch.Mapping.Properties? Mapping { get; set; } + ... +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class DynamicTemplate +{ + ... + public Elastic.Clients.Elasticsearch.Mapping.IProperty? Mapping { get; set; } + ... +} +---- + +[discrete] +==== TypeMapping + +Among other uses, `TypeMapping` forms part of the `GetIndexRespone`. + +* The `DynamicTemplates` property has been simplified to make it easier to work +with and to fix deserialization failures on certain responses. Rather than use a +`Union` to describe the fact that this property may be a single dictionary of +dynamic templates, or an array of dictionaries, this is now code-generated as a +specialised single or many collection. The API exposes this as an `ICollection` +of dictionaries and the JSON converter is able to handle either an array or +individual dictionary in responses. + +*_Before_* + +[source,csharp] +---- +public sealed partial class TypeMapping +{ + ... + public Union?, ICollection>?>? DynamicTemplates { get; set; } + ... +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class TypeMapping +{ + ... + public ICollection>? DynamicTemplates { get; set; } + ... +} +---- + +[discrete] +==== SystemTextJsonSerializer + +The `SystemTextJsonSerializer` is used as a base type for the built-in serializers. Two breaking changes have been made after adding better support for <>. + +The public `Options` property has been made internal. + +A new public abstract method `CreateJsonSerializerOptions` has been added, which derived types must implement. + +[source,csharp] +---- +protected abstract JsonSerializerOptions CreateJsonSerializerOptions(); +---- diff --git a/docs/release-notes/release-notes-8.0.7.asciidoc b/docs/release-notes/release-notes-8.0.7.asciidoc new file mode 100644 index 00000000000..fd5c2261708 --- /dev/null +++ b/docs/release-notes/release-notes-8.0.7.asciidoc @@ -0,0 +1,7 @@ +[[release-notes-8.0.7]] +== Release notes v8.0.7 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7337[#7337] Fix code-gen for dynamic_templates. (issue: https://github.com/elastic/elasticsearch-net/issues/7234[#7234]) \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.0.8.asciidoc b/docs/release-notes/release-notes-8.0.8.asciidoc new file mode 100644 index 00000000000..9952e1c6cee --- /dev/null +++ b/docs/release-notes/release-notes-8.0.8.asciidoc @@ -0,0 +1,7 @@ +[[release-notes-8.0.8]] +== Release notes v8.0.8 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7456[#7456] Fix CompletionSuggester based on spec fixes. (issue: https://github.com/elastic/elasticsearch-net/issues/7454[#7454]) diff --git a/docs/release-notes/release-notes-8.0.9.asciidoc b/docs/release-notes/release-notes-8.0.9.asciidoc new file mode 100644 index 00000000000..b9086a404b5 --- /dev/null +++ b/docs/release-notes/release-notes-8.0.9.asciidoc @@ -0,0 +1,34 @@ +[[release-notes-8.0.9]] +== Release notes v8.0.9 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7446[#7446] Fix byte properties +in index stats types. (issue: https://github.com/elastic/elasticsearch-net/issues/7445[#7445]) + +[discrete] +=== Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7467[#7467] Source serialization +always sends fractional format for double and floats. (issue: https://github.com/elastic/elasticsearch-net/issues/7051[#7051]) + +[discrete] +=== Breaking changes + +[discrete] +==== Source serialization of float and double properties + +By default, when serializing `double` and `float` properties, the `System.Text.Json` +serializer uses the "G17" format when serializing double types. This format omits +the decimal point and/or trailing zeros if they are not required for the data to +roundtrip. This is generally correct, as JSON doesn't specify a type for numbers. + +However, in the case of source serialization, mappings for numeric properties may +be incorrectly inferred if trailing zeros are omitted. In this release, we have +included a new custom converter for `float` and `double` types when serialized using +the default source serializer. These converters ensure that at least one precision +digit is included after a decimal point, even for round numbers. + +You may therefore observe changes to the serialized source document after +upgrading to this version. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.1.0.asciidoc b/docs/release-notes/release-notes-8.1.0.asciidoc new file mode 100644 index 00000000000..3fdac1643ff --- /dev/null +++ b/docs/release-notes/release-notes-8.1.0.asciidoc @@ -0,0 +1,90 @@ +[[release-notes-8.1.0]] +== Release notes v8.1.0 + +A core theme of the 8.1.0 release is the reintroduction of many features which +were missing from the 8.0 releases. The 8.x client still does NOT have full +feature parity with NEST and we continue to work on closing these gaps. + +[discrete] +=== Enhancements + +[discrete] +==== Support for additional endpoints + +Adds support for the following endpoints: + +- Cluster.AllocationExplain +- Cluster.Stats +- Cluster.PendingTasks +- DanglingIndices.List +- Enrich.DeletePolicy +- Enrich.ExecutePolicy +- Enrich.PutPolicy +- Enrich.Stats +- Graph.Explore +- IndexManagement.UpdateAliases +- Ingest.GeoIpStats +- Ingest.GetPipeline +- Ingest.ProcessorGrok +- Ingest.PutPipeline +- Ingest.Simulate +- MultiTermVectors +- RenderSearchTemplate +- SearchTemplate +- Tasks.Cancel +- Tasks.Get +- Tasks.List +- TermVectors + +[discrete] +==== Support for additional queries + +Adds support for the following queries: + +- Geo distance +- Geo bounding box +- Geo polygon +- Pinned +- Range queries (date and numeric) +- Raw (can be used as a client specific fallback for missing queries by sending raw JSON) + +[discrete] +==== Support for additional aggregations + +Adds support for the following aggregations: + +- Boxplot +- Bucket sort +- Composite +- Cumulative sum +- Geo bounds +- Geo centroid +- Geo distance +- Geo line +- Geohash grid +- Geohex grid +- Geotile grid +- IP prefix +- Multi terms +- Rare terms +- Significant terms +- Weighted average + +[discrete] +==== Other enhancements + +- *Add support for geo distance sorting.* +Adds support for specifying a `GeoDistanceSort` on `SortOptions`. +- *Add support for weight score on FunctionScore.* +Adds support for specifying a weight score value on the `FunctionScore` type. +- *Code generate XML doc comments.* +The code generator now adds XML doc comments to types and members when present in +the Elasticsearch specification. This acts as an aid when exploring the API in an +IDE such as Visual Studio. +- *Add additional client overloads.* +Adds additional overloads to the `ElasticsearchClient` and namespaced sub-clients +that allow consumers to provide a descriptor instance used when building requests. +- *Add support for bool query operators in Query DSL for object initializer syntax* +Adds support for using operators `&&``, `||`, `!` and `+` to build up bool queries +using the object initializer syntax. NOTE: Operators are not yet supported for +combining queires defined using the fluent descriptor syntax. \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.1.1.asciidoc b/docs/release-notes/release-notes-8.1.1.asciidoc new file mode 100644 index 00000000000..100b6e99f12 --- /dev/null +++ b/docs/release-notes/release-notes-8.1.1.asciidoc @@ -0,0 +1,72 @@ +[[release-notes-8.1.1]] +== Release notes v8.1.1 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7667[#7667] Fix SQL missing +Rows on QueryResponse (issue: https://github.com/elastic/elasticsearch-net/issues/7663[#7663]) +- https://github.com/elastic/elasticsearch-net/pull/7676[#7676] Ensure async client +methods pass through cancellation token (issue: https://github.com/elastic/elasticsearch-net/issues/7665[#7665]) + +[discrete] +=== Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7684[#7684] Regenerated code +with latest spec fixes for 8.7 + +[discrete] +=== Breaking changes + +This release includes the following breaking changes as a result of specification fixes: + +[discrete] +==== AsyncSearch and MultisearchBody KnnQuery + +The type for the `SubmitAsyncSearchRequest.Knn` and `MultisearchBody.Knn` properties +has changed to an `ICollection` from a single `KnnQuery` since it is +possible to include more than one query in a request. + +*_Before_* + +[source,csharp] +---- +public sealed partial class SubmitAsyncSearchRequest +{ + ... + public Elastic.Clients.Elasticsearch.KnnQuery? Knn { get; set; } + ... +} +---- + +[source,csharp] +---- +public sealed partial class MultisearchBody +{ + ... + public Elastic.Clients.Elasticsearch.KnnQuery? Knn { get; set; } + ... +} +---- + +*_After_* + +[source,csharp] +---- +public sealed partial class SubmitAsyncSearchRequest +{ + ... + public ICollection? Knn { get; set; } + ... +} +---- + +[source,csharp] +---- +public sealed partial class MultisearchBody +{ + ... + public ICollection? Knn { get; set; } + ... +} +---- \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.1.2.asciidoc b/docs/release-notes/release-notes-8.1.2.asciidoc new file mode 100644 index 00000000000..71c34bca8b2 --- /dev/null +++ b/docs/release-notes/release-notes-8.1.2.asciidoc @@ -0,0 +1,17 @@ +[[release-notes-8.1.2]] +== Release notes v8.1.2 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7718[#7718] Regen index setting blocks based on fixed spec (issue: https://github.com/elastic/elasticsearch-net/issues/7714[#7714]) + +[discrete] +=== Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7781[#7781] Bump dependencies (issue: https://github.com/elastic/elasticsearch-net/issues/7752[#7752]) + +[discrete] +=== Docs + +- https://github.com/elastic/elasticsearch-net/pull/7772[#7772] [Backport 8.1] [Backport 8.7][DOCS] Adds getting started content based on the template (issue: https://github.com/elastic/elasticsearch-net/pull/7770[#7770]) \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.1.3.asciidoc b/docs/release-notes/release-notes-8.1.3.asciidoc new file mode 100644 index 00000000000..e436f226717 --- /dev/null +++ b/docs/release-notes/release-notes-8.1.3.asciidoc @@ -0,0 +1,19 @@ +[[release-notes-8.1.3]] +== Release notes v8.1.3 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7737[#7737] Boosted non-exhaustive enum deserialization (issue: https://github.com/elastic/elasticsearch-net/issues/7729[#7729]) +- https://github.com/elastic/elasticsearch-net/pull/7738[#7738] Complted buckets JSON converter (issue: https://github.com/elastic/elasticsearch-net/issues/7713[#7713]) +- https://github.com/elastic/elasticsearch-net/pull/7753[#7753] Number converters should not fall through and throw exceptions in non NETCore builds (issue: https://github.com/elastic/elasticsearch-net/issues/7757[#7757]) +- https://github.com/elastic/elasticsearch-net/pull/7811[#7811] Fix localization issue with floating-point deserialization from string + +[discrete] +=== Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7730[#7730] Refactoring and tiny behavior fix for Ids +- https://github.com/elastic/elasticsearch-net/pull/7731[#7731] No allocations in `ResponseItem.IsValid`` property +- https://github.com/elastic/elasticsearch-net/pull/7733[#7733] Fixed the equality contract on Metrics type +- https://github.com/elastic/elasticsearch-net/pull/7735[#7735] Removed unused `JsonIgnore` +- https://github.com/elastic/elasticsearch-net/pull/7736[#7736] Optimized `FieldConverter` \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.10.0.asciidoc b/docs/release-notes/release-notes-8.10.0.asciidoc new file mode 100644 index 00000000000..9b587de7bea --- /dev/null +++ b/docs/release-notes/release-notes-8.10.0.asciidoc @@ -0,0 +1,13 @@ +[[release-notes-8.10.0]] +== Release notes v8.10.0 + +[discrete] +=== Features & Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7931[#7931] Refactor OpenTelemetry implementation with updated Transport (issue: https://github.com/elastic/elasticsearch-net/issues/7885[#7885]) +- https://github.com/elastic/elasticsearch-net/pull/7953[#7953] Add `TDigestPercentilesAggregate` (issues: https://github.com/elastic/elasticsearch-net/issues/7923[#7923], https://github.com/elastic/elasticsearch-net/issues/7879[#7879]) + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7956[#7956] Add `Similarity` to `KnnQuery` (issue: https://github.com/elastic/elasticsearch-net/issues/7952[#7952]) \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.11.0.asciidoc b/docs/release-notes/release-notes-8.11.0.asciidoc new file mode 100644 index 00000000000..1e002cb0b4e --- /dev/null +++ b/docs/release-notes/release-notes-8.11.0.asciidoc @@ -0,0 +1,13 @@ +[[release-notes-8.11.0]] +== Release notes v8.11.0 + +[discrete] +=== Features & Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7978[#7978] Regenerate client for 8.11 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7979[#7979] Add workaround for stringified properties which are not marked properly in specification +- https://github.com/elastic/elasticsearch-net/pull/7965[#7965] Fix `Stringified` converters \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.9.0.asciidoc b/docs/release-notes/release-notes-8.9.0.asciidoc new file mode 100644 index 00000000000..0a622988037 --- /dev/null +++ b/docs/release-notes/release-notes-8.9.0.asciidoc @@ -0,0 +1,15 @@ +[[release-notes-8.9.0]] +== Release notes v8.9.0 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7839[#7839] Use `Stringified` for `preserve_original` and `indexing_complete` (issue: https://github.com/elastic/elasticsearch-net/issues/7755[#7755]) +- https://github.com/elastic/elasticsearch-net/pull/7840[#7840] Update `Elastic.*` dependencies (issue: https://github.com/elastic/elasticsearch-net/issues/7823[#7823]) +- https://github.com/elastic/elasticsearch-net/pull/7841[#7841] Fix typing of `BulkUpdateOperation.RetryOnConflict` (issue: https://github.com/elastic/elasticsearch-net/issues/7838[#7838]) +- https://github.com/elastic/elasticsearch-net/pull/7854[#7854] Fix custom floating-point JSON converters (issue: https://github.com/elastic/elasticsearch-net/issues/7757[#7757]) + +[discrete] +=== Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7836[#7836] Regenerate client using 8.9 specification \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.9.1.asciidoc b/docs/release-notes/release-notes-8.9.1.asciidoc new file mode 100644 index 00000000000..ea5a4f19aeb --- /dev/null +++ b/docs/release-notes/release-notes-8.9.1.asciidoc @@ -0,0 +1,7 @@ +[[release-notes-8.9.1]] +== Release notes v8.9.1 + +[discrete] +=== Bug fixes + +- https://github.com/elastic/elasticsearch-net/pull/7864[#7864] Fix `TextExpansionQuery` definition diff --git a/docs/release-notes/release-notes-8.9.2.asciidoc b/docs/release-notes/release-notes-8.9.2.asciidoc new file mode 100644 index 00000000000..e148e5a1673 --- /dev/null +++ b/docs/release-notes/release-notes-8.9.2.asciidoc @@ -0,0 +1,14 @@ +[[release-notes-8.9.2]] +== Release notes v8.9.2 + +[discrete] +=== Bug fixes + + - https://github.com/elastic/elasticsearch-net/pull/7875[#7875] Fix `aggregations` property not being generated for `MultisearchBody` (issue https://github.com/elastic/elasticsearch-net/issues/7873[#7873]) + - https://github.com/elastic/elasticsearch-net/pull/7875[#7875] Remove invalid properties from `SlowlogTresholds` (issue https://github.com/elastic/elasticsearch-net/issues/7865[#7865]) + - https://github.com/elastic/elasticsearch-net/pull/7883[#7883] Remove leading `/` character from API urls (issue: https://github.com/elastic/elasticsearch-net/issues/7878[#7878]) + +[discrete] +=== Features & Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7869[#7869] Add support for `SimpleQueryStringQuery.flags property (issue: https://github.com/elastic/elasticsearch-net/issues/7863[#7863]) \ No newline at end of file diff --git a/docs/release-notes/release-notes-8.9.3.asciidoc b/docs/release-notes/release-notes-8.9.3.asciidoc new file mode 100644 index 00000000000..efffe9685d9 --- /dev/null +++ b/docs/release-notes/release-notes-8.9.3.asciidoc @@ -0,0 +1,10 @@ +[[release-notes-8.9.3]] +== Release notes v8.9.3 + +[discrete] +=== Features & Enhancements + +- https://github.com/elastic/elasticsearch-net/pull/7894[#7894] Reintroduce suggestion feature (issue: https://github.com/elastic/elasticsearch-net/issues/7390[#7390]) +- https://github.com/elastic/elasticsearch-net/pull/7923[#7923] Add `PercentilesAggregation` and `PercentileRanksAggregation` (issue: https://github.com/elastic/elasticsearch-net/issues/7879[#7879]) +- https://github.com/elastic/elasticsearch-net/pull/7914[#7914] Update `Elastic.Transport` dependency +- https://github.com/elastic/elasticsearch-net/pull/7920[#7920] Regenerate client using the latest specification \ No newline at end of file diff --git a/docs/release-notes/release-notes.asciidoc b/docs/release-notes/release-notes.asciidoc new file mode 100644 index 00000000000..71416f69ca9 --- /dev/null +++ b/docs/release-notes/release-notes.asciidoc @@ -0,0 +1,68 @@ +[[release-notes]] += Release notes + +* <> + +[discrete] +== Version 8.11 + +* <> + +[discrete] +== Version 8.10 + +* <> + +[discrete] +== Version 8.9 + +* <> +* <> +* <> +* <> + +[discrete] +== Version 8.1 + +* <> +* <> +* <> +* <> + +[discrete] +== Version 8.0 + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +include::breaking-change-policy.asciidoc[] +include::release-notes-8.11.0.asciidoc[] +include::release-notes-8.10.0.asciidoc[] +include::release-notes-8.9.3.asciidoc[] +include::release-notes-8.9.2.asciidoc[] +include::release-notes-8.9.1.asciidoc[] +include::release-notes-8.9.0.asciidoc[] +include::release-notes-8.1.3.asciidoc[] +include::release-notes-8.1.2.asciidoc[] +include::release-notes-8.1.1.asciidoc[] +include::release-notes-8.1.0.asciidoc[] +include::release-notes-8.0.10.asciidoc[] +include::release-notes-8.0.9.asciidoc[] +include::release-notes-8.0.8.asciidoc[] +include::release-notes-8.0.7.asciidoc[] +include::release-notes-8.0.6.asciidoc[] +include::release-notes-8.0.5.asciidoc[] +include::release-notes-8.0.4.asciidoc[] +include::release-notes-8.0.3.asciidoc[] +include::release-notes-8.0.2.asciidoc[] +include::release-notes-8.0.1.asciidoc[] +include::release-notes-8.0.0.asciidoc[] \ No newline at end of file diff --git a/docs/release-notes/toc.yml b/docs/release-notes/toc.yml deleted file mode 100644 index a4100679473..00000000000 --- a/docs/release-notes/toc.yml +++ /dev/null @@ -1,5 +0,0 @@ -toc: - - file: index.md - - file: known-issues.md - - file: breaking-changes.md - - file: deprecations.md \ No newline at end of file diff --git a/docs/troubleshooting.asciidoc b/docs/troubleshooting.asciidoc new file mode 100644 index 00000000000..c30c6bac554 --- /dev/null +++ b/docs/troubleshooting.asciidoc @@ -0,0 +1,45 @@ +[[troubleshooting]] += Troubleshooting + +[partintro] +-- +The client can provide rich details about what occurred in the request pipeline during the process +of making a request, as well as be configured to provide the raw request and response JSON + +* <> + +* <> + +-- + +[[logging]] +== Logging + +Whilst developing with Elasticsearch using NEST, it can be extremely valuable to see the requests that +NEST generates and sends to Elasticsearch, as well as the responses returned. + +There are a couple of popular ways of capturing this information + +* <> + +* <> + +include::client-concepts/troubleshooting/logging-with-on-request-completed.asciidoc[] + +include::client-concepts/troubleshooting/logging-with-fiddler.asciidoc[] + +[[debugging]] +== Debugging + +When things are going awry, you want to be provided with as much information as possible, to resolve +the issue! + +Elasticsearch.Net and NEST provide an <> and <> to +help get you back on the happy path. + +include::client-concepts/troubleshooting/audit-trail.asciidoc[] + +include::client-concepts/troubleshooting/debug-information.asciidoc[] + +include::client-concepts/troubleshooting/debug-mode.asciidoc[] + diff --git a/docs/usage/aggregations.asciidoc b/docs/usage/aggregations.asciidoc deleted file mode 100644 index 1f159763aaa..00000000000 --- a/docs/usage/aggregations.asciidoc +++ /dev/null @@ -1,131 +0,0 @@ -[[aggregations]] -== Aggregation examples - -This page demonstrates how to use aggregations. - -[discrete] -=== Top-level aggreggation - -[discrete] -==== Fluent API - -[source,csharp] ----- -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .MatchAll(_ => {}) - ) - .Aggregations(aggregations => aggregations - .Add("agg_name", aggregation => aggregation - .Max(max => max - .Field(x => x.Age) - ) - ) - ) - .Size(10) - ); ----- - -[discrete] -==== Object initializer API - -[source,csharp] ----- -var response = await client.SearchAsync(new SearchRequest("persons") -{ - Query = Query.MatchAll(new MatchAllQuery()), - Aggregations = new Dictionary - { - { "agg_name", Aggregation.Max(new MaxAggregation - { - Field = Infer.Field(x => x.Age) - })} - }, - Size = 10 -}); ----- - -[discrete] -==== Consume the response - -[source,csharp] ----- -var max = response.Aggregations!.GetMax("agg_name")!; -Console.WriteLine(max.Value); ----- - -[discrete] -=== Sub-aggregation - -[discrete] -==== Fluent API - -[source,csharp] ----- -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .MatchAll(_ => {}) - ) - .Aggregations(aggregations => aggregations - .Add("firstnames", aggregation => aggregation - .Terms(terms => terms - .Field(x => x.FirstName) - ) - .Aggregations(aggregations => aggregations - .Add("avg_age", aggregation => aggregation - .Max(avg => avg - .Field(x => x.Age) - ) - ) - ) - ) - ) - .Size(10) - ); ----- - -[discrete] -==== Object initializer API - -[source,csharp] ----- -var topLevelAggregation = Aggregation.Terms(new TermsAggregation -{ - Field = Infer.Field(x => x.FirstName) -}); - -topLevelAggregation.Aggregations = new Dictionary -{ - { "avg_age", new MaxAggregation - { - Field = Infer.Field(x => x.Age) - }} -}; - -var response = await client.SearchAsync(new SearchRequest("persons") -{ - Query = Query.MatchAll(new MatchAllQuery()), - Aggregations = new Dictionary - { - { "firstnames", topLevelAggregation} - }, - Size = 10 -}); ----- - -[discrete] -==== Consume the response - -[source,csharp] ----- -var firstnames = response.Aggregations!.GetStringTerms("firstnames")!; -foreach (var bucket in firstnames.Buckets) -{ - var avg = bucket.Aggregations.GetAverage("avg_age")!; - Console.WriteLine($"The average age for persons named '{bucket.Key}' is {avg}"); -} ----- diff --git a/docs/usage/esql.asciidoc b/docs/usage/esql.asciidoc new file mode 100644 index 00000000000..7b7c1a0fe42 --- /dev/null +++ b/docs/usage/esql.asciidoc @@ -0,0 +1,69 @@ +[[esql]] +== ES|QL in the .NET client +++++ +Using ES|QL +++++ + +This page helps you understand and use {ref}/esql.html[ES|QL] in the +.NET client. + +There are two ways to use ES|QL in the .NET client: + +* Use the Elasticsearch {es-docs}/esql-apis.html[ES|QL API] directly: This +is the most flexible approach, but it's also the most complex because you must handle +results in their raw form. You can choose the precise format of results, +such as JSON, CSV, or text. +* Use ES|QL high-level helpers: These helpers take care of parsing the raw +response into something readily usable by the application. Several helpers are +available for different use cases, such as object mapping, cursor +traversal of results (in development), and dataframes (in development). + +[discrete] +[[esql-how-to]] +=== How to use the ES|QL API + +The {es-docs}/esql-query-api.html[ES|QL query API] allows you to specify how +results should be returned. You can choose a +{es-docs}/esql-rest.html#esql-rest-format[response format] such as CSV, text, or +JSON, then fine-tune it with parameters like column separators +and locale. + +The following example gets ES|QL results as CSV and parses them: + +[source,charp] +---- +var response = await client.Esql.QueryAsync(r => r + .Query("FROM index") + .Format("csv") +); +var csvContents = Encoding.UTF8.GetString(response.Data); +---- + +[discrete] +[[esql-consume-results]] +=== Consume ES|QL results + +The previous example showed that although the raw ES|QL API offers maximum +flexibility, additional work is required in order to make use of the +result data. + +To simplify things, try working with these three main representations of ES|QL +results (each with its own mapping helper): + +* **Objects**, where each row in the results is mapped to an object from your +application domain. This is similar to what ORMs (object relational mappers) +commonly do. +* **Cursors**, where you scan the results row by row and access the data using +column names. This is similar to database access libraries. +* **Dataframes**, where results are organized in a column-oriented structure that +allows efficient processing of column data. + +[source,charp] +---- +// ObjectAPI example +var response = await client.Esql.QueryAsObjectsAsync(x => x.Query("FROM index")); +foreach (var person in response) +{ + // ... +} +---- diff --git a/docs/usage/examples.asciidoc b/docs/usage/examples.asciidoc new file mode 100644 index 00000000000..501c63b89e9 --- /dev/null +++ b/docs/usage/examples.asciidoc @@ -0,0 +1,122 @@ +[[examples]] +== CRUD usage examples + +This page helps you to understand how to perform various basic {es} CRUD +(create, read, update, delete) operations using the .NET client. It demonstrates +how to create a document by indexing an object into {es}, read a document back, +retrieving it by ID or performing a search, update one of the fields in a +document and delete a specific document. + +These examples assume you have an instance of the `ElasticsearchClient` +accessible via a local variable named `client` and several using directives in +your C# file. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tags=using-directives;create-client] +---- +<1> The default constructor, assumes an unsecured {es} server is running and +exposed on 'http://localhost:9200'. See <> for examples +of connecting to secured servers and https://www.elastic.co/cloud[Elastic Cloud] +deployments. + +The examples operate on data representing tweets. Tweets are modelled in the +client application using a C# class named 'Tweet' containing several properties +that map to the document structure being stored in {es}. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=tweet-class] +---- +<1> By default, the .NET client will try to find a property called `Id` on the +class. When such a property is present it will index the document into {es} +using the ID specified by the value of this property. + + +[discrete] +[[indexing-net]] +=== Indexing a document + +Documents can be indexed by creating an instance representing a tweet and +indexing it via the client. In these examples, we will work with an index named +'my-tweet-index'. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=create-tweet] +---- +<1> Create an instance of the `Tweet` class with relevant properties set. +<2> Prefer the async APIs, which require awaiting the response. +<3> Check the `IsValid` property on the response to confirm that the request and +operation succeeded. +<4> Access the `IndexResponse` properties, such as the ID, if necessary. + +[discrete] +[[getting-net]] +=== Getting a document + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=get-tweet] +---- +<1> The `GetResponse` is mapped 1-to-1 with the Elasticsearch JSON response. +<2> The original document is deserialized as an instance of the Tweet class, +accessible on the response via the `Source` property. + + +[discrete] +[[searching-net]] +=== Searching for documents + +The client exposes a fluent interface and a powerful query DSL for searching. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=search-tweet-fluent] +---- +<1> The generic type argument specifies the `Tweet` class, which is used when +deserialising the hits from the response. +<2> The index can be omitted if a `DefaultIndex` has been configured on +`ElasticsearchClientSettings`, or a specific index was configured when mapping +this type. +<3> Execute a term query against the `user` field, searching for tweets authored +by the user 'stevejgordon'. +<4> Documents matched by the query are accessible via the `Documents` collection +property on the `SearchResponse`. + +You may prefer using the object initializer syntax for requests if lambdas +aren't your thing. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=search-tweet-object-initializer] +---- +<1> Create an instance of `SearchRequest`, setting properties to control the +search operation. +<2> Pass the request to the `SearchAsync` method on the client. + +[discrete] +[[updating-net]] +=== Updating documents + +Documents can be updated in several ways, including by providing a complete +replacement for an existing document ID. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=update-tweet] +---- +<1> Update a property on the existing tweet instance. +<2> Send the updated tweet object in the update request. + + +[discrete] +[[deleting-net]] +=== Deleting documents + +Documents can be deleted by providing the ID of the document to remove. + +[source,csharp] +---- +include::{doc-tests-src}/Usage/CrudExamplesTests.cs[tag=delete-tweet] +---- diff --git a/docs/usage/index.asciidoc b/docs/usage/index.asciidoc new file mode 100644 index 00000000000..f4ae4474730 --- /dev/null +++ b/docs/usage/index.asciidoc @@ -0,0 +1,25 @@ +[[usage]] += Using the .NET Client + +[partintro] +The sections below provide tutorials on the most frequently used and some less obvious features of {es}. + +For a full reference, see the {ref}/[Elasticsearch documentation] and in particular the {ref}/rest-apis.html[REST APIs] section. The {net-client} follows closely the JSON structures described there. + +A .NET API reference documentation for the Elasticsearch client package is available https://elastic.github.io/elasticsearch-net[here]. + +If you're new to {es}, make sure also to read {ref}/getting-started.html[Elasticsearch's quick start] that provides a good introduction. + +* <> +* <> +* <> + +NOTE: This is still a work in progress, more sections will be added in the near future. + +include::aggregations.asciidoc[] +include::esql.asciidoc[] +include::examples.asciidoc[] +include::mappings.asciidoc[] +include::query.asciidoc[] +include::recommendations.asciidoc[] +include::transport.asciidoc[] diff --git a/docs/usage/mappings.asciidoc b/docs/usage/mappings.asciidoc deleted file mode 100644 index 13d62f63147..00000000000 --- a/docs/usage/mappings.asciidoc +++ /dev/null @@ -1,34 +0,0 @@ -[[mappings]] -== Custom mapping examples - -This page demonstrates how to configure custom mappings on an index. - -[discrete] -=== Configure mappings during index creation - -[source,csharp] ----- -await client.Indices.CreateAsync(index => index - .Index("index") - .Mappings(mappings => mappings - .Properties(properties => properties - .IntegerNumber(x => x.Age!) - .Keyword(x => x.FirstName!, keyword => keyword.Index(false)) - ) - ) -); ----- - -[discrete] -=== Configure mappings after index creation - -[source,csharp] ----- -await client.Indices.PutMappingAsync(mappings => mappings - .Indices("index") - .Properties(properties => properties - .IntegerNumber(x => x.Age!) - .Keyword(x => x.FirstName!, keyword => keyword.Index(false)) - ) -); ----- diff --git a/docs/usage/query.asciidoc b/docs/usage/query.asciidoc deleted file mode 100644 index b365825cdbd..00000000000 --- a/docs/usage/query.asciidoc +++ /dev/null @@ -1,50 +0,0 @@ -[[query]] -== Query examples - -This page demonstrates how to perform a search request. - -[discrete] -=== Fluent API - -[source,csharp] ----- -var response = await client - .SearchAsync(search => search - .Index("persons") - .Query(query => query - .Term(term => term - .Field(x => x.FirstName) - .Value("Florian") - ) - ) - .Size(10) - ); ----- - -[discrete] -=== Object initializer API - -[source,csharp] ----- -var response = await client - .SearchAsync(new SearchRequest("persons") - { - Query = Query.Term(new TermQuery(Infer.Field(x => x.FirstName)) - { - Value = "Florian" - }), - Size = 10 - }); ----- - - -[discrete] -=== Consume the response - -[source,csharp] ----- -foreach (var person in response.Documents) -{ - Console.WriteLine(person.FirstName); -} ----- diff --git a/docs/usage/recommendations.asciidoc b/docs/usage/recommendations.asciidoc new file mode 100644 index 00000000000..b7f02a3589c --- /dev/null +++ b/docs/usage/recommendations.asciidoc @@ -0,0 +1,37 @@ +[[recommendations]] +== Usage recommendations + +To achieve the most efficient use of the {net-client}, we recommend following +the guidance defined in this article. + +[discrete] +=== Reuse the same client instance + +When working with the {net-client} we recommend that consumers reuse a single +instance of `ElasticsearchClient` for the entire lifetime of the application. +When reusing the same instance: + +- initialization overhead is limited to the first usage. +- resources such as TCP connections can be pooled and reused to improve +efficiency. +- serialization overhead is reduced, improving performance. + +The `ElasticsearchClient` type is thread-safe and can be shared and reused +across multiple threads in consuming applications. Client reuse can be achieved +by creating a singleton static instance or by registering the type with a +singleton lifetime when using dependency injection containers. + +[discrete] +=== Prefer asynchronous methods + +The {net-client} exposes synchronous and asynchronous methods on the +`ElasticsearchClient`. We recommend always preferring the asynchronous methods, +which have the `Async` suffix. Using the {net-client} requires sending HTTP +requests to {es} servers. Access to {es} is sometimes slow or delayed, and some +complex queries may take several seconds to return. If such operations are +blocked by calling the synchronous methods, the thread must wait until the HTTP +request is complete. In high-load scenarios, this can cause significant thread +usage, potentially affecting the throughput and performance of consuming +applications. By preferring the asynchronous methods, application threads can +continue with other work that doesn't depend on the web resource until the +potentially blocking task completes. \ No newline at end of file diff --git a/docs/reference/transport.md b/docs/usage/transport.asciidoc similarity index 79% rename from docs/reference/transport.md rename to docs/usage/transport.asciidoc index 8af799958ab..3e15fbd0b90 100644 --- a/docs/reference/transport.md +++ b/docs/usage/transport.asciidoc @@ -1,13 +1,10 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/transport.html ---- - -# Transport example [transport] +[[transport]] +== Transport example This page demonstrates how to use the low level transport to send requests. -```csharp +[source,csharp] +---- public class MyRequestParameters : RequestParameters { public bool Pretty @@ -22,7 +19,7 @@ public class MyRequestParameters : RequestParameters var body = """ { "name": "my-api-key", - "expiration": "1d", + "expiration": "1d", "...": "..." } """; @@ -36,7 +33,7 @@ var pathAndQuery = requestParameters.CreatePathWithQueryStrings("/_security/api_ client.ElasticsearchClientSettings); var endpointPath = new EndpointPath(Elastic.Transport.HttpMethod.POST, pathAndQuery); -// Or, if the path does not contain query parameters: +// Or, if the path does not contain query parameters: // new EndpointPath(Elastic.Transport.HttpMethod.POST, "my_path") var response = await client.Transport @@ -47,5 +44,4 @@ var response = await client.Transport null, cancellationToken: default) .ConfigureAwait(false); -``` - +----