Introduce file-based database export and import (#758)

## Usage and product changes
Introduce interfaces to export databases into schema definition and data
files and to import databases using these files. Database import
supports files exported from both TypeDB 2.x and TypeDB 3.x.

Both operations are blocking and may take a significant amount of time
to execute for large databases. Use parallel connections to continue
operating with the server and its other databases.

Usage examples in Rust:
```rust
// export
let db = driver.databases().get(db_name).await.unwrap();
db.export_to_file(schema_file_path, data_file_path).await.unwrap();

// import
let schema = read_to_string(schema_file_path).unwrap();
driver.databases().import_from_file(db_name2, schema, data_file_path).await.unwrap();
```

Usage examples in Python:
```py
# export
database = driver.databases.get(db_name)
database.export_to_file(schema_file_path, data_file_path)

# import
with open(schema_file_path, 'r', encoding='utf-8') as f:
    schema = f.read()
driver.databases.import_from_file(db_name2, schema, data_file_path)
```

Usage examples in Java:
```java
// export
Database database = driver.databases().get(dbName);
database.exportToFile(schemaFilePath, dataFilePath);

// import
String schema = Files.readString(Path.of(schemaFilePath));
driver.databases().importFromFile(dbName2, schema, dataFilePath);
```

## Implementation
Implemented the updated
[protocol](typedb/typedb-protocol#224).

As both operations work with streaming, the implementation is similar to
transactions. The behavior is split into the file processing logic and
networking (specialized for sync and async modes). The exposed
interfaces present only the file-based versions, but additional
interfaces for direct work with streams can be presented in future
updates.

In Rust, paths are accepted as Rust `Path`s. In other languages working
through C interfaces, it's pure strings for C layer transmission.

### Database export 
Implemented through the `database` interface and accepts two target
files for export. Does not require a specific format of naming (so it's
necessarily `.typeql` or `.typedb`. If any of the target files already
exist, an error is returned.

The export operation consists of these steps:
* prepare the output files
* open a unidirectional GRPC stream from the server to the client
* "block" on server response listening until an error or a "done" is
received (blocking is implemented through a loop which resolves a
`listen` promise presented by the network layer)
* if there is a schema message, write it to the schema file and flush it
right away
* if there is a data items message, encode the items and write them to
the data file
* in case of an error, the output files are deleted (we own them as we
create them at the beginning)

The network layer is basically just a task listening for the GRPC stream
and transmitting the converted messages to the processing loop.

### Database import
Implemented through the `database_manager` interface and accepts a
database name, a schema definition query string (can be read from the
exported file), and an exported data file. No naming requirements as
well.

The import operation consists of these steps:
* open the input file
* open a bidirectional GRPC stream between the server and the client,
send the initial request with the database's name and schema
* eagerly start reading and decoding data items from the input file one
by one, storing up to 250 items in the buffer (this number can be easily
changed)
* once the buffer is full, attempt items sending operation: this
operation will check a potential early error signal from the server and
then send the batch, returning to processing the rest of the file
* once the file is read, send a "done" message and block until the
server responds with either an error or its "done" message

The network layer consists of a blocking task for client-side requests
and a listening task waiting for a one-shot signal from the server
(either an error or a "done" message). Errors can be received at any
time of processing, while "done" is expected only after a client-side
"done" request. When a response is received, either an async or a sync
sink receives this message, which should be checked before any
client-side network operations to ensure proper interruption.

Author: farost

.factory/automation.yml

@@ -80,11 +80,17 @@ build:
         find "${DOCS_DIRS[@]}" -type f ! -name 'api-reference.adoc' -exec rm -f {} \;
         tool/docs/update.sh
         git add "${DOCS_DIRS[@]}"
-        git diff --exit-code HEAD "${DOCS_DIRS[@]}" || echo "Failed to verify docs files: please update it manually and verify the changes"
+        git diff --exit-code HEAD "${DOCS_DIRS[@]}" || { 
+          echo "Failed to verify docs files: please update it manually and verify the changes"
+          exit 1
+        }
         
         tool/docs/update_readme.sh
         git add .
-        git diff --exit-code || echo "Failed to verify README files: please update it manually and verify the changes"
+        git diff --exit-code || { 
+          echo "Failed to verify README files: plese update it manually and verify the changes"
+          exit 1
+        }
 
     test-rust-unit-integration:
       image: typedb-ubuntu-20.04 # Ubuntu 20.04 has GLIBC version 2.31 (2020) which we should verify to compile against

Cargo.lock

@@ -31,7 +31,7 @@ features = {}
 
 	[dependencies.chrono]
 		features = ["alloc", "android-tzdata", "clock", "default", "iana-time-zone", "js-sys", "now", "oldtime", "serde", "std", "wasm-bindgen", "wasmbind", "winapi", "windows-link"]
-		version = "0.4.40"
+		version = "0.4.41"
 		default-features = false
 
 	[dependencies.itertools]

c/Cargo.toml

@@ -19,14 +19,13 @@
 
 use std::{ffi::c_char, path::Path};
 
-use itertools::Itertools;
 use typedb_driver::{Credentials, DriverOptions, TypeDBDriver};
 
 use super::{
     error::{try_release, unwrap_void},
     memory::{borrow, free, string_view},
 };
-use crate::memory::{release, string_array_view};
+use crate::memory::release;
 
 const DRIVER_LANG: &'static str = "c";
 

c/src/connection.rs

@@ -17,7 +17,7 @@
  * under the License.
  */
 
-use std::{ffi::c_char, ptr::addr_of_mut, sync::Arc};
+use std::{ffi::c_char, path::Path, ptr::addr_of_mut, sync::Arc};
 
 use typedb_driver::{box_stream, info::ReplicaInfo, Database};
 
@@ -26,7 +26,7 @@ use super::{
     iterator::{iterator_next, CIterator},
     memory::{borrow, borrow_mut, free, release, release_optional, release_string, take_ownership},
 };
-use crate::memory::{decrement_arc, take_arc};
+use crate::memory::{decrement_arc, string_view, take_arc};
 
 /// Frees the native rust <code>Database</code> object
 #[no_mangle]
@@ -58,6 +58,23 @@ pub extern "C" fn database_type_schema(database: *const Database) -> *mut c_char
     try_release_string(borrow(database).type_schema())
 }
 
+/// Export a database into a schema definition and a data files saved to the disk.
+/// This is a blocking operation and may take a significant amount of time depending on the database size.
+///
+/// @param database The <code>Database</code> object to export from.
+/// @param schema_file The path to the schema definition file to be created.
+/// @param data_file The path to the data file to be created.
+#[no_mangle]
+pub extern "C" fn database_export_to_file(
+    database: *const Database,
+    schema_file: *const c_char,
+    data_file: *const c_char,
+) {
+    let schema_file_path = Path::new(string_view(schema_file));
+    let data_file_path = Path::new(string_view(data_file));
+    unwrap_void(borrow(database).export_to_file(schema_file_path, data_file_path))
+}
+
 // /// Iterator over the <code>ReplicaInfo</code> corresponding to each replica of a TypeDB cloud database.
 // pub struct ReplicaInfoIterator(CIterator<ReplicaInfo>);
 //

c/src/database.rs

@@ -17,7 +17,7 @@
  * under the License.
  */
 
-use std::{ffi::c_char, ptr::addr_of_mut, sync::Arc};
+use std::{ffi::c_char, path::Path, ptr::addr_of_mut, sync::Arc};
 
 use typedb_driver::{box_stream, Database, TypeDBDriver};
 
@@ -28,7 +28,7 @@ use super::{
 };
 use crate::{error::try_release_arc, iterator::iterator_arc_next};
 
-/// An <code>Iterator</code> over databases present on the TypeDB server
+/// An <code>Iterator</code> over databases present on the TypeDB server.
 pub struct DatabaseIterator(CIterator<Arc<Database>>);
 
 /// Forwards the <code>DatabaseIterator</code> and returns the next <code>Database</code> if it exists,
@@ -38,27 +38,47 @@ pub extern "C" fn database_iterator_next(it: *mut DatabaseIterator) -> *const Da
     unsafe { iterator_arc_next(addr_of_mut!((*it).0)) }
 }
 
-/// Frees the native rust <code>DatabaseIterator</code> object
+/// Frees the native rust <code>DatabaseIterator</code> object.
 #[no_mangle]
 pub extern "C" fn database_iterator_drop(it: *mut DatabaseIterator) {
     free(it);
 }
 
-/// Returns a <code>DatabaseIterator</code> over all databases present on the TypeDB server
+/// Returns a <code>DatabaseIterator</code> over all databases present on the TypeDB server.
 #[no_mangle]
 pub extern "C" fn databases_all(driver: *mut TypeDBDriver) -> *mut DatabaseIterator {
     try_release(
         borrow_mut(driver).databases().all().map(|dbs| DatabaseIterator(CIterator(box_stream(dbs.into_iter())))),
     )
 }
 
-/// Create a database with the given name
+/// Create a database with the given name.
 #[no_mangle]
 pub extern "C" fn databases_create(driver: *mut TypeDBDriver, name: *const c_char) {
     unwrap_void(borrow_mut(driver).databases().create(string_view(name)));
 }
 
-/// Checks if a database with the given name exists
+/// Create a database with the given name based on previously exported another database's data
+/// loaded from a file.
+/// This is a blocking operation and may take a significant amount of time depending on the database
+/// size.
+///
+/// @param driver The <code>TypeDBDriver</code> object.
+/// @param name The name of the database to be created.
+/// @param schema The schema definition query string for the database.
+/// @param data_file The exported database file path to import the data from.
+#[no_mangle]
+pub extern "C" fn databases_import_from_file(
+    driver: *mut TypeDBDriver,
+    name: *const c_char,
+    schema: *const c_char,
+    data_file: *const c_char,
+) {
+    let data_file_path = Path::new(string_view(data_file));
+    unwrap_void(borrow_mut(driver).databases().import_from_file(string_view(name), string_view(schema), data_file_path))
+}
+
+/// Checks if a database with the given name exists.
 #[no_mangle]
 pub extern "C" fn databases_contains(driver: *mut TypeDBDriver, name: *const c_char) -> bool {
     unwrap_or_default(borrow_mut(driver).databases().contains(string_view(name)))

c/src/database_manager.rs

@@ -25,7 +25,7 @@ def typedb_artifact():
         artifact_name = "typedb-all-{platform}-{version}.{ext}",
         tag_source = deployment["artifact"]["release"]["download"],
         commit_source = deployment["artifact"]["snapshot"]["download"],
-        commit = "d83f3b2b40c673c30d00b377ce327ac0ff233056"
+        commit = "a45d7b0003bb95e7b36ab097be468acf2398991b"
     )
 
 #def typedb_cloud_artifact():

dependencies/typedb/artifacts.bzl

@@ -21,19 +21,19 @@ def typedb_dependencies():
     git_repository(
         name = "typedb_dependencies",
         remote = "https://github.com/typedb/typedb-dependencies",
-        commit = "ab777bf067b1930e35146fd8e25a76a4a360aa74",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_dependencies
+        commit = "4ffeaabde31c41cee271cbb563f17168f4229a93",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_dependencies
     )
 
 def typedb_protocol():
     git_repository(
         name = "typedb_protocol",
         remote = "https://github.com/typedb/typedb-protocol",
-        tag = "3.2.0",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_protocol
+        commit = "f6528beec33d6e3c31cb5dafc30e8f0f097c3f82",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_protocol
     )
 
 def typedb_behaviour():
     git_repository(
         name = "typedb_behaviour",
         remote = "https://github.com/typedb/typedb-behaviour",
-        commit = "8f9345de853ad7d0ae66e7afefd16be2cfa3dced",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_behaviour
+        commit = "65258a4a3ad80be5918f33b74b1b08e25ee6fd7b",  # sync-marker: do not remove this comment, this is used for sync-dependencies by @typedb_behaviour
     )

dependencies/typedb/repositories.bzl

@@ -27,6 +27,40 @@ Deletes this database.
 database.delete()
 ----
 
+[#_Database_exportToFile_java_lang_String_java_lang_String]
+==== exportToFile
+
+[source,java]
+----
+void exportToFile​(java.lang.String schemaFilePath,
+                  java.lang.String dataFilePath)
+           throws TypeDBDriverException
+----
+
+Export a database into a schema definition and a data files saved to the disk. This is a blocking operation and may take a significant amount of time depending on the database size. 
+
+
+[caption=""]
+.Input parameters
+[cols=",,"]
+[options="header"]
+|===
+|Name |Description |Type
+a| `schemaFilePath` a| The path to the schema definition file to be created a| `java.lang.String`
+a| `dataFilePath` a| The path to the data file to be created a| `java.lang.String`
+|===
+
+[caption=""]
+.Returns
+`void`
+
+[caption=""]
+.Code examples
+[source,java]
+----
+database.exportToFile("schema.typeql", "data.typedb")
+----
+
 [#_Database_name_]
 ==== name
 

docs/modules/ROOT/partials/java/connection/Database.adoc

@@ -16,7 +16,7 @@ java.util.List<Database> all()
                       throws TypeDBDriverException
 ----
 
-Retrieves all databases present on the TypeDB server 
+Retrieves all databases present on the TypeDB server. 
 
 
 [caption=""]
@@ -40,7 +40,7 @@ boolean contains​(java.lang.String name)
           throws TypeDBDriverException
 ----
 
-Checks if a database with the given name exists 
+Checks if a database with the given name exists. 
 
 
 [caption=""]
@@ -72,7 +72,7 @@ void create​(java.lang.String name)
      throws TypeDBDriverException
 ----
 
-Create a database with the given name 
+Create a database with the given name. 
 
 
 [caption=""]
@@ -128,5 +128,41 @@ a| `name` a| The name of the database to retrieve a| `java.lang.String`
 driver.databases().get(name)
 ----
 
+[#_DatabaseManager_importFromFile_java_lang_String_java_lang_String_java_lang_String]
+==== importFromFile
+
+[source,java]
+----
+void importFromFile​(java.lang.String name,
+                    java.lang.String schema,
+                    java.lang.String dataFilePath)
+             throws TypeDBDriverException
+----
+
+Create a database with the given name based on previously exported another database's data loaded from a file. This is a blocking operation and may take a significant amount of time depending on the database size. 
+
+
+[caption=""]
+.Input parameters
+[cols=",,"]
+[options="header"]
+|===
+|Name |Description |Type
+a| `name` a| The name of the database to be created a| `java.lang.String`
+a| `schema` a| The schema definition query string for the database a| `java.lang.String`
+a| `dataFilePath` a| The exported database file path to import the data from a| `java.lang.String`
+|===
+
+[caption=""]
+.Returns
+`void`
+
+[caption=""]
+.Code examples
+[source,java]
+----
+driver.databases().importFromFile(name, schema, "data.typedb")
+----
+
 // end::methods[]
 

docs/modules/ROOT/partials/java/connection/DatabaseManager.adoc

@@ -82,5 +82,59 @@ a| `includeInstanceTypes` a| Whether to include instance types in ConceptRow ans
 options.includeInstanceTypes(includeInstanceTypes);
 ----
 
+[#_QueryOptions_prefetchSize_]
+==== prefetchSize
+
+[source,java]
+----
+@CheckReturnValue
+public java.util.Optional<java.lang.Integer> prefetchSize()
+----
+
+Returns the value set for the prefetch size in this ``QueryOptions`` object. If set, specifies the number of extra query responses sent before the client side has to re-request more responses. Increasing this may increase performance for queries with a huge number of answers, as it can reduce the number of network round-trips at the cost of more resources on the server side. 
+
+
+[caption=""]
+.Returns
+`public java.util.Optional<java.lang.Integer>`
+
+[caption=""]
+.Code examples
+[source,java]
+----
+options.prefetchSize();
+----
+
+[#_QueryOptions_prefetchSize_int]
+==== prefetchSize
+
+[source,java]
+----
+public QueryOptions prefetchSize​(int prefetchSize)
+----
+
+Explicitly set the prefetch size. If set, specifies the number of extra query responses sent before the client side has to re-request more responses. Increasing this may increase performance for queries with a huge number of answers, as it can reduce the number of network round-trips at the cost of more resources on the server side. Minimal value: 1. 
+
+
+[caption=""]
+.Input parameters
+[cols=",,"]
+[options="header"]
+|===
+|Name |Description |Type
+a| `prefetchSize` a| Whether to include instance types in ConceptRow answers. a| `int`
+|===
+
+[caption=""]
+.Returns
+`public QueryOptions`
+
+[caption=""]
+.Code examples
+[source,java]
+----
+options.prefetchSize(prefetchSize);
+----
+
 // end::methods[]