88
99A key (str) can look like:
1010
11- - 0123456789abcdef... (usually a long, hex-encoded hash value)
12- - Any other pure ASCII string without "/" or ".." or " " .
11+ - 0123456789abcdef... (usually a long, hex-encoded hash value)
12+ - Any other pure ASCII string without '/', '..', or spaces .
1313
1414
1515Namespaces
1616----------
1717
18- To keep stuff apart , keys should get prefixed with a namespace, like :
18+ To keep things separate , keys should be prefixed with a namespace, such as :
1919
2020- config/settings
2121- meta/0123456789abcdef...
2222- data/0123456789abcdef...
2323
2424Please note:
2525
26- 1. you should always use namespaces.
27- 2. nested namespaces like namespace1/namespace2/key are not supported.
28- 3. the code could work without a namespace (namespace ""), but then you
29- can't add another namespace later, because then you would have created
26+ 1. You should always use namespaces.
27+ 2. Nested namespaces like namespace1/namespace2/key are not supported.
28+ 3. The code can work without a namespace (empty namespace ""), but then you
29+ can't add another namespace later, because that would create
3030 nested namespaces.
3131
3232Values
@@ -38,20 +38,20 @@ Store Operations
3838----------------
3939
4040The high-level Store API implementation transparently deals with nesting and
41- soft deletion, so the caller doesn't have to care much for that and the Backend
41+ soft deletion, so the caller doesn't have to care much about that, and the backend
4242API can be much simpler:
4343
4444- create/destroy: initialize or remove the whole store.
45- - list: flat list of the items in the given namespace (by default only not
46- soft deleted items, optionally only soft deleted items).
47- - store: write a new item into the store (giving its key/value pair)
48- - load: read a value from the store (giving its key), partial loads giving
49- offset and/or size are supported.
50- - info: get information about an item via its key (exists? size? ...)
51- - delete: immediately remove an item from the store (giving its key)
52- - move: implements rename, soft delete / undelete, move to current
53- nesting level
54- - stats: api call counters, time spent in api methods, data volume/throughput
45+ - list: flat list of the items in the given namespace (by default, only non-deleted
46+ items; optionally, only soft- deleted items).
47+ - store: write a new item into the store (providing its key/value pair).
48+ - load: read a value from the store (given its key); partial loads specifying
49+ an offset and/or size are supported.
50+ - info: get information about an item via its key (exists, size, ...).
51+ - delete: immediately remove an item from the store (given its key).
52+ - move: implements rename, soft delete/ undelete, and moving to the current
53+ nesting level.
54+ - stats: API call counters, time spent in API methods, data volume/throughput.
5555- latency/bandwidth emulator: can emulate higher latency (via BORGSTORE_LATENCY
5656 [us]) and lower bandwidth (via BORGSTORE_BANDWIDTH [bit/s]) than what is
5757 actually provided by the backend.
@@ -61,15 +61,14 @@ Store operations (and per-op timing and volume) are logged at DEBUG log level.
6161Automatic Nesting
6262-----------------
6363
64- For the Store user, items have names like e.g. :
64+ For the Store user, items have names such as :
6565
6666- namespace/0123456789abcdef...
6767- namespace/abcdef0123456789...
6868
6969If there are very many items in the namespace, this could lead to scalability
70- issues in the backend, thus the Store implementation offers transparent
71- nesting, so that internally the Backend API will be called with
72- names like e.g.:
70+ issues in the backend. The Store implementation therefore offers transparent
71+ nesting, so that internally the backend API is called with names such as:
7372
7473- namespace/01/23/56/0123456789abcdef...
7574- namespace/ab/cd/ef/abcdef0123456789...
@@ -80,68 +79,68 @@ there can be different nesting configurations depending on the namespace.
8079The Store supports operating at different nesting levels in the same
8180namespace at the same time.
8281
83- When using nesting depth > 0, the backends will assume that keys are hashes
84- (have hex digits) because some backends will want to pre-create the nesting
85- directories at backend initialization time to optimize for better performance
86- while using the backend.
82+ When using nesting depth > 0, the backends assume that keys are hashes
83+ (contain hex digits) because some backends pre-create the nesting
84+ directories at initialization time to optimize performance while using the backend.
8785
8886Soft deletion
8987-------------
9088
91- To soft delete an item (so its value could be still read or it could be
89+ To soft delete an item (so its value can still be read or it can be
9290undeleted), the store just renames the item, appending ".del" to its name.
9391
9492Undelete reverses this by removing the ".del" suffix from the name.
9593
9694Some store operations have a boolean flag "deleted" to choose whether they
97- shall consider soft deleted items.
95+ should consider soft- deleted items.
9896
9997Backends
10098--------
10199
102- The backend API is rather simple, one only needs to provide some very
100+ The backend API is rather simple; one only needs to provide some very
103101basic operations.
104102
105- Existing backends are listed below, more might come in future.
103+ Existing backends are listed below; more might come in the future.
106104
107105posixfs
108106~~~~~~~
109107
110108Use storage on a local POSIX filesystem:
111109
112110- URL: ``file:///absolute/path ``
113- - it is the caller's task to create an absolute fs path from a relative one.
114- - namespaces: directories
115- - values: in key-named files
111+ - It is the caller's responsibility to convert a relative path into an absolute
112+ filesystem path.
113+ - Namespaces: directories
114+ - Values: in key-named files
116115
117116sftp
118117~~~~
119118
120- Use storage on a sftp server:
119+ Use storage on an SFTP server:
121120
122121- URL: ``sftp://user@server:port/relative/path `` (strongly recommended)
123122
124- For user's and admin's convenience, mapping the URL path to the server fs path
123+ For users' and admins' convenience, mapping the URL path to the server filesystem path
125124 depends on the server configuration (home directory, sshd/sftpd config, ...).
126125 Usually the path is relative to the user's home directory.
127126- URL: ``sftp://user@server:port//absolute/path ``
128127
129- As this uses an absolute path, things are more difficult here :
128+ As this uses an absolute path, some things are more difficult:
130129
131- - user's config might break if server admin moves a user home to a new location.
132- - users must know the full absolute path of space they have permission to use.
133- - namespaces : directories
134- - values : in key-named files
130+ - A user's configuration might break if a server admin moves a user's home to a new location.
131+ - Users must know the full absolute path of the space they have permission to use.
132+ - Namespaces : directories
133+ - Values : in key-named files
135134
136135rclone
137136~~~~~~
138137
139138Use storage on any of the many cloud providers `rclone <https://rclone.org/ >`_ supports:
140139
141- - URL: ``rclone:remote:path ``, we just prefix "rclone:" and give all to the right
142- of that to rclone, see: https://rclone.org/docs/#syntax-of-remote-paths
143- - implementation of this primarily depends on the specific remote.
144- - rclone binary path can be set via the environment variable RCLONE_BINARY (default value : "rclone")
140+ - URL: ``rclone:remote:path `` — we just prefix "rclone:" and pass everything to the right
141+ of that to rclone; see: https://rclone.org/docs/#syntax-of-remote-paths
142+ - The implementation primarily depends on the specific remote.
143+ - The rclone binary path can be set via the environment variable `` RCLONE_BINARY `` (default: "rclone").
145144
146145
147146s3
@@ -165,25 +164,25 @@ Use storage on an S3-compliant cloud service:
165164 .. note ::
166165
167166 There is a known issue with some S3-compatible services (e.g., **Backblaze B2 **).
168- If you encounter problems, try using ``b2: `` instead of ``s3: `` in the url .
167+ If you encounter problems, try using ``b2: `` instead of ``s3: `` in the URL .
169168
170- - namespaces : directories
171- - values : in key-named files
169+ - Namespaces : directories
170+ - Values : in key-named files
172171
173172
174173Scalability
175174-----------
176175
177176- Count of key/value pairs stored in a namespace: automatic nesting is
178177 provided for keys to address common scalability issues.
179- - Key size: there are no special provisions for extremely long keys (like:
180- more than backend limitations). Usually this is not a problem though.
178+ - Key size: there are no special provisions for extremely long keys (e.g.,
179+ exceeding backend limitations). Usually this is not a problem, though.
181180- Value size: there are no special provisions for dealing with large value
182- sizes (like: more than free memory, more than backend storage limitations,
181+ sizes (e.g., more than available memory, more than backend storage limitations,
183182 etc.). If one deals with very large values, one usually cuts them into
184- chunks before storing them into the store.
183+ chunks before storing them in the store.
185184- Partial loads improve performance by avoiding a full load if only a part
186- of the value is needed (e.g. a header with metadata).
185+ of the value is needed (e.g., a header with metadata).
187186
188187Installation
189188------------
@@ -195,42 +194,42 @@ Install without the ``sftp:`` or ``s3:`` backend::
195194
196195Install with the ``sftp: `` backend (more dependencies)::
197196
198- pip install "borgstore[sftp]"
197+ pip install "borgstore[sftp]"
199198
200199Install with the ``s3: `` backend (more dependencies)::
201200
202- pip install "borgstore[s3]"
201+ pip install "borgstore[s3]"
203202
204- Please note that ``rclone: `` also supports sftp or s3 remotes.
203+ Please note that ``rclone: `` also supports SFTP or S3 remotes.
205204
206205Want a demo?
207206------------
208207
209- Run this to get instructions how to run the demo:
208+ Run this to get instructions on how to run the demo: :
210209
211- python3 -m borgstore
210+ python3 -m borgstore
212211
213212State of this project
214213---------------------
215214
216215**API is still unstable and expected to change as development goes on. **
217216
218217**As long as the API is unstable, there will be no data migration tools,
219- like e.g. for upgrading an existing store's data to a new release. **
218+ such as tools for upgrading an existing store's data to a new release. **
220219
221- There are tests and they succeed for the basic functionality, so some of the
222- stuff is already working well.
220+ There are tests, and they pass for the basic functionality, so some of the
221+ functionality is already working well.
223222
224- There might be missing features or optimization potential, feedback is welcome!
223+ There might be missing features or optimization potential. Feedback is welcome!
225224
226- There are a lot of possible, but still missing backends . If you want to create
227- and support one: pull requests are welcome.
225+ There are many possible backends that are still missing. If you want to create
226+ and support one, pull requests are welcome.
228227
229228Borg?
230229-----
231230
232231Please note that this code is currently **not ** used by the stable release of
233- BorgBackup (aka "borg"), but only by borg2 beta 10+ and master branch.
232+ BorgBackup (also known as "borg"), but only by Borg 2 beta 10+ and the master branch.
234233
235234License
236235-------
0 commit comments