wasm poc #272

seanses · 2025-04-25T17:27:43Z

This implements uploading through Xet protocol in WASM environment, and makes necessary changes to make dependent crates WASM compatible.

Uploading through Xet protocol is done in hf_xet_wasm crate;
Separate Cas Client trait definitions into upload and download functionality groups and disable download for WASM;
Disable Cas Client request retry in WASM environment, which isn't critical for a POC (until we have a retry strategy that doesn't depends on time);
Disable async CasObject deserialization;
Enable in-memory global dedup;

wasm_xet/web/wasm_xet_bg.wasm.d.ts

This PR makes the cas_client crate compile into WASM (at least with wasm-pack). It separates out download & upload traits and functions when necessary to then target_family block the downloads path (easy way out). There's other generally necessary changes particularly having to do with async_trait that are added. --------- Co-authored-by: seanses <[email protected]>

Adds a CI build test on hf_xet_wasm.

rajatarya

(Partially completed, will continue with latest.)

cas_client/src/error.rs

rajatarya · 2025-06-13T00:26:43Z

cas_client/src/http_client.rs


 /// Builds authenticated HTTP Client to talk to CAS.
 /// Includes retry middleware with exponential backoff.
+#[allow(unused_variables)]


What is considered unused? Can this warning be removed?

I'll remove these. It only gives this error when linting in webassembly where the retry config arg is not used.

cas_client/src/http_client.rs

cas_client/src/interface.rs

rajatarya

Overall this is great, thank you @seanses and @assafvayner for putting this together!

The vast majority of my comments are asking for more documentation/comments/explanation. It will be hard for future you or anyone new to the repo to remember the context under which all the WASM changes were made, and that makes maintaining them without documentation risky. The more comments now the better.

I think with ~60m of effort to add comments to this PR it will be ready to merge.

Again, great work on this - really appreciate the effort.

cas_client/src/http_client.rs

rajatarya · 2025-06-16T21:54:34Z

cas_client/src/http_client.rs

 pub struct SessionMiddleware(String);

-#[async_trait::async_trait]
+#[cfg_attr(not(target_family = "wasm"), async_trait::async_trait)]


A comment explaining the compilation flag changes would be helpful here - they both look so similiar but I imagine have very different behaviors.

this is explained in hf_xet_wasm README.md; there's too many occurrences in other places to state it everywhere.

My main concern is that the wasm crate isn't something most xet-core maintainers will need to read/understand. So it might make sense to repeat or reference the message in wasm readme in a comment, something like: "# see hf_xet_wasm README.md for context

cas_client/src/http_client.rs

rajatarya · 2025-06-20T18:25:35Z

hf_xet_wasm/README.md

+# hf_xet_wasm: xet-core for WebAssembly
+
+This crate enables functionality to use the xet upload protocol from the browser with the use of a wasm based binary replicating the functionality of the `hf_xet` python library.
+Functionality included but not limited to chunking, global deduplication, xorb formation, xorb upload, shard formation, shard upload.


I would rephrase this to be explicit about what is included and what is missing - so future maintainers can understand what to expect from the rest of the repo. Something like:

hf_xet_wasm has: chunking, global deduplication, xorb formation, xorb upload, shard formation, shard upload

hf_xet_wasm is missing: complete download support (xorbs, shards, chunk caching)

hf_xet_wasm/src/wasm_file_cleaner.rs

rajatarya · 2025-06-20T19:19:27Z

hf_xet_wasm/src/wasm_file_upload_session.rs

+
+    pub async fn finalize(self: Arc<Self>) -> Result<()> {
+        // Register the remaining xorbs for upload.
+        let data_agg = take(&mut *self.current_session_data.lock().await);


How is take() different from just self.current_session_data.lock().await;? In other places in this impl the lock is received with a .lock().await - how come this code does a take() on top of that?

https://doc.rust-lang.org/std/mem/fn.take.html

take replaces the value being referenced with a default, more importantly moving out the value referenced to use in the current scope.

hf_xet_wasm/src/wasm_file_upload_session.rs

rajatarya · 2025-06-20T19:21:46Z

hf_xet_wasm/src/wasm_timer.rs

+
+static PROFILING: bool = false;
+
+pub struct Timer {


nit: I'd rename this to LoggingTimer or ConsoleLogTimer because nothing in the implementation (and no comments) indicate that it is a helper object to time operations in JS and report them to the console.

rajatarya · 2025-06-20T19:24:35Z

hf_xet_wasm/src/xorb_uploader.rs

+}
+
+impl XorbUploaderLocalSequential {
+    #[allow(dead_code)]


why need this?

was used for linting, removing the macro it and letting the warning stick

rajatarya

Love the changes, let's merge!

seanses and others added 4 commits April 25, 2025 10:08

make dedup critical crates compat with wasm

c6fc21b

wasm-bindgen-blob-async-reader

35a3e55

import blob_reader to wasm_xet

2be330e

wasm_xet skeleton

b79245d

seanses changed the base branch from main to di/wasm-compat April 25, 2025 17:30

update instructions

a0a5523

assafvayner reviewed Apr 25, 2025

View reviewed changes

wasm_xet/web/wasm_xet_bg.wasm.d.ts Outdated Show resolved Hide resolved

seanses and others added 4 commits April 25, 2025 13:32

rearrange example; fix BlobReader bug

8757749

make cas_object wasm compat

52f102c

gitignore

9edbd10

wasm_xet -> hf_xet_wasm

30bdebe

seanses force-pushed the wasm-poc branch from a3fd313 to 30bdebe Compare April 25, 2025 21:00

seanses mentioned this pull request Apr 25, 2025

Make dedup critical crates compilation-compat with wasm #271

Merged

seanses changed the base branch from di/wasm-compat to main April 28, 2025 07:07

assafvayner and others added 6 commits April 30, 2025 11:29

everything put together; something is wrong with worker getting message

bd9fbf1

Ok

ffdeed6

wasm poc ci (#289)

c58baf9

Adds a CI build test on hf_xet_wasm.

wasm poc interface (#294)

8e7d9d4

bug fixes; profiling setup

971feea

seanses force-pushed the wasm-poc branch from 5155f5d to 971feea Compare May 30, 2025 06:03

seanses added 4 commits June 2, 2025 23:48

merge main and update code

f8bb499

merge main again and update code

31c84be

fix fmt

7822ff4

delete an unnecessary test

d76b679

seanses force-pushed the wasm-poc branch from ecc0bcb to d76b679 Compare June 2, 2025 16:15

seanses marked this pull request as ready for review June 2, 2025 16:17

seanses requested a review from assafvayner June 2, 2025 16:18

seanses added 2 commits June 3, 2025 18:07

change wasm CI build

813fef5

merge main again+ and update code

e7ba324

assafvayner added 7 commits June 10, 2025 10:15

Merge branch 'main' of github.com:huggingface/xet-core into wasm-poc

881197f

update to make compile with updates

cd8837f

rm commented out code

e1b2549

rm worker.js, there's a token in it

53406a7

Merge branch 'main' of github.com:huggingface/xet-core into wasm-poc

ec661df

readme updates

d80737d

fmt

b67e2cf

rajatarya reviewed Jun 16, 2025

View reviewed changes

assafvayner added 3 commits June 16, 2025 14:58

updates

7093bc2

change format of http client creation

d871f04

update readme with token refresher info, reintro retry middleware

84047cb

rajatarya approved these changes Jun 20, 2025

View reviewed changes

assafvayner added 2 commits June 23, 2025 11:20

update based on comments

4a5e6ca

fmt

d5a1201

assafvayner approved these changes Jun 24, 2025

View reviewed changes

Merge remote-tracking branch 'origin' into wasm-poc

929aaf2

rajatarya approved these changes Jun 25, 2025

View reviewed changes

assafvayner merged commit 9fbd234 into main Jun 25, 2025
5 checks passed

assafvayner deleted the wasm-poc branch June 25, 2025 19:08

wasm poc #272

wasm poc #272

Uh oh!

Conversation

seanses commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seanses commented Apr 25, 2025 •

edited

Loading