Selective binary by OS and Node version 🤔

[rtritto]

After uWebSockets.js is installed, all OS and Node versions are built.

Reproduction

• yarn init -y
• yarn set version berry
• yarn add uWebSockets.js@github:uNetworking/uWebSockets.js#v20.49.0
• open directory /.yarn/unplugged/uWebSockets.js-https-<HASH>/node_modules/uWebSockets.js
• .node files are 24 (uws_darwin_*, uws_linux_*, uws_win32_*)

Expected

In directory /.yarn/unplugged/uWebSockets.js-https-<HASH>/node_modules/uWebSockets.js there is only 1 file filtered by OS and Node version

[uNetworkingAB]

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do. So it's a tradeoff between being easily downloaded with npm client and saving disk space.

[rtritto]

So the issue is both npm and yarn.
I opened an issue to Yarn Berry yarnpkg/berry#6565

[rtritto]

@uNetworkingAB

yarnpkg/berry#6565 (comment)

It's not an issue in Yarn Berry or any other package manager. The issue is how uNetworking/uWebSockets.js is distributed. They don't use os and cpu fields of package.json and instead distribute all platform specific files in a single package:
https://github.com/uNetworking/uWebSockets.js/tree/v20.49.0

[larixer]

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do.

The platform can be communicated to npm client via os and cpu fields in package.json. The JavaScript package managers should also support installing packages from GitHub subfolders. Combining these two features might work in your case to decrease network bandwidth during package installation at a price of some complication of releasing process on GitHub. E.g. you can keep binaries in some branch, but use multiple subfolders, like native-linux-x64, native-darwin-x64, etc, each subfolder having package.json with os and cpu fields for the target platform. Platform-agnostic and user-facing package can be kept in some other branch and should be tagged on each release as you do it now. User-facing package can then have all binary packages corresponding to its version as dependencies or optionalDependencies. Will this work well with all JavaScript package managers or not is hard to tell, in theory - it should, I am not aware of the package that do it already, so, YMMV and it's an open question if additional complexity for distribution is worth it.

[uNetworkingAB]

Very interesting! Do you think it will work with the same user experience?

npm install uNetworking/uWebSockets.js#v20.48.0

and

require("uWebSockets.js")

?

[larixer]

@uNetworkingAB Yes, it should work with the same user user experience.

[porsager]

It will still download the various abi versions, but that's still an improvement! I can make a quick sample setup for you to see

[porsager]

In my projects I'm just using this loader and a postinstall to only download a single binary, so that's an option too:

import fs from 'node:fs'
import path from 'node:path'
import https from 'node:https'
import { pipeline } from 'node:stream/promises'
import { createRequire } from 'node:module'

const binaryName = `uws_${ process.platform }_${ process.arch }_${ process.versions.modules }.node`
const binaryPath = path.join(import.meta.dirname, binaryName)

fs.existsSync(binaryPath) || await download()

let uws
try {
  uws = createRequire(import.meta.url)(binaryPath)
} catch(e) {
  await download()
  uws = createRequire(import.meta.url)(binaryPath)
}

export default uws

async function download(url = 'https://raw.githubusercontent.com/uNetworking/uWebSockets.js/v20.47.0/' + binaryName, retries = 0) {
  return new Promise((resolve, reject) => {
    https.get(url, async res => {
      if (retries > 10)
        return reject(new Error('Could not download uWebSockets binary - too many redirects - latest: ' + res.headers.location))

      if (res.statusCode === 302)
        return (res.destroy(), resolve(download(res.headers.location, retries + 1)))

      if (res.statusCode !== 200)
        return reject(new Error('Could not download uWebSockets binary - error code: ' + res.statusCode + ' - ' + url))

      pipeline(res, fs.createWriteStream(binaryPath)).then(resolve, reject)
    })
    .on('error', reject)
    .end()
  })
}

If the postinstall doesn't run, it'll download it on first run after instead.

[larixer]

@porsager @uNetworkingAB I have put together a demo repo:
https://github.com/larixer/multi-platform

You can check how it works via:
npm init
npm add larixer/multi-platform#1.0.0

You can then go into node_modules and see that user-facing multi-platform and one for your current platform package is installed, provided, if you are on Linux, Win32 or Mac.

Note, that I have used 4 branches, 1 for user-facing package main, and 3 other for linux-x64, darwin-x64 and win32-x64, because in fact support for Git subfolders is not that great in JS package managers, but they support tags and branches and this should be enough.