Throwaway PR: Share hacked setup for debugging fcntl() Asyncify errors. #2242
Conversation
|
There may be something I forgot to mention, but please feel free to DM me if you have any issues. |
|
How would we know if this was a wrong way of adding an async call in the sys fcntl function? Note how our async functions are shipped via the C file and wrapped in the ES macros - different for jspi and asyncify. Any chance we're not telling Emscripten to expect stack switching on this entire fcntl code path? Would we have to patch musl? |
|
@adamziel, I believe what we are doing follows the same pattern as the JS Emscripten generates with the EM_ASYNC_JS() macro. |
|
If you look at one of those macro-wrapped functions in the generated JS, I believe they are async functions wrapped with Asyncify.handleAsync(). I might be missing something though. Is that what you see when you look at the generated JS? |
|
I am mobile but will double check in a bit. |
OK, I see how we are often wrapping them differently between EM_JS and EM_ASYNC_JS: wordpress-playground/packages/php-wasm/compile/php/php_wasm.c Lines 97 to 103 in c04e108 There is one function that we always wrap with EM_ASYNC_JS: wordpress-playground/packages/php-wasm/compile/php/php_wasm.c Lines 417 to 419 in c04e108 The docs for the macro imply that it can be used for both Asyncify and JSPI builds:
The EM_ASYNC_JS macro looks like this: #define EM_ASYNC_JS(ret, name, params, ...) _EM_JS(ret, name, __asyncjs__##name, params, \
"{ return Asyncify.handleAsync(async () => " #__VA_ARGS__ "); }")And it does three things:
In this PR, DEFAULT_ASYNCIFY_IMPORTS = ['__asyncjs__*'] |
|
There is some other Emscripten code that auto-adds JS functions to ASYNCIFY_IMPORTS: From jsifier.mjs, this code identifies async library functions: isAsyncFunction =
LibraryManager.library[symbol + '__async'] ||
original.constructor.name == 'AsyncFunction';and adds those functions to an and relays the list in this data structure. Then that list appears to be added to ASYNCIFY_IMPORTS here: settings.ASYNCIFY_IMPORTS += ['*.' + x for x in js_info['asyncFuncs']]It's no big deal either way, but I think our current declaration of __syscall_fcntl64() could be auto-added to the list because it is an async function: __syscall_fcntl64: async function __syscall_fcntl64(fd, cmd, varargs) { |
|
So, it's possible I'm wrong, but I think that the way |
|
Good investigation, thank you @brandonpayton! I've played with adding all SQLite3 exports to asyncify list but no bueno yet. There may be some private, unexported functions that we need to add, and some additional core PHP functions |
|
Still no bueno. Here's my process so far: I'm using the following PHP script to reproduce the problem: <?php
// Create (or open) a new SQLite database file
// Use PDO to connect to SQLite database
$dsn = 'sqlite:example3.db';
$db = new SQLite3($dsn);
// Create a new table
$db->exec('CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
email TEXT NOT NULL
)');
// Insert a few records
$db->exec("INSERT INTO users (name, email) VALUES ('Alice', '[email protected]')");
$db->exec("INSERT INTO users (name, email) VALUES ('Bob', '[email protected]')");
$db->exec("INSERT INTO users (name, email) VALUES ('Charlie', '[email protected]')");I run it as follows: PHP=8.3 node --stack-trace-limit=1000 --experimental-strip-types --experimental-transform-types --import ./packages/meta/src/node-es-module-loader/register.mts --watch ./packages/php-wasm/cli/src/main.ts sqlite.phpAlso, I've made the following changes to the PHP building Dockerfile: + export OPTIMIZATION_FLAGS="-O0 -g3"; \
- if [ "${WITH_SOURCEMAPS}" = "yes" ] || [ "${WITH_DEBUG}" = "yes" ]; then \
- # TODO: Revert this to -O0 after debugging.
- export OPTIMIZATION_FLAGS="-O3"; \
- fi; \
PLATFORM_SPECIFIC_ARGS=''; \
if [ "$EMSCRIPTEN_ENVIRONMENT" = "node" ]; then \
PLATFORM_SPECIFIC_ARGS="$PLATFORM_SPECIFIC_ARGS --js-library /root/phpwasm-emscripten-library-fcntl-for-node.js"; \
fi; \
emcc $OPTIMIZATION_FLAGS \
--js-library /root/phpwasm-emscripten-library.js \
--js-library /root/phpwasm-emscripten-library-known-undefined-functions.js \
$PLATFORM_SPECIFIC_ARGS \
-I . \
-I ext \
-I ext/json \
-I Zend \
-I main \
-I TSRM/ \
-I /root/lib/include \
-L/root/lib -L/root/lib/lib/ \
-D__x86_64__\
-lproxyfs.js \
$ASYNCIFY_FLAGS \
$(cat /root/.emcc-php-wasm-flags) \
-s EXPORTED_FUNCTIONS="$EXPORTED_FUNCTIONS" \
-s WASM_BIGINT=1 \
-s MIN_NODE_VERSION=200900 \
-s INITIAL_MEMORY=1024MB \
-s ALLOW_MEMORY_GROWTH=1 \
+ -s ASSERTIONS=3 \
+ -s ASYNCIFY_STACK_SIZE=1000 \I've rebuilt as follows: npm run recompile:php:node:asyncify:8.3I've extracted all the C function names from the bundled libraries using brew install --quiet universal-ctags # one-time
ctags -x --c-kinds=f your_file.cFirst, I downloaded SQLite3 amalgamation from https://www.sqlite.org/download.html and added all those functions to the list. It did not solve the problem: ctags -x --c-kinds=f ~/Downloads/sqlite-amalgamation-3490100/sqlite3.c | awk $(echo '{print "\"" $1 "\",\\\\"}') | pbcopyThen I used https://repomix.com/ to get all .c files from the pdo_sqlite PHP extension and extracted all those C functions. It also did not solve the problem. I've noticed some functions are defined via the PHP_METHOD macro, e.g. I copied sqlite3.c from https://github.com/php/php-src/blob/master/ext/sqlite3/sqlite3.c and added all its functions to the list. I've noticed this line: Aha, so SQLite3->exec() is Then I started doubting I'm correctly assessing whether all the functions are present in the Dockerfile so I wrote a small script to confirm I'm right. Functions come through stdin, information comes out through stdout. Sadly, all the functions from the stack trace were already present in the Dockerfile: import fs from 'fs';
import path from 'path';
const DockerfilePath = path.resolve(
__dirname,
'packages/php-wasm/compile/php/Dockerfile'
);
function addAsyncifyFunctionsToDockerfile(functions: string[]) {
const Dockerfile = fs.readFileSync(DockerfilePath, 'utf8') + '';
const cleanedUpFunctions = functions.map((candidate) =>
candidate
.trim()
.replace(/^"|^\s*\* php\.wasm\.|"$/g, '')
.replace('byn$fpcast-emu$', '')
);
console.log('Looking for these functions in the Dockerfile:');
console.log(cleanedUpFunctions);
const missingCandidates = cleanedUpFunctions.filter(
(candidate) => !Dockerfile.includes(`"${candidate}"`)
);
if (missingCandidates.length) {
console.log('Missing functions:');
console.log(missingCandidates.join('\n'));
} else {
console.log('All functions are present in the Dockerfile.');
}
}
let input = '';
process.stdin.setEncoding('utf8');
process.stdin.on('data', (chunk) => {
input += chunk;
});
process.stdin.on('end', () => {
const lines = input
.split('\n')
.map((line) => line.trim())
.filter((line) => line.length > 0);
addAsyncifyFunctionsToDockerfile(lines);
});Then I thought:
I cloned the php-src repo and switched to PHP-8.3 branch. I extracted all the extensions-related C functions via Still no luck! Next up I'll look into libc/ musl. |
I could not reproduce the success yet, but I've managed to trigger the problem via a different code path: This is useful because it points to a problem outside of the SQLite3 library. |
|
Aha, it worked with Okay. Your comments from yesterday gave me an idea @brandonpayton. I'll try to get a list of the automatically enlisted functions by setting |
|
This Dockerfile finally yielded a build that successfully ran my test PHP script! https://gist.github.com/adamziel/e4d85b7575787f684b7f154f45d0741e I used the list of functions produced by ASYNCIFY_ADVISE. Turns out it was missing a bunch of items such as I added the newly found functions to Dockerfile and rebuilt PHP. This time I got different hits. After repeating this a few times, I got a working build. Yay! We should automate that process for other PHP versions and SQLite3/PDO SQLite code paths. We have this commant This is the script that orchestrates it all: The Dockerfile is updated inside the specific unit test files. I know, I know, it smells bad. But it solves the problem pretty well. We'll also likely want to remove a lot of functions from that Dockerfile. But still – yay! PHP 8.3 build is 14M. I'm not sure why not 17M like 8.2 build, maybe some extensions are disabled? In either case, we can work with this from here. |
|
Looping in @mho22 for inspiration - this discussion may be helpful in the "XDebug on Asyncify" project. Let's still ship the JSPI version first. |
|
I got it to work on all PHP versions. I just pushed my changes to this branch. |
|
Amazing, @adamziel! Thank you very much. It's a huge help. |
@adamziel, here is my current debug setup.
To test, I am effectively doing the following:
many-writesfolder from this project root to itswp-contentdirnvm install 23npm cinpx nx recompile-php:asyncify php-wasm-node -- --PHP_VERSION=8.3 --WITH_DEBUG=yes