You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tracking issue for problems on RISC-V Linux platform #4099
(all versions from at least v16.x) Sporadic Nodejs segfault on sg2042 revyos/revyos#27. This one only triggers on SG2042, which has 64 cores. It is mysterious as it happens by chance and when gdb is attached, the call stack is already corrupted so the cause is still unknown.
Thanks for collating these links. I'm dumping this info here just now so I have it logged and shared. I'm running various builds at the moment natively (trying to populate my ccaches!) and with the cross-compilers. At the moment the cross compilers are having problems building, and since those are how the unofficial builds need to run it's something we'll want to resolve.
For reference in 24.x I'm getting:
CXX(host) /home/node/node/out/Release/obj.host/v8_base_without_compiler/deps/v8/src/api/api-arguments.o
In file included from ../deps/v8/src/execution/vm-state-inl.h:12,
from ../deps/v8/src/api/api-arguments-inl.h:13,
from ../deps/v8/src/api/api-arguments.cc:7:
../deps/v8/src/execution/simulator.h: In static member function 'static int v8::internal::SimulatorStack::JSStackLimitMargin()':
../deps/v8/src/execution/simulator.h:66:55: error: 'JSStackLimitMargin' is not a member of 'v8::internal::Simulator'
66 | static int JSStackLimitMargin() { return Simulator::JSStackLimitMargin(); }
| ^~~~~~~~~~~~~~~~~~
make: *** [tools/v8_gypfiles/v8_base_without_compiler.host.mk:1140: /home/node/node/out/Release/obj.host/v8_base_without_compiler/deps/v8/src/api/api-arguments.o] Error 1
rm f41dec747f7d07bc6189224d714bfe6f70ce7700.intermediate 4b9faa377f69f07a06bcf45eb52d6b09bd3dc776.intermediate 45471a7c870f238f988ac4c73ccc531d343b1d2d.intermediate 5cf52c47b2c7a3d1debdf7e64552adabc181a77d.intermediate 91e5efd5c01b316d5563811267cb2f34bf83226a.intermediate
make: Leaving directory '/home/node/node/out'
and for main I'm getting:
In file included from ../deps/v8/third_party/highway/src/hwy/ops/scalar-inl.h:24,
from ../deps/v8/third_party/highway/src/hwy/highway.h:722,
from ../deps/v8/src/json/json-stringifier.cc:10:
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h: In instantiation of 'struct hwy::N_SCALAR::detail::FixedTagChecker<unsigned char, 16>':
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:423:7: required by substitution of 'template<class T, long unsigned int kNumLanes> using hwy::N_SCALAR::FixedTag = typename hwy::N_SCALAR::detail::FixedTagChecker::type [with T = unsigned char; long unsigned int kNumLanes = 16]'
423 | using FixedTag = typename detail::FixedTagChecker<T, kNumLanes>::type;
| ^~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3379:29: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendStringSIMD(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3379 | hw::FixedTag<SrcChar, 16> tag;
| ^~~
../deps/v8/src/json/json-stringifier.cc:3316:28: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendString(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3316 | return AppendStringSIMD(chars, length, no_gc);
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:2531:17: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::SerializeString(v8::internal::Tagged<v8::internal::HeapObject>, const v8::internal::DisallowGarbageCollection&) [with StringT = v8::internal::SeqOneByteString; Char = unsigned char; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
2531 | AppendString(chars, length, no_gc);
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:2554:47: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::TrySerializeSimpleObject(v8::internal::Tagged<v8::internal::Union<v8::internal::Smi, v8::internal::HeapNumber, v8::internal::BigInt, v8::internal::String, v8::internal::Symbol, v8::internal::Boolean, v8::internal::Null, v8::internal::Undefined, v8::internal::JSReceiver> >) [with Char = unsigned char]'
2554 | return SerializeString<SeqOneByteString>(obj, no_gc);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3167:38: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::SerializeObject(v8::internal::Tagged<v8::internal::Union<v8::internal::Smi, v8::internal::HeapNumber, v8::internal::BigInt, v8::internal::String, v8::internal::Symbol, v8::internal::Boolean, v8::internal::Null, v8::internal::Undefined, v8::internal::JSReceiver> >, const v8::internal::DisallowGarbageCollection&) [with Char = unsigned char; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3167 | FastJsonStringifierResult result = TrySerializeSimpleObject(object);
| ^~~~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3509:43: required from here
3509 | one_byte_stringifier.SerializeObject(*object, no_gc);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:369:27: error: static assertion failed: Too many lanes
369 | static_assert(kNumLanes <= HWY_LANES(T), "Too many lanes");
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:369:27: note: the comparison reduces to '(16 <= 1)'
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h: In instantiation of 'struct hwy::N_SCALAR::Simd<unsigned char, 16, 0>':
../deps/v8/src/json/json-stringifier.cc:3379:29: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendStringSIMD(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
In file included from ../deps/v8/third_party/highway/src/hwy/ops/scalar-inl.h:24,
from ../deps/v8/third_party/highway/src/hwy/highway.h:722,
from ../deps/v8/src/json/json-stringifier.cc:10:
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h: In instantiation of 'struct hwy::N_SCALAR::detail::FixedTagChecker<unsigned char, 16>':
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:423:7: required by substitution of 'template<class T, long unsigned int kNumLanes> using hwy::N_SCALAR::FixedTag = typename hwy::N_SCALAR::detail::FixedTagChecker::type [with T = unsigned char; long unsigned int kNumLanes = 16]'
423 | using FixedTag = typename detail::FixedTagChecker<T, kNumLanes>::type;
| ^~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3379:29: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendStringSIMD(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3379 | hw::FixedTag<SrcChar, 16> tag;
| ^~~
../deps/v8/src/json/json-stringifier.cc:3316:28: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendString(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3316 | return AppendStringSIMD(chars, length, no_gc);
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:2531:17: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::SerializeString(v8::internal::Tagged<v8::internal::HeapObject>, const v8::internal::DisallowGarbageCollection&) [with StringT = v8::internal::SeqOneByteString; Char = unsigned char; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
2531 | AppendString(chars, length, no_gc);
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:2554:47: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::TrySerializeSimpleObject(v8::internal::Tagged<v8::internal::Union<v8::internal::Smi, v8::internal::HeapNumber, v8::internal::BigInt, v8::internal::String, v8::internal::Symbol, v8::internal::Boolean, v8::internal::Null, v8::internal::Undefined, v8::internal::JSReceiver> >) [with Char = unsigned char]'
2554 | return SerializeString<SeqOneByteString>(obj, no_gc);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3167:38: required from 'v8::internal::FastJsonStringifierResult v8::internal::FastJsonStringifier<Char>::SerializeObject(v8::internal::Tagged<v8::internal::Union<v8::internal::Smi, v8::internal::HeapNumber, v8::internal::BigInt, v8::internal::String, v8::internal::Symbol, v8::internal::Boolean, v8::internal::Null, v8::internal::Undefined, v8::internal::JSReceiver> >, const v8::internal::DisallowGarbageCollection&) [with Char = unsigned char; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
3167 | FastJsonStringifierResult result = TrySerializeSimpleObject(object);
| ^~~~~~~~~~~~~~~~~~~~~~~~
../deps/v8/src/json/json-stringifier.cc:3509:43: required from here
3509 | one_byte_stringifier.SerializeObject(*object, no_gc);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:369:27: error: static assertion failed: Too many lanes
369 | static_assert(kNumLanes <= HWY_LANES(T), "Too many lanes");
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h:369:27: note: the comparison reduces to '(16 <= 1)'
../deps/v8/third_party/highway/src/hwy/ops/shared-inl.h: In instantiation of 'struct hwy::N_SCALAR::Simd<unsigned char, 16, 0>':
../deps/v8/src/json/json-stringifier.cc:3379:29: required from 'bool v8::internal::FastJsonStringifier<Char>::AppendStringSIMD(const SrcChar*, size_t, const v8::internal::DisallowGarbageCollection&) [with SrcChar = unsigned char; Char = unsigned char; size_t = long unsigned int; v8::internal::DisallowGarbageCollection = v8::internal::PerThreadAssertScopeEmpty<false, v8::internal::SAFEPOINTS_ASSERT, v8::internal::HEAP_ALLOCATION_ASSERT>]'
This is probably caused by google/highway@f38ee25 ,
which will treat the gcc compiler as broken if the version is less than 16.0, which is not even released yet.
When the gcc compiler is detected as broken, it will fallback to HWY_SCALAR instead of HWY_EMU128 and causing the assertion error.
So one way is to accept that the compiler might do broken stuff and compile anyway with -DHWY_BROKEN_EMU128=0, another way is to use GCC >= 16 or clang.
(Noting for my own reference that the BPiF3 (SpacemiT-K1) takes about 45 minutes to run the v22 test suite, excluding test-doc. That included timeouts - probably due to a crash - on the single-executable-application tests)
Thanks @kxxt I'll take a look at those since I haven't run tests with those included yet and some of the failures were OOMs which could be related to 60591. Was your SG2042 the Milk-V Pioneer or a different board?
Thanks @kxxt I'll take a look at those since I haven't run tests with those included yet and some of the failures were OOMs which could be related to 60591. Was your SG2042 the Milk-V Pioneer or a different board?
Yes. It's a Milk-V Pioneer that I could access remotely(Thanks to felixonmars!). Unfortunately there is a hard-to-debug problem(revyos/revyos#27) specific to that board that could cause node.js to segfault sometimes, which happens sometimes during tests.
There is also bug in old vendor kernel(5.10.x) for TH1520, which could cause weird issues for node.js. That is also mentioned in revyos/revyos#27 (The case of lpi4a).
I had remote access to someone else's Pioneer too at one point but it's not available now. Mine are running a 5.10-based kernel from Ubuntu 24.04 so that link is very useful. It is about time I updated my lpi4a though so I should probably get hold of one of the later kernels as you suggested. I do worry about how reliable the C910/C920 cores are though as we've seen unusual crashes with java on systems with those cores. I'll perhaps need to do a direct compare with the same build on bpiF3 and TH1520 and see if they give the same results, but I've been building different versions on each so far.
I remember that on SG2042, nodejs 25.2.1 with nodejs/node#60591 and nodejs/node#60588 applied passed all tests except test-snapshot-reproducible.js,
@kxxt Did you have to do anything special to get the single executable appliation tests in test/sea to pass? On a TH1520 with 25.x which includes those two commits I'm getting SIGILLs (Could it be trying to run RVV1.0 stuff? Doesn't look like node's v8-options lets you disable those) and elsewhere (e.g. under qemu or on the bpiF3) I'm still getting consistent segmentation faults. (EDIT: These have passed in some runs in the TH1520, but I'm not yet sure what was special about those!)
I remember that on SG2042, nodejs 25.2.1 with nodejs/node#60591 and nodejs/node#60588 applied passed all tests except test-snapshot-reproducible.js,
@kxxt Did you have to do anything special to get the single executable appliation tests in test/sea to pass? On a TH1520 with 25.x which includes those two commits I'm getting SIGILLs (Could it be trying to run RVV1.0 stuff? Doesn't look like node's v8-options lets you disable those) and elsewhere (e.g. under qemu or on the bpiF3) I'm still getting consistent segmentation faults. (EDIT: These have passed in some runs in the TH1520, but I'm not yet sure what was special about those!)
That's useful information. Given that there are multiple failures on different hardware here I've raised another issue on this and will put excludes in for RISC-V on the SEA tests (which is an experimental feature just now anyway and not looking like it's going to be anything else in the meantime). We can look at doing further debugging on this in the new issue.
Was your passing test suite done with that patch applied on the SG2042?
I'll put some more results in the issue for information but I have successfully been able to run the SEA tests using the build from the TH1520 under qemu on my x64 laptop, but sine they will fail on most people's boards I think it's fair to exclude them for now.
I could open a PR for that patch if you think that's the reasonable default. But I bet folks from ubuntu targeting rva23 profile (with V extension included) would not want it.
I agree. Ideally we'd have this as a runtime switch (or have highway detect it in a better way) so I think I'm ok with leaving the default as-is for now and just disabling the tests.
In the interests of logging issues for search fodder even if they have solutions I'll note that if you try to build with a version of gcc <14 you'll hit the following error (Spotted in native builds). Not confirmed but I expect it's from the changes in the V8 14.2.231.9 PR which landed back in October:
In file included from ../deps/v8/src/base/cpu.cc:60:
/usr/lib/gcc/riscv64-linux-gnu/13/include/riscv_vector.h:32:2: error: #error "Vector intrinsics require the vector extension."
32 | #error "Vector intrinsics require the vector extension."
| ^~~~~
../deps/v8/src/base/cpu.cc:66:67: warning: target attribute is not supported on this machine [-Wattributes]
66 | __attribute__((target("arch=+v"))) static unsigned vlen_intrinsic() {
| ^
../deps/v8/src/base/cpu.cc: In function ‘unsigned int vlen_intrinsic()’:
../deps/v8/src/base/cpu.cc:67:32: error: ‘__riscv_vlenb’ was not declared in this scope
67 | return static_cast<unsigned>(__riscv_vlenb() * 8);
| ^~~~~~~~~~~~~
make[1]: *** [tools/v8_gypfiles/v8_libbase.target.mk:225: /home/sxa/node/out/Release/obj.target/v8_libbase/deps/v8/src/base/cpu.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Hi, I saw the earlier discussion in #2876 and would like to understand the current status.
What is the main remaining blocker for official linux/riscv64 binaries today? If the community wants to help, what kind of contribution would be most useful?
The CITGM run @sxa started in July, to find which modules have no riscv64 prebuilds. Still needs a gap analysis against the other platforms.
The Single Executable Application failures (nodejs/node#61110) are parked rather than fixed, since SEA is experimental anyway.
On what would help: boards and test cycles. Most of what is left is board-specific, and the K3 failure has only ever shown up on that silicon. If you have a riscv64 machine that is neither a K1 nor a K3, running the test suite on it and reporting what breaks would add a data point nobody has today.
@sxa is the one actually doing this work, so he is better placed than me to say where the effort should go.
@gounthar has summarised it well. The only one of the above that I'm currently trying to look at is the illegal instruction but that's slow work building V8 :-) An initial build did not show the crash when running V8's own test suite so I need to identify how to trigger it properly. Also I'm relatively new to the V8 development process at the moment though.
The other failures I'm not actively looking at just now. As per some of the comments earlier in this issue I have run the tests on a TH1520 and JH7110 (VF2) boards but I've not done that for a while so it might be worth re-running to see if there are still issues on those that we're not closely tracking at the moment. Also as Bruno said the gap analysis to identify which of the failures in the CitGM npm module testing are specific to RISC-V would be useful. Which of the issues to pick up really depends on your level of knowledge and interest.
I'll also point out that other than the test failures/crashes the other blocker at the moment for pushing it up to a higher tier would be availability of machines of a suitable performance for inclusion in our regular CI. The K3 that I have is quick enough so I'm hoping that we'll be able to get access to more of them at some point to be able to include them in the regular CI that occurs on each PR.
Error information from the failing run saved here for reference
13:27:04 [PASS] i64 mutability
13:27:04 [EXPECTED_FAILURE][FAIL] Calling setter without argument
13:27:04 set WebAssembly.Global.value): Argument 0 is required
13:27:04 TypeError: set WebAssembly.Global.value): Argument 0 is required
13:27:04 at Test.<anonymous> (/home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/test/fixtures/wpt/wasm/jsapi/global/value-get-set.any.js:140:24)
13:27:04 at Test.step (/home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/test/fixtures/wpt/resources/testharness.js:2869:25)
13:27:04 at test (/home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/test/fixtures/wpt/resources/testharness.js:633:30)
13:27:04 at /home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/test/fixtures/wpt/wasm/jsapi/global/value-get-set.any.js:125:1
13:27:04 at Script.runInThisContext (node:vm:138:12)
13:27:04 at runInThisContext (node:vm:318:38)
13:27:04 at Object.<anonymous> (/home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/test/common/wpt/worker.js:102:3)
13:27:04 at Module._compile (node:internal/modules/cjs/loader:1934:14)
13:27:04 at Object..js (node:internal/modules/cjs/loader:2074:10)
13:27:04 at Module.load (node:internal/modules/cjs/loader:1656:32)
13:27:04 Command: /home/iojs/workspace/sxa-node-test-commit-rvlinux/nodes/riscv64/out/Release/node --experimental-wasm-modules undefined 'global/value-get-set.any.js'
13:27:04
13:27:04 [PASS] Stray argument
The last two are flaky. The second one appears to be a regression.
[kxxt@centiskorch node-riscv]$ out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/kxxt/node-riscv/test/sequential/test-watch-mode.mjs
Test failure: 'should reload env variables when --env-file changes'
Location: test/sequential/test-watch-mode.mjs:227:3
Error: Timed out waiting for restart
at Timeout.<anonymous> (file:///home/kxxt/node-riscv/test/sequential/test-watch-mode.mjs:96:25)
at listOnTimeout (node:internal/timers:685:17)
at process.processTimers (node:internal/timers:618:7)
Test failure: 'should not crash when --env-file-if-exists points to a missing file'
Location: test/sequential/test-watch-mode.mjs:306:3
Error: Timed out waiting for restart
at Timeout.<anonymous> (file:///home/kxxt/node-riscv/test/sequential/test-watch-mode.mjs:96:25)
at listOnTimeout (node:internal/timers:685:17)
at process.processTimers (node:internal/timers:618:7)
Test failure: 'should support multiple --env-file flags'
Location: test/sequential/test-watch-mode.mjs:944:3
Error: Timed out waiting for restart
at Timeout.<anonymous> (file:///home/kxxt/node-riscv/test/sequential/test-watch-mode.mjs:96:25)
at listOnTimeout (node:internal/timers:685:17)
at process.processTimers (node:internal/timers:618:7)
Yeah I have seen failures in test-watch-mode.js too but I wasn't yet certain if it was consistent as I've been focussing on RVV stuff on v26.
Did you only get the failure on that single test and not the other test-watch-mode-* ones?
Yes. I only get the failure for this single test. I am runni g my CI again to double check.
This should be a recent regression as I didn't get this failure on Jul 25.
I've just done a build from the current main (commit 85e93a165aa21f7796de4f38e2be7c07a232d0eb plus my RVV patch on top) on both the K3 and - at least on the command line - all four of those tests seem to be passing now (tested with 20 iterations) :-)
I do get your timeout error on the slower A100 cores on the K3 though but the exact failure point is not consistent, and also not always at the same line number of the test which does indicate that it's intermittent and the timeouts may need increasing. I've had a quick try with modifying a couple of the timeouts (The 1000 one at line 89) although that gave me another error:
Test failure: 'should not crash when --env-file-if-exists points to a missing file'
Location: test/sequential/test-watch-mode.mjs:306:3
AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected
[
+ "Change detected in '/home/sxa/git/node/test/.tmp.0/34.env'",
+ "Restarting '/home/sxa/git/node/test/.tmp.0/9.js'",
'ENV: undefined',
"Completed running '/home/sxa/git/node/test/.tmp.0/9.js'. Waiting for file changes before restarting..."
]
at TestContext.<anonymous> (file:///home/sxa/git/node/test/sequential/test-watch-mode.mjs:318:14)
at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
at async Test.run (node:internal/test_runner/test:1409:7)
at async Promise.all (index 6)
at async Suite.run (node:internal/test_runner/test:1905:7)
at async startSubtestAfterBootstrap (node:internal/test_runner/harness:387:3) {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: [Array],
expected: [Array],
operator: 'deepStrictEqual',
diff: 'simple'
}
Hopefully that won't be too hard to fix. (Although I'll also mention that I'm about to go on holiday!)
EDIT: I also had a quite try with the same commit without my RVV patch and that still passed the other three tests (test-watch-mode.mjs gave an illegal instruction) so it's possible the other problems are now resolved in the codebase, which would be nice, although it looks like they still failed in jenkins.
EDIT2: I tried with the commit that you used (outside jenkins) and got the same failures so they have been fixed in later versions :-)
The V8 15.5 update in nodejs/node#66537 (which went in faster than I expected after 15.2 - nice work!) has a change that breaks my RISC-V builds. The commit causing the problem is v8/v8@16d931d - the compiler has std::hardware_destructive_interference_size defined as 32 where before that commit a value of 64 was hardcoded. It appears that clang-22 on RISC-V sets it to 64 so one option other than reverting/change that commit) is to require that version of clang on RISC-V.
the compiler has std::hardware_destructive_interference_size defined as 32 where before that commit a value of 64 was hardcoded. It appears that clang-22 on RISC-V sets it to 64 so one option other than reverting/change that commit) is to require that version of clang on RISC-V.
In recent weeks that check has been reinstated based on whether the hwprobe syscall succeeds, but it means that the fallback mechanism is not in the current version of Node. I suspect we can backport that at an appropriate time but I'd prefer to get some of the fixes for RVV-related crashes in first.
Activity
sxa commentedon Dec 5, 2025
Thanks for collating these links. I'm dumping this info here just now so I have it logged and shared. I'm running various builds at the moment natively (trying to populate my ccaches!) and with the cross-compilers. At the moment the cross compilers are having problems building, and since those are how the unofficial builds need to run it's something we'll want to resolve.
For reference in 24.x I'm getting:
and for
mainI'm getting:kxxt commentedon Dec 6, 2025
This is probably caused by google/highway@f38ee25 ,
which will treat the gcc compiler as broken if the version is less than 16.0, which is not even released yet.
When the gcc compiler is detected as broken, it will fallback to
HWY_SCALARinstead ofHWY_EMU128and causing the assertion error.So one way is to accept that the compiler might do broken stuff and compile anyway with
-DHWY_BROKEN_EMU128=0, another way is to use GCC >= 16 or clang.Link: google/highway#2406 (comment)
sxa commentedon Dec 6, 2025
Notes to self:
c2843b722cabuilds ok with the cross-compiler and is possibly the last one to do so.CXXFLAGS=-DHWY_BROKEN_EMU128=0as per the comment abovesxa commentedon Dec 12, 2025
Initial test failure list
Failures on v25.x on TH1520
out/Release/node /home/sxa/node/test/sea/test-single-executable-application-empty.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-esm-detect-ambiguous.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-esm-loader-entry-url.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-esm-tla-syntax-errors-not-recognized-as-tla-error.mjs
out/Release/node /home/sxa/node/test/sea/test-single-executable-application-exec-argv.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-typescript-commonjs.mjs
out/Release/node /home/sxa/node/test/sea/test-single-executable-application-snapshot-and-code-cache.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-typescript-eval.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-typescript-transform.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/es-module/test-typescript.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-assert-partial-deep-equal.js
out/Release/node --expose_gc --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-abortsignal-drop-settled-signals.mjs
out/Release/node /home/sxa/node/test/parallel/test-buffer-indexof.js
out/Release/node /home/sxa/node/test/parallel/test-compile-cache-typescript-commonjs.js
out/Release/node /home/sxa/node/test/parallel/test-compile-cache-typescript-strip-miss.js
out/Release/node /home/sxa/node/test/parallel/test-compile-cache-typescript-transform.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-config-file.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-module-strip-types.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-node-output-eval.mjs
out/Release/node /home/sxa/node/test/parallel/test-repl-paste-big-data.js
out/Release/node /home/sxa/node/test/parallel/test-runner-cli.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-runner-global-setup-teardown.mjs
out/Release/node /home/sxa/node/test/parallel/test-snapshot-reproducible.js
out/Release/node /home/sxa/node/test/parallel/test-util-getcallsites.js
out/Release/node /home/sxa/node/test/parallel/test-worker-syntax-error.js
out/Release/node --expose-internals /home/sxa/node/test/sequential/test-worker-heapsnapshot-options.js
Failures on bpi-F3 on Node 24
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/report/test-report-fatalerror-oomerror-set.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/report/test-report-fatalerror-oomerror-compact.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/report/test-report-fatalerror-oomerror-filename.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-repl-paste-big-data.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-runner-run.mjs
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-snapshot-reproducible.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-strace-openat-openssl.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-stream2-read-sync-stack.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/parallel/test-worker-resource-limits.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/abort/test-abort-fatal-error.js
out/Release/node --expose-internals --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sequential/test-worker-heapsnapshot-options.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-asset-keys.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-asset-keys-empty.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-assets.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-assets-raw.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-disable-experimental-sea-warning.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-empty.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-exec-argv.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-exec-argv-empty.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-exec-argv-extension-cli.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-exec-argv-extension-env.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-exec-argv-extension-none.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-inspect.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-inspect-in-sea-flags.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-snapshot.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-snapshot-and-code-cache.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-snapshot-worker.js
out/Release/node --test-reporter=./test/common/test-error-reporter.js --test-reporter-destination=stdout /home/sxa/node/test/sea/test-single-executable-application-use-code-cache.js
(Edited to add the following ones after the initial results above)
v22.x on BPiF3
v22.x on BPiF3 with backports from the next comment
Note: cherry-picked commits are
cdc3ca8a7edand22a3eb00e85and only affect the tests:v25.x-staging (commit c57a40c898c) on bpiF3
(Noting for my own reference that the BPiF3 (SpacemiT-K1) takes about 45 minutes to run the v22 test suite, excluding
test-doc. That included timeouts - probably due to a crash - on thesingle-executable-applicationtests)kxxt commentedon Dec 13, 2025
I remember that on SG2042, nodejs 25.2.1 with nodejs/node#60591 and nodejs/node#60588 applied passed all tests except
test-snapshot-reproducible.js, which is currently being tracked upstream at https://issues.chromium.org/issues/460332153sxa commentedon Dec 13, 2025
Thanks @kxxt I'll take a look at those since I haven't run tests with those included yet and some of the failures were OOMs which could be related to 60591. Was your SG2042 the Milk-V Pioneer or a different board?
kxxt commentedon Dec 13, 2025
Yes. It's a Milk-V Pioneer that I could access remotely(Thanks to felixonmars!). Unfortunately there is a hard-to-debug problem(revyos/revyos#27) specific to that board that could cause node.js to segfault sometimes, which happens sometimes during tests.
There is also bug in old vendor kernel(5.10.x) for TH1520, which could cause weird issues for node.js. That is also mentioned in revyos/revyos#27 (The case of lpi4a).
sxa commentedon Dec 13, 2025
I had remote access to someone else's Pioneer too at one point but it's not available now. Mine are running a 5.10-based kernel from Ubuntu 24.04 so that link is very useful. It is about time I updated my lpi4a though so I should probably get hold of one of the later kernels as you suggested. I do worry about how reliable the C910/C920 cores are though as we've seen unusual crashes with java on systems with those cores. I'll perhaps need to do a direct compare with the same build on bpiF3 and TH1520 and see if they give the same results, but I've been building different versions on each so far.
sxa commentedon Dec 17, 2025
@kxxt Did you have to do anything special to get the single executable appliation tests in
test/seato pass? On a TH1520 with 25.x which includes those two commits I'm getting SIGILLs (Could it be trying to run RVV1.0 stuff? Doesn't look like node'sv8-optionslets you disable those) and elsewhere (e.g. under qemu or on the bpiF3) I'm still getting consistent segmentation faults.(EDIT: These have passed in some runs in the TH1520, but I'm not yet sure what was special about those!)
kxxt commentedon Dec 18, 2025
I don't think I have done anything special for those tests. But the SIGILLs might be related to RVV 1.0 stuff in highway.
It would be a better idea to confirm it in gdb.
Because RVV runtime dispatch is broken in highway(https://github.com/google/highway/blob/4262db0ef703c7a3372f7a921a7b69ffd5db5829/hwy/detect_targets.h#L784 , note the
&& 0at the end), IIRC it would unconditionally enable RVV.So as a distribution targeting
rv64gcwithoutVextension, we disable RVV completely in highway with a patch https://github.com/felixonmars/archriscv-packages/blob/master/nodejs-lts-krypton/hwy-broken-rvv.diffI could open a PR for that patch if you think that's the reasonable default. But I bet folks from ubuntu targeting
rva23profile (withVextension included) would not want it.Here is a build and test log for 24.11.1: https://archriscv.felixc.at/.status/log.htm?url=logs/nodejs-lts-krypton/nodejs-lts-krypton-24.11.1-1.log (With the patches in https://github.com/felixonmars/archriscv-packages/tree/master/nodejs-lts-krypton)
sxa commentedon Dec 18, 2025
That's useful information. Given that there are multiple failures on different hardware here I've raised another issue on this and will put excludes in for RISC-V on the SEA tests (which is an experimental feature just now anyway and not looking like it's going to be anything else in the meantime). We can look at doing further debugging on this in the new issue.
Was your passing test suite done with that patch applied on the SG2042?
I'll put some more results in the issue for information but I have successfully been able to run the SEA tests using the build from the TH1520 under qemu on my x64 laptop, but sine they will fail on most people's boards I think it's fair to exclude them for now.
I agree. Ideally we'd have this as a runtime switch (or have highway detect it in a better way) so I think I'm ok with leaving the default as-is for now and just disabling the tests.
kxxt commentedon Dec 18, 2025
Yes. IIRC when targeting rv64gc with
-march=rv64gcand using clang, node.js would throw build errors in highway without that patch.RevySR commentedon Feb 4, 2026
fixed: v8/v8@6b1b9bc
sxa commentedon Feb 19, 2026
In the interests of logging issues for search fodder even if they have solutions I'll note that if you try to build with a version of gcc <14 you'll hit the following error (Spotted in native builds). Not confirmed but I expect it's from the changes in the V8 14.2.231.9 PR which landed back in October:
70 remaining items
Polaris-911 commentedon Aug 17, 2026
Hi, I saw the earlier discussion in #2876 and would like to understand the current status.
What is the main remaining blocker for official linux/riscv64 binaries today? If the community wants to help, what kind of contribution would be most useful?
Thanks.
gounthar commentedon Aug 17, 2026
I am not a Node collaborator, so treat this as one contributor's reading of the thread.
The checklist at the top of this issue is the live status. The items that look most load-bearing to me:
test-code-cachefails on a Linux/riscv64 build with FFI enabled. node#64717,test-code-cachefailing when FFI is enabled. Reproducible on a BPI-F3 (K1), but the trail points at the toolchain: it passes with gcc 15, clang-22 on x64 does not show it, and @kxxt could not reproduce it on an SG2042 at all.The Single Executable Application failures (nodejs/node#61110) are parked rather than fixed, since SEA is experimental anyway.
On what would help: boards and test cycles. Most of what is left is board-specific, and the K3 failure has only ever shown up on that silicon. If you have a riscv64 machine that is neither a K1 nor a K3, running the test suite on it and reporting what breaks would add a data point nobody has today.
@sxa is the one actually doing this work, so he is better placed than me to say where the effort should go.
sxa commentedon Aug 18, 2026
@gounthar has summarised it well. The only one of the above that I'm currently trying to look at is the illegal instruction but that's slow work building V8 :-) An initial build did not show the crash when running V8's own test suite so I need to identify how to trigger it properly. Also I'm relatively new to the V8 development process at the moment though.
The other failures I'm not actively looking at just now. As per some of the comments earlier in this issue I have run the tests on a TH1520 and JH7110 (VF2) boards but I've not done that for a while so it might be worth re-running to see if there are still issues on those that we're not closely tracking at the moment. Also as Bruno said the gap analysis to identify which of the failures in the CitGM npm module testing are specific to RISC-V would be useful. Which of the issues to pick up really depends on your level of knowledge and interest.
I'll also point out that other than the test failures/crashes the other blocker at the moment for pushing it up to a higher tier would be availability of machines of a suitable performance for inclusion in our regular CI. The K3 that I have is quick enough so I'm hoping that we'll be able to get access to more of them at some point to be able to include them in the regular CI that occurs on each PR.
sxa commentedon Aug 26, 2026
Latest run (albeit patched to stop the crashes on the RVV issue and including nodejs/node#62251) on the K3's X100 cores:
test-wasm-jsapi.mjs is the only new failure relative to the results in #4099 (comment) but it passed in the next run
Error information from the failing run saved here for reference
sxa commentedon Aug 26, 2026
node-api SEA test is being excluded under nodejs/node#65569
kxxt commentedon Aug 29, 2026
Latest test results on SG2042 at commit 05447419fbc708fbedaa7bc9740dcd242fab0d08
The last two are flaky. The second one appears to be a regression.
sxa commentedon Sep 1, 2026
Yeah I have seen failures in
test-watch-mode.jstoo but I wasn't yet certain if it was consistent as I've been focussing on RVV stuff on v26.Did you only get the failure on that single test and not the other
test-watch-mode-*ones?kxxt commentedon Sep 1, 2026
Yes. I only get the failure for this single test. I am runni g my CI again to double check.
This should be a recent regression as I didn't get this failure on Jul 25.
sxa commentedon Sep 3, 2026
I've just done a build from the current
main(commit85e93a165aa21f7796de4f38e2be7c07a232d0ebplus my RVV patch on top) on both the K3 and - at least on the command line - all four of those tests seem to be passing now (tested with 20 iterations) :-)I do get your timeout error on the slower A100 cores on the K3 though but the exact failure point is not consistent, and also not always at the same line number of the test which does indicate that it's intermittent and the timeouts may need increasing. I've had a quick try with modifying a couple of the timeouts (The
1000one at line 89) although that gave me another error:Hopefully that won't be too hard to fix. (Although I'll also mention that I'm about to go on holiday!)
EDIT: I also had a quite try with the same commit without my RVV patch and that still passed the other three tests (
test-watch-mode.mjsgave an illegal instruction) so it's possible the other problems are now resolved in the codebase, which would be nice, although it looks like they still failed in jenkins.EDIT2: I tried with the commit that you used (outside jenkins) and got the same failures so they have been fixed in later versions :-)
test: isolate styleText tests from the environment
test: isolate styleText tests from the environment
test: isolate styleText tests from the environment
sxa commentedon Oct 8, 2026
The V8 15.5 update in nodejs/node#66537 (which went in faster than I expected after 15.2 - nice work!) has a change that breaks my RISC-V builds. The commit causing the problem is v8/v8@16d931d - the compiler has std::hardware_destructive_interference_size defined as 32 where before that commit a value of 64 was hardcoded. It appears that clang-22 on RISC-V sets it to 64 so one option other than reverting/change that commit) is to require that version of clang on RISC-V.
kxxt commentedon Oct 8, 2026
For GCC, I proposed to update hardware_destructive_interference_size to 64 for RISC-V in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116662 and the patch for it lands in GCC 15 IIRC.
sxa commentedon Oct 9, 2026
A quick note on something relevant: There was a refactor of some of the CPU-specific code in V8 15 which landed in Node as part of the V8 15.5 update. This removed
src/base/cpu.ccwhich included the fallback mechanism which meant that when building on glibc prior to 2.39 it would check the CPU fields (effectively/proc/cpuinfo) to determine whether V8 would use things like the vector extensions: https://github.com/v8/v8/blob/0b9dd147a10b9f12751a5ecf1e5f59e88a2a2b7b/src/base/cpu.cc#L1018In recent weeks that check has been reinstated based on whether the
hwprobesyscall succeeds, but it means that the fallback mechanism is not in the current version of Node. I suspect we can backport that at an appropriate time but I'd prefer to get some of the fixes for RVV-related crashes in first.