Fake `INVALID` on mainnet slot 13164552

[tersec]

Description
Nethermind v1.35.2 falsely responded that the block at canonical https://light-mainnet.beaconcha.in/slot/13164552 was INVALID via the engine API.

Steps to Reproduce
In order to replicate the behavior, please provide a detailed list of steps:

1. Have run some Nimbus v25.11.1 and Nethermind 1.35.2 pairs mainnet around the Fusaka fork

Actual behavior
A fake engine API INVALID across 11 hosts around the same time.

Expected behavior
No fake engine API INVALID.

Screenshots
If applicable, please include screenshots to help illustrate the problem.

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: nixos-25.05
• Version: 1.35.2
• Installation Method: build from source
• Consensus Client: Nimbus v25.11.1

Additional context
Please provide any additional context about the problem.

Logs

From Nimbus CL:

Dec 03 21:50:50 node-03 nimbus_beacon_node[1347361]: INF 2025-12-03 21:50:50.866+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: 46339486, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, state_root: 8182db49, receipts_root: c9a20f42, prev_randao: 40515cc6, block_number: 23935699, gas_limit: 60000000, gas_used: 59973609, timestamp: 1764798647, extra_data: BuilderNet (Flashbots), base_fee_per_gas: 25588596, block_hash: 401f3ba5, num_transactions: 345, num_withdrawals: 16, blob_gas_used: 786432, excess_blob_gas: 1572864)" blck="(blck: (slot: 13164552, proposer_index: 2060282, parent_root: b0eac6a9, state_root: 71a2c3ec, eth1data: (deposit_root: 2ebc563c, deposit_count: 2045305, block_hash: 0958d835), graffiti: , proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 478, block_number: 23935699, block_hash: 0x401f3ba5d0c8240fb889981a8d80f4744409d21052a53df471c922beef54da68, parent_hash: 0x4633948676924db21b27780428aa7e3e7ca52cb43da85ab2df164a76c4311c61, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 6), signature: 8518eaae)"

From Nethermind: attached.

fusaka_neth.txt

[jakubgs]

We attempted to re-sync the Nethermind nodes affected but after ~30 minutes of syncing the attached beacon nodes started to de-sync because of errors like this coming from the freshly purged and still-syncing EL:

INF 2025-12-04 00:04:08.187+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: 90ab530e, fee_recipient: 0x396343362be2a4da1ce0c1c210945346fb82aa49, state_root: 7c78c18d, receipts_root: 7f247781, prev_randao: a6ab32f5, block_number: 23936290, gas_limit: 60000000, gas_used: 27305315, timestamp: 1764806231, extra_data: 0xe29ca82051756173617220287175617361722e77696e2920e29ca8, base_fee_per_gas: 24742621, block_hash: c20dabf3, num_transactions: 257, num_withdrawals: 16, blob_gas_used: 655360, excess_blob_gas: 71827433)" blck="(blck: (slot: 13165184, proposer_index: 221303, parent_root: 57a0bb9d, state_root: 2188b7fe, eth1data: (deposit_root: 2ebc563c, deposit_count: 2045305, block_hash: 0958d835), graffiti: blockscape.network, proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 500, block_number: 23936290, block_hash: 0xc20dabf34ee91493a7228da357642f13addf0ab2f67e7845945a5a67aaa8c00a, parent_hash: 0x90ab530e8b9023eb6ab027052e38587d1057bfe44c9731d3a0daf50ddc40237a, fee_recipient: 0x396343362be2a4da1ce0c1c210945346fb82aa49, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 5), signature: 85f3abfc)"

Which seems to mean we can't even re-sync them.

[LukaszRozmej]

Installation Method: build from source

Do you mean Nethermind was built from source? Which commit?

[jakubgs]

We used the 1.35.2 tag to build it. Unfortunately commit is not present in the build:

 > /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/bin/nethermind --version
Version:     1.35.2
Commit:      
Source date: 1980-01-01 00:00:00Z
Runtime:     .NET 9.0.9
Platform:    Linux x64

I can check if I can verify the commit tho.

[LukaszRozmej]

Can you attach the binary?

[jakubgs]

Looks correct to me based on contents, for example:

 > nix-store --query --deriver /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2
/nix/store/4ww0nqn1s1h152kf7cnhxmy6qajxj6pp-nethermind-1.35.2.drv

 > nix derivation show /nix/store/4ww0nqn1s1h152kf7cnhxmy6qajxj6pp-nethermind-1.35.2.drv | grep src
      "src": "/nix/store/c73dq4g5p5gf8l0f329yyn0jchn2fyvx-source",

/nix/store/c73dq4g5p5gf8l0f329yyn0jchn2fyvx-source
 > grep 1.35.2 src/Nethermind/Directory.Build.props
    <VersionPrefix>1.35.2</VersionPrefix>

Done in this commit: b35164c

[jakubgs]

Not really as it's stored in our private binary cache, and it would not work without fetching all necessary deps:

 > nix-store --query --requisites /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2
...(omitted)...
/nix/store/md3sdd3c8n3cjsgasps6j7xdvkg6fpz6-dotnet-aspnetcore-runtime-9.0.9
/nix/store/m5vcs6gfjdl5v5117xh04b3nxj87bvf9-dotnet-aspnetcore-runtime-wrapped-9.0.9
/nix/store/wqahxs411zx3nykdqa8c2j3d127r3cff-zlib-1.3.1-dev
/nix/store/snkn8qc61d3vcsfqk1slwlc6nf985qw1-rocksdb-10.2.1

[jakubgs]

I could probably put it in a public Nix cache if it's necessary for the sake of debugging this.

[LukaszRozmej]

I could probably put it in a public Nix cache if it's necessary for the sake of debugging this.

hard to say, might be beneficial to get to the root cause.

Also if you can run it with --Init.AutoDump=All and post all 4 generated files.

[jakubgs]

It tries to write to /nix/store which is read-only:

 > /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/bin/nethermind --Init.AutoDump=All
04 Dec 01:04:48 | Nethermind is starting up 
04 Dec 01:04:48 | Version: 1.35.2 
04 Dec 01:04:48 | Loading embedded plugins 
04 Dec 01:04:48 |   Found plugin type Nethermind.Analytics.AnalyticsPlugin 
...
04 Dec 01:04:48 |   Found plugin type Nethermind.Xdc.XdcPlugin 
04 Dec 01:04:48 | Loading 14 assemblies from /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/lib/nethermind/plugins 
04 Dec 01:04:48 | Loading assembly Nethermind.ExternalSigner.Plugin 
...
04 Dec 01:04:48 | Loading assembly Nethermind.HealthChecks 
04 Dec 01:04:48 | Loading configuration from /nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/lib/nethermind/configs/mainnet.json 
04 Dec 01:04:48 | File logging failed. Using console logging. 
04 Dec 01:04:48 | A critical error has occurred. System.IO.IOException: Read-only file system : '/nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/lib/nethermind/logs'
   at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode)
   at System.IO.Directory.CreateDirectory(String path)
   at Nethermind.Logging.NLog.NLogManager.SetupLogDirectory(String logDirectory) in /build/source/src/Nethermind/Nethermind.Logging.NLog/NLogManager.cs:line 60
   at Nethermind.Logging.NLog.NLogManager.Setup(String logFileName, String logDirectory, String logRules) in /build/source/src/Nethermind/Nethermind.Logging.NLog/NLogManager.cs:line 38
   at Nethermind.Logging.NLog.NLogManager..ctor(String logFileName, String logDirectory, String logRules) in /build/source/src/Nethermind/Nethermind.Logging.NLog/NLogManager.cs:line 30
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__RunAsync|2>d.MoveNext() in /build/source/src/Nethermind/Nethermind.Runner/Program.cs:line 156
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.InvocationPipeline.InvokeAsync(ParseResult parseResult, CancellationToken cancellationToken)
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__ConfigureAsync|1>d.MoveNext() in /build/source/src/Nethermind/Nethermind.Runner/Program.cs:line 135
--- End of stack trace from previous location ---
   at Program.<Main>$(String[] args) in /build/source/src/Nethermind/Nethermind.Runner/Program.cs:line 66

Not sure how to deal with that.

[jakubgs]

I need to go to sleep for now. Will get back to this tomorrow.

[LukaszRozmej]

Done in this commit: b35164c

While the commit looks good, we didn't manage to replicate, nor got any other report of this issue.
This makes me still suspicious about your setup - especially as it blows up consistently for you - which should be happening for everyone.

This keeps me thinking it has to be something either with your setup or in your setup that is uncovering the issue.

Could you:

• Build new version from master and try on that (we are in the middle of releasing it and our QA process didn't find any issues yet, so we consider it quite stable).
• List your full configuration changes for the node, any additional configs, CLI params or environment variables that affect how the node operates.

[jakubgs]

Sure, we can try building from master.

Here's our flags:

ExecStart=/nix/store/im6lknbrlnk4480vvslfc0kfkxgna1v9-nethermind-1.35.2/bin/nethermind --datadir %S/nethermind-mainnet \
--JsonRpc.JwtSecretFile %d/jwtsecret \
--config mainnet \
--log DEBUG \
--HealthChecks.Enabled true \
--JsonRpc.Enabled true \
--JsonRpc.EngineHost 0.0.0.0 \
--JsonRpc.EnginePort 8552 \
--JsonRpc.Host 0.0.0.0 \
--JsonRpc.Port 8547 \
--JsonRpc.WebSocketsPort 8548 \
--Metrics.Enabled true \
--Metrics.ExposePort 6061 \
--Network.DiscoveryPort 30304 \
--Network.P2PPort 30304 \

[jakubgs]

Though latest master doesn't look that good:

[jakubgs]

But I can try in a bit.

[LukaszRozmej]

Though latest master doesn't look that good:

I think those are mostly either hive test issues or test-compatibility issues, let me confirm...