NotALeader Errors

[nixxholas]

Greetings guys, facing this errors and tried alternatives around this. The only way is to use the neo4j crate by another developer (But it does not have the bells and whistles we have here).

Error: unexpected response for RUN: Failure(Failure { metadata: BoltMap { value: {BoltString { value: "message" }: String(BoltString { value: "No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER" }), BoltString { value: "code" }: String(BoltString { value: "Neo.ClientError.Cluster.NotALeader" })} } })

Anyway we can workaround this? Using Neo4J Aura, seems like routing is already dealt with on the server side.

[knutwalker]

Hi @nixxholas,

how are you connecting to the database, that is, which protocol? Do you use neo4j/neo4j+s or bolt/bolt+s?

[nixxholas]

Hi @nixxholas,

how are you connecting to the database, that is, which protocol? Do you use neo4j/neo4j+s or bolt/bolt+s?

Greetings!

I did both +s variants. The current workaround is to spam re-initialising Graph until the right routing is active.

Even then, the connection can get routed wrongly, wondering what I should do next?

I've had no success with directly targeting the write node.

[knutwalker]

Is it an Aura db or something self hosted?
We haven't implemented client-side routing yet and it looks like the server is not doing server-side routing properly (which would only happen with a neo4j connection)

[nixxholas]

Is it an Aura db or something self hosted? We haven't implemented client-side routing yet and it looks like the server is not doing server-side routing properly (which would only happen with a neo4j connection)

It is indeed AuraDB! Yeah, am aware! But I'm getting the same set of errors even with bolt connections. Not sure how this should be overcome.

[knutwalker]

What neo4rs version are you using?

[nixxholas]

What neo4rs version are you using?

0.7.2!

[knutwalker]

I could reproduce the issue and I believe I have a fix in #183. I also backported this to the 0.7 branch and will do a 0.7.3 shortly.

If you want to test it out before that you can use

[dependencies]
neo4rs = { git = "https://github.com/neo4j-labs/neo4rs.git", branch = "0.7" }

Note that server side routing on Aura is only enabled for database version 5; databases with version 4 expect clients to do all routing and will continue to not work with this client for some time.

[nixxholas]

You're awesome @knutwalker! Do I have to add anything new in the existing initialisation of an instance? Seems like I'm still getting the same issue.

2024-07-31T19:23:41.875871Z  INFO neo4rs::pool: creating connection pool with max size 100    
Successfully connected to Neo4j!
2024-07-31T19:23:43.967005Z  INFO bird_api::clients::twitter: Ingesting 955 followings for weremeow
2024-07-31T19:23:43.967064Z  INFO bird_api::clients::twitter: Starting batch user ingestion process. Total users: 955
2024-07-31T19:23:43.970637Z  INFO neo4rs::pool: creating new connection...    
2024-07-31T19:23:46.871732Z  WARN bird_api::clients::twitter: Not a leader. Retrying in 1s. Error: unexpected response for RUN: Failure(Failure { metadata: BoltMap { value: {BoltString { value: "code" }: String(BoltString { value: "Neo.ClientError.Cluster.NotALeader" }), BoltString { value: "message" }: String(BoltString { value: "No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER" })} } })    
2024-07-31T19:23:46.999776Z  WARN neo4rs::connection: This client does not yet support client-side routing.The routing context passed as a query to the URI is ignored.    
2024-07-31T19:23:46.999799Z  INFO neo4rs::pool: creating connection pool with max size 100    
Successfully connected to Neo4j!
2024-07-31T19:23:46.999916Z  INFO neo4rs::pool: creating new connection...    
2024-07-31T19:23:49.995267Z  WARN bird_api::clients::twitter: Not a leader. Retrying in 1s. Error: unexpected response for RUN: Failure(Failure { metadata: BoltMap { value: {BoltString { value: "message" }: String(BoltString { value: "No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER" }), BoltString { value: "code" }: String(BoltString { value: "Neo.ClientError.Cluster.NotALeader" })} } })    
2024-07-31T19:23:50.123568Z  WARN neo4rs::connection: This client does not yet support client-side routing.The routing context passed as a query to the URI is ignored.    
2024-07-31T19:23:50.123594Z  INFO neo4rs::pool: creating connection pool with max size 100    
Successfully connected to Neo4j!
2024-07-31T19:23:50.123752Z  INFO neo4rs::pool: creating new connection...    
2024-07-31T19:23:53.029470Z  WARN bird_api::clients::twitter: Not a leader. Retrying in 1s. Error: unexpected response for RUN: Failure(Failure { metadata: BoltMap { value: {BoltString { value: "code" }: String(BoltString { value: "Neo.ClientError.Cluster.NotALeader" }), BoltString { value: "message" }: String(BoltString { value: "No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER" })} } })    
2024-07-31T19:23:53.177934Z  WARN neo4rs::connection: This client does not yet support client-side routing.The routing context passed as a query to the URI is ignored.    
2024-07-31T19:23:53.177967Z  INFO neo4rs::pool: creating connection pool with max size 100    
Successfully connected to Neo4j!
2024-07-31T19:23:53.178110Z  INFO neo4rs::pool: creating new connection...    
2024-07-31T19:23:56.166727Z  WARN bird_api::clients::twitter: Not a leader. Retrying in 1s. Error: unexpected response for RUN: Failure(Failure { metadata: BoltMap { value: {BoltString { value: "code" }: String(BoltString { value: "Neo.ClientError.Cluster.NotALeader" }), BoltString { value: "message" }: String(BoltString { value: "No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER" })} } })    
2024-07-31T19:23:56.300013Z  WARN neo4rs::connection: This client does not yet support client-side routing.The routing context passed as a query to the URI is ignored.    
2024-07-31T19:23:56.300044Z  INFO neo4rs::pool: creating connection pool with max size 100    
Successfully connected to Neo4j!

[knutwalker]

No, there should not not be a change in usage. My testing against an AuraDB 5 instance seemed to have fixed it 🤔
Do you mind sharing some more details with me? If you could tell me the database ID I can try to see if the DB is properly configured for SSR. And maybe some code around how you connect the driver (without credentials of course)?. You can find me on the community discord and DM me (also knutwalker over there) the details so you don't have to share them publicly.

[knutwalker]

0.7.3 added an empty routing table, indicating servers to use server-side routing and 0.8.0 added an automatic retry on certain errors (including NotALeader). The retry only works for Graph::run or Graph::execute, not for manually committed transactions (which mirrors how other drivers behave).

The server still needs to have SSR enabled (which requires AuraDB 5, it's not enabled for AuraDB 4) and one can still overload the server and have it fail often enough for the retries to give up, in which case only a recude in load would help.

We still need to implement client-side routing at some point.

[nixxholas]

@knutwalker Mind leading the way with a few links to Neo4J docs where we can correctly understand how CSR works?

[knutwalker]

@nixxholas There is https://neo4j.com/docs/operations-manual/current/clustering/setup/routing/#clustering-client-side-routing and https://neo4j.com/docs/bolt/current/driver-api/#client-side-routing and https://neo4j.com/docs/bolt/current/bolt/message/#messages-route

The ROUTE message requires Bolt protocol 4.3 and there is a bigger refactor in progress before we can move to newer protocol versions (and would also have to support all other changes that are required by this)