Interrupted system call

[tim-hanssen]

We are looking at an issue that all of our workers in (in this case Laravel Horizon) are sometimes crashing jobs due to the "Interrupted system call" error.

• OS: PHP 8.4 CLI (docker container)
• Neo4j Version 4.4
• Bolt Library Version 7.2.4

It looks like the issues started when we updated the Laudis PHP client last week, using bolt driver 7.3.1. But we also experience this with the older 7.1.4. We think it has something to do with the connection pools being reused, but it now happens so often, we are a bit in the dark on what is triggering it. So maybe @stefanak-michal you have any idea what we could look at.

[stefanak-michal]

Hi. Do you have any stack trace of this error? Or at least some php error line by which function is it caused?

[tim-hanssen]

Hey @stefanak-michal sorry somehow that got lost in my initial issue:

Bolt\error\ConnectException: Interrupted system call in /code/vendor/stefanak-michal/bolt/src/connection/Socket.php:129
--
Stack trace:
#0 /code/vendor/stefanak-michal/bolt/src/connection/Socket.php(84): Bolt\connection\Socket->throwConnectException()
#1 /code/vendor/stefanak-michal/bolt/src/protocol/AProtocol.php(83): Bolt\connection\Socket->read()
#2 /code/vendor/stefanak-michal/bolt/src/protocol/v4/PullMessage.php(36): Bolt\protocol\AProtocol->read()
#3 [internal function]: Bolt\protocol\V4_4->_pull()
#4 /code/vendor/stefanak-michal/bolt/src/protocol/AProtocol.php(145): Generator->current()
#5 /code/vendor/stefanak-michal/bolt/src/protocol/AProtocol.php(123): Bolt\protocol\AProtocol->getResponse()
#6 /code/vendor/laudis/neo4j-php-client/src/Bolt/BoltConnection.php(317): Bolt\protocol\AProtocol->getResponses()
#7 /code/vendor/laudis/neo4j-php-client/src/Bolt/BoltResult.php(102): Laudis\Neo4j\Bolt\BoltConnection->pull()
#8 /code/vendor/laudis/neo4j-php-client/src/Bolt/BoltResult.php(79): Laudis\Neo4j\Bolt\BoltResult->fetchResults()
#9 [internal function]: Laudis\Neo4j\Bolt\BoltResult->iterator()

[stefanak-michal]

I did some research and it would be possible to add retry when SOCKET_EINTR occurs.

[stefanak-michal]

@tim-hanssen it's possible for you to try master branch and see if the problem was resolved? Or do you need me to make release?

[tim-hanssen]

Hey @stefanak-michal I can try that! :) i let you know.

[tim-hanssen]

Hey @stefanak-michal I put the new version in the app, but same issue:

 #0 /code/vendor/stefanak-michal/bolt/src/connection/Socket.php(86): Bolt\connection\Socket->throwConnectException()
#1 /code/vendor/stefanak-michal/bolt/src/protocol/AProtocol.php(83): Bolt\connection\Socket->read()
#2 /code/vendor/stefanak-michal/bolt/src/protocol/v3/RunMessage.php(37): Bolt\protocol\AProtocol->read()
#3 [internal function]: Bolt\protocol\V4_4->_run()
#4 /code/vendor/stefanak-michal/bolt/src/protocol/AProtocol.php(145): Generator->current()
#5 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Bolt/BoltConnection.php(256): Bolt\protocol\AProtocol->getResponse()
#6 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Bolt/BoltUnmanagedTransaction.php(138): Laudis\Neo4j\Bolt\BoltConnection->run()
#7 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Bolt/Session.php(73): Laudis\Neo4j\Bolt\BoltUnmanagedTransaction->runStatement()
#8 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Client.php(110): Laudis\Neo4j\Bolt\Session->runStatements()
#9 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Client.php(80): Laudis\Neo4j\Client->runStatements()
#10 /code/vendor/prepr/neo4j-php-client-laudis-rest6/src/Client.php(75): Laudis\Neo4j\Client->runStatement()
#11 /code/app/Models/Model.php(71): Laudis\Neo4j\Client->run()

[stefanak-michal]

Thanks for trying it out. That is a strange and persistent problem. I will think more about it.

[stefanak-michal]

@transistive hi, would you be so nice and please explain a bit how you are pooling connections in your library?

[stefanak-michal]

@tim-hanssen my driver also contains another connection class you could try. But I'm not sure how to tell in neo4j client to use it. @transistive is it possible to tell client which connection class to use from driver?

[stefanak-michal]

@tim-hanssen error code can vary based on platform. So i updated the fix a little bit. Could you try master branch again?

I'm putting it into master so it will trigger workflows with tests. I don't have access to computer these days, therefore i cannot run those tests myself. i do everything on the phone.

[transistive]

Hello @stefanak-michal ,

Thank you, everyone, for your patience and for providing this information.

Connection pooling relies heavily on the Server State Protocol to determine which connections are safe to reuse, and each connection is lazily initialised.

A driver maintains a pool of connections, creating a new one if none are available, up to the configured limit.

Some of the logs shared with me through other channels indicate a Bolt protocol ServerState mismatch so that I will start my investigation there. Since we dug deep into TestKit, we’ve identified a few subtle state-handling issues, and it looks like this one may still exist.

I have blocked out some time tomorrow to work on the problem, as the driver’s connection lifecycle and pooling logic are primary areas of investigation.

In the meantime, @tim-hanssen, it would be invaluable to get Neo4j server logs that correlate with these issues. We may find connection resets or related information there that helps narrow this down further.

Regarding your next question, @stefanak-michal, it should be a trivial change on our end to allow users to replace the Connection class. For now, the driver autodetects it from the PHP environment.

[transistive]

@stefanak-michal We have made it possible to select the socket instance manually through he driver configuration object. It is released under version 3.4.1neo4j-php/neo4j-php-client#284