A few days ago, I updated neo4j-enterprise to v4.4.6. The upgrade appears to have broken Neo4J so that my ingestor code -- that has been running smoothly for nearly 2 years -- hangs while doing things that have worked for a long time.
I've spent the last two days trying to diagnose the issue with little success.
I see some unusual entries in debug.log that I don't remember seeing before. I'm trying to pin down precisely what provokes them -- I think it is a call to neo4j-admin to load a database from a snapshot.
These entries seem to occur in either pairs or even quadruples. Here is an example:
2022-05-17 16:50:26.493+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 18. Encountered: <EOF> after : "", reference 0886e4cc-e4df-409a-81b4-2dbd94d6e6bb.
2022-05-17 16:50:26.493+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 18. Encountered: <EOF> after : "", reference 0886e4cc-e4df-409a-81b4-2dbd94d6e6bb.
org.neo4j.cypher.internal.parser.javacc.TokenMgrException: Lexical error at line 1, column 18. Encountered: <EOF> after : ""
at org.neo4j.cypher.internal.parser.javacc.CypherTokenManager.getNextToken(CypherTokenManager.java:7616) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.jj_ntk_f(Cypher.java:23635) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.SymbolicDatabaseNameStringOrParameter(Cypher.java:18004) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateDatabase(Cypher.java:17135) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateCommand(Cypher.java:7263) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.SingleQueryOrCommand(Cypher.java:288) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.Statement(Cypher.java:262) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.Statements(Cypher.java:188) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.ast.factory.neo4j.JavaCCParser$.parse(JavaCCParser.scala:54) ~[neo4j-cypher-ast-factory-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.helpers.Neo4jJavaCCParserWithFallback$.parse(Neo4jJavaCCParserWithFallback.scala:48) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:40) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.$anonfun$transform$1(Phase.scala:36) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.$anonfun$closing$1(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.using(package.scala:32) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.closing(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.transform(Phase.scala:35) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.transform$(Phase.scala:33) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.transform(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.fabric.pipeline.FabricFrontEnd$Pipeline$parseAndPrepare$.process(FabricFrontEnd.scala:139) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$computePlan$1(FabricPlanner.scala:87) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.trace(FabricPlanner.scala:127) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.computePlan(FabricPlanner.scala:86) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$plan$1(FabricPlanner.scala:79) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.cache.FabricQueryCache.computeIfAbsent(FabricQueryCache.scala:52) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan$lzycompute(FabricPlanner.scala:80) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan(FabricPlanner.scala:76) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.executor.FabricExecutor.run(FabricExecutor.java:123) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.bolt.BoltFabricDatabaseService$BoltTransactionImpl.executeQuery(BoltFabricDatabaseService.java:186) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractTransactionStateMachineSPI$AbstractBoltResultHandle.start(AbstractTransactionStateMachineSPI.java:128) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State.startExecution(TransactionStateMachine.java:530) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State$2.run(TransactionStateMachine.java:326) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:89) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:82) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.transaction.StatementProcessorTxManager.runQuery(StatementProcessorTxManager.java:92) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v4.runtime.InTransactionState.processRunMessage(InTransactionState.java:95) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:52) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:154) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:102) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:93) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:156) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:252) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:187) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:177) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:257) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:240) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [apoc-4.4.0.3-core.jar:4.4.0.3]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-05-17 16:50:27.650+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 25. Encountered: <EOF> after : "", reference 326d903b-31ee-413f-8ba0-6943bd994f4d.
2022-05-17 16:50:27.650+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 25. Encountered: <EOF> after : "", reference 326d903b-31ee-413f-8ba0-6943bd994f4d.
org.neo4j.cypher.internal.parser.javacc.TokenMgrException: Lexical error at line 1, column 25. Encountered: <EOF> after : ""
at org.neo4j.cypher.internal.parser.javacc.CypherTokenManager.getNextToken(CypherTokenManager.java:7616) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.jj_ntk_f(Cypher.java:23635) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.SymbolicDatabaseNameStringOrParameter(Cypher.java:18004) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateDatabase(Cypher.java:17135) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateCommand(Cypher.java:7263) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.SingleQueryOrCommand(Cypher.java:288) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.Statement(Cypher.java:262) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.parser.javacc.Cypher.Statements(Cypher.java:188) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.ast.factory.neo4j.JavaCCParser$.parse(JavaCCParser.scala:54) ~[neo4j-cypher-ast-factory-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.helpers.Neo4jJavaCCParserWithFallback$.parse(Neo4jJavaCCParserWithFallback.scala:48) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:40) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.$anonfun$transform$1(Phase.scala:36) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.$anonfun$closing$1(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.using(package.scala:32) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.helpers.package$.closing(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.transform(Phase.scala:35) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.Phase.transform$(Phase.scala:33) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.transform(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
at org.neo4j.fabric.pipeline.FabricFrontEnd$Pipeline$parseAndPrepare$.process(FabricFrontEnd.scala:139) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$computePlan$1(FabricPlanner.scala:87) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.trace(FabricPlanner.scala:127) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.computePlan(FabricPlanner.scala:86) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$plan$1(FabricPlanner.scala:79) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.cache.FabricQueryCache.computeIfAbsent(FabricQueryCache.scala:52) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan$lzycompute(FabricPlanner.scala:80) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan(FabricPlanner.scala:76) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.executor.FabricExecutor.run(FabricExecutor.java:123) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.fabric.bolt.BoltFabricDatabaseService$BoltTransactionImpl.executeQuery(BoltFabricDatabaseService.java:186) ~[neo4j-fabric-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractTransactionStateMachineSPI$AbstractBoltResultHandle.start(AbstractTransactionStateMachineSPI.java:128) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State.startExecution(TransactionStateMachine.java:530) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State$2.run(TransactionStateMachine.java:326) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:89) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:82) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.transaction.StatementProcessorTxManager.runQuery(StatementProcessorTxManager.java:92) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v4.runtime.InTransactionState.processRunMessage(InTransactionState.java:95) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:52) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:154) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:102) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:93) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:156) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:252) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:187) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:177) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:257) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:240) ~[neo4j-bolt-4.4.6.jar:4.4.6]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [apoc-4.4.0.3-core.jar:4.4.0.3]
at java.lang.Thread.run(Thread.java:829) [?:?]
I don't know enough about neo4j to know what this is trying to tell me.
I also notice that the most recent labs plugin is apoc-4.4.0.3-core.jar, while neo4j-enterprise is at version 4.4.6. My impression was that this apoc plugin was expected to track the running version.
I mention this because I see the following complaints in debug.log after each startup:
2022-05-16 20:49:14.173+0000 INFO [c.n.c.n.Server] [backup-server] bound to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'
2022-05-16 20:49:15.296+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.google.cloud.GCStorageURLConnection` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/cloud/storage/Storage
2022-05-16 20:49:15.383+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.hdfs.HDFSUtils` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataOutputStream
2022-05-16 20:49:15.384+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.hdfs.HDFSUtils$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
2022-05-16 20:49:15.434+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3Aws` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials
2022-05-16 20:49:15.435+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3Aws$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/services/s3/model/S3ObjectInputStream
2022-05-16 20:49:15.437+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3URLConnection` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/ClientConfiguration
2022-05-16 20:49:15.438+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3OutputStream$S3UploadManager` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: Lcom/amazonaws/services/s3/AmazonS3;
2022-05-16 20:49:15.848+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeViewer$VariableExtentProvide` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/NodeExtentProvider
2022-05-16 20:49:15.849+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreePostScriptGenerator` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/Configuration
2022-05-16 20:49:15.989+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeViewer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/Configuration
2022-05-16 20:49:16.017+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreePostScriptGenerator$VariableExtentProvide` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/NodeExtentProvider
2022-05-16 20:49:16.026+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.unicode.UnicodeDataTemplateController` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/ibm/icu/text/UnicodeSet
2022-05-16 20:49:16.118+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeLayoutAdaptor` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/TreeForTreeLayout
2022-05-16 20:49:17.285+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.json.GsonJsonProvider` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/JsonElement
2022-05-16 20:49:17.314+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.json.JsonOrgJsonProvider` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/json/JSONException
2022-05-16 20:49:17.397+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.mapper.GsonMappingProvider$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/Gson
2022-05-16 20:49:17.398+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.mapper.GsonMappingProvider$2` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/Gson
2022-05-16 20:49:17.571+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodePointer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointer
2022-05-16 20:49:17.572+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorBase` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.573+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorAttribute` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.573+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorChildren` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.574+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodePointerFactory` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointerFactory
2022-05-16 20:49:17.575+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.XPathExpressionEngine` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointerFactory
2022-05-16 20:49:17.575+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.XPathContextFactory` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/JXPathContext
2022-05-16 20:49:17.576+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationAttributePointer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointer
2022-05-16 20:49:17.839+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.io.VFSFileSystem` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/vfs2/FileSystemException
2022-05-16 20:49:17.905+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.interpol.ExprLookup` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jexl2/JexlContext
2022-05-16 20:49:17.911+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.reloading.VFSFileHandlerReloadingDetector` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/vfs2/FileSystemException
2022-05-16 20:49:17.937+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/CatalogManager
2022-05-16 20:49:17.938+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver$Catalog` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/Catalog
2022-05-16 20:49:17.938+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver$CatalogManager` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/CatalogManager
2022-05-16 20:49:17.939+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.spring.ConfigurationPropertySource` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/springframework/core/env/EnumerablePropertySource
2022-05-16 20:49:17.940+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.spring.ConfigurationPropertiesFactoryBean` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/springframework/beans/factory/InitializingBean
2022-05-16 20:49:18.308+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `module-info` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: module-info is not a class because access_flag ACC_MODULE is set
2022-05-16 20:49:20.634+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.xerces.util.XMLCatalogResolver` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/readers/CatalogReader
2022-05-16 20:49:20.838+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.runtime.tree.DOTTreeGenerator` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/antlr/stringtemplate/StringTemplate
2022-05-16 20:49:21.039+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.AvalonLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/avalon/framework/logger/Logger
2022-05-16 20:49:21.041+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.Log4JLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log4j/Priority
2022-05-16 20:49:21.043+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.LogKitLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log/Logger
2022-05-16 20:49:22.344+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.logging.Log4JLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log4j/Priority
2022-05-16 20:49:22.360+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.Hidden$NettyBlockHoundIntegration$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: reactor/blockhound/integration/BlockHoundIntegration
2022-05-16 20:49:22.392+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.Hidden$NettyBlockHoundIntegration` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: reactor/blockhound/integration/BlockHoundIntegration
2022-05-16 20:49:22.399+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.logging.Log4J2Logger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/ExtendedLoggerWrapper
2022-05-16 20:49:22.535+0000 INFO [c.n.m.g.GlobalMetricsExtension] Sending metrics to CSV file at /var/lib/neo4j/metrics
...
2022-05-17 16:50:13.772+0000 WARN [c.n.m.g.GlobalMetricsExtension] Unable to remove database specific metric files because metric namespaces are not enabled. In order to remove this warning you have to set 'metrics.namespaces.enabled=true' in your configuration file.
So far as I know, I don't have, don't want, and don't use a backup server (regarding the first complaint). I also see no mention of metrics.namespaces.enabled anywhere in neo4.conf or the documentation (regarding the last complaint)
My users expect this site to be updated every day. I've been unable to do so since 11-May (almost a week now).
I welcome any guidance the Neo4J community might offer.