I have created a custom dockerfile for neo4j that imports necessary plugins. Next I created a deployment yaml which I am running on Kubernetes on Azure. The deployment works very well and I can reach the browser on the expected URL, which works great. However,
- the moment I attach a Volume, neo4j becomes unable to start with the below provided error stack.
- There is data in the mount volume (azure files), but it is only
/data/dbms
, not/data/databases
. Probably because the last directory already comes pre-filled image? in
How would I be able to initialize Neo4j successfully and start mapping the content of /data/databases/ when it data is added to the graph?
Find below the error stack and deployment yaml.
Deployment yaml
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: mygraph
ports:
- name: bolt
protocol: TCP
port: 7687
targetPort: 7687
nodePort: 30076
- name: http
protocol: TCP
port: 7474
targetPort: 7474
nodePort: 30074
- name: https
protocol: TCP
port: 7473
targetPort: 7473
nodePort: 30073
type: LoadBalancer
loadBalancerIP: 40.118.172.46
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mygraph-deployment
labels:
app: mygraph
spec:
replicas: 1
selector:
matchLabels:
app: mygraph
template:
metadata:
labels:
app: mygraph
spec:
containers:
- name: mygraph
image: mycontainer.azurecr.io/mygraph:dev
securityContext:
privileged: true
ports:
- containerPort: 7687
- containerPort: 7474
- containerPort: 7473
env:
- name: NEO4J_dbms_jvm_additional
value: "-Xms512m -Xmx6g"
# When adding the below lines the deployment failes. Without it, it works great!
volumeMounts:
- name: neo4jdata
mountPath: /var/lib/neo4j/data/
volumes:
- name: neo4jdata
persistentVolumeClaim:
claimName: azurefile
Error stack
2019-09-05 07:02:12.522+0000 INFO ======== Neo4j 3.3.9 ========
2019-09-05 07:02:12.560+0000 INFO Starting...
2019-09-05 07:02:14.122+0000 INFO Bolt enabled on 0.0.0.0:7687.
2019-09-05 07:02:14.274+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:220)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:211)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:126)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:58)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:88)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter@134d26af' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:207)
... 9 more
Caused by: org.neo4j.kernel.StoreLockException: Unable to create path for store dir: /var/lib/neo4j/data/databases/graph.db. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
at org.neo4j.kernel.internal.locker.StoreLocker.storeLockException(StoreLocker.java:116)
at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:79)
at org.neo4j.kernel.internal.locker.GlobalStoreLocker.checkLock(GlobalStoreLocker.java:60)
at org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:36)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
... 11 more
Caused by: java.io.IOException: Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.
at org.neo4j.io.fs.DefaultFileSystemAbstraction.mkdirs(DefaultFileSystemAbstraction.java:124)
at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:73)
... 14 more
2019-09-05 07:02:14.317+0000 INFO Neo4j Server shutdown initiated by request