Can't perform a Backup on new instance after restoring a backup to a new instance

Neo4j Graph Platform Import / Export
,
1

Hi,

I am in the process of upgrading our neo4j stack to 3.5.16 (Running via aws [Neo4j Enterprise Causal Cluster] Subscription) to make it easier i just created a new Causal Cluster and performed an import of the backup i made on the older instance (3.5.12).

The reason for the move was we were previously getting this issue Inconsistency report NullPointerException consistency after full backup · Issue #12337 · neo4j/neo4j · GitHub

To restore i ran the following (on all 3 servers), there seemed to be some permissions issues on the restored db so i fixed those too.

sudo -u neo4j neo4j-admin unbind
sudo neo4j-admin restore --from=/var/lib/neo4j/backups/graph.db-backup-manual --database=graph.db --force
sudo chown -R neo4j /var/lib/neo4j/data/databases/graph.db/

At this point the DB came up with all the data cluster overview is showing all 3 instances, i can query data and use the database perfectly fine.

Now when trying to perform the backup it its 30% and then just hangs, CPU usage sitting on 100%

root@ip-10-0-0-50:/home/ubuntu# neo4j-admin backup --from=127.0.0.1 --backup-dir=/var/lib/neo4j/backups/ --name=graph.db-backup-manual --pagecache=2G
2020-03-30 10:27:00.738+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
// Omitted for brevity
2020-03-30 10:27:22.750+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish receiving transactions at 7293882, took 16s 252ms
Full Consistency Check
....................  10%
....................  20%
....................  30%
..........

I have tried to change the heap size in case that was the issue

export JAVA_OPTS='-Xms8192m -Xmx8192m'

At this point i am kind of out of ideas not sure what is exactly the issue, apart from what the neo4j-admin backup command outputs is there any other log i can check to see if there is an isssue?

2

I've had similar issues before, when pushing a database into an Aura cluster.

Have you considered trying to restore the backup into a local graph running through Neo4j desktop? I can't say for certain but you might get some additional debug info that could be helpful (in our case, it was a corrupted index).

3

Will give that a try.

4

Interestingly I get the exact same issue if i restore to my local graph and try and run a backup.

Your corrupt index thing got me thinking, so i removed all the indexes on the database and tried to run the backup again.

This time it gets further, but then seems to run out of memory, not sure what i need to change to see if it completes.

F:\neo4j\neo4jDatabases\database-8c93f0af-eab1-468d-adeb-90fa81518bb6\installation-3.5.16\bin>neo4j-admin backup --from=127.0.0.1 --backup-dir=F:\neo4j\backup --name=graph.db-backup-manual --pagecache=4G
2020-03-31 11:23:10.355+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
//removed for brevity
2020-03-31 11:23:27.806+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish receiving store files, took 17s 449ms
2020-03-31 11:23:31.940+0000 INFO [o.n.b.i.BackupOutputMonitor] Start recovering store
2020-03-31 11:23:48.639+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish recovering store, took 16s 699ms
2020-03-31 11:23:54.052+0000 INFO [o.n.b.i.BackupOutputMonitor] Finished, took 43s 697ms
Full Consistency Check
....................  10%
....................  20%
....................  30%
....................  40%
....Exception in thread "ParallelRecordScanner-Stage8_PS_Props-2" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-3" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-6" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
        at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
        at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
        at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706434560 bytes committed virtual memory, and 9467637760 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
        at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 22 more
Exception in thread "ParallelRecordScanner-Stage8_PS_Props-1" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-5" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706328064 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
        at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
        at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
        at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706221568 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
        at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 22 more
Exception in thread "ParallelRecordScanner-Stage8_PS_Props-4" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-0" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706106880 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
        at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 22 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706000384 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
        at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
        at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
        at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
        at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1705893888 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
        at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
        at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
        at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
        at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
        at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
        at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
        at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
        at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
        at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
        at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
        at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
        at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
        at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
        ... 22 more

what is even more interesting if i add all the indexes that i removed back, then the backup gets stuck again at 30% point like before but just sits and hangs no errors

5

Anyone have any ideas that i can try?

6

I am actually still stuck with this :worried:

7

Hi, You can try to change the parameters in neo4j.conf file. The parameters are:

dbms.memory.heap.initial_size=3500m
dbms.memory.heap.max_size=3500m
dbms.memory.pagecache.size=3000m

The values depends your RAM size

Thanks