Neo4j Transaction Retention Policy not working;

[benjamin.squire]

Neo4j Transaction Retention Policy not adhering to setting; Enterprise 3.5.7

• neo4j version: 3.5.7 Enterprise
• neo4j.log and debug.log

I changed my configuration file to have the following transaction retention policy to avoid filling up my disk with old transactions.

dbms.tx_log.rotation.retention_policy=15 files

Here is what my graph.db folder currently looks like:

-rw-r--r-- 1 root root 259M Aug 19 16:00 neostore.transaction.db.1410                                                                                                                         [264/462]
-rw-r--r-- 1 root root 259M Aug 19 16:01 neostore.transaction.db.1411
-rw-r--r-- 1 root root 287M Aug 19 16:01 neostore.transaction.db.1412
-rw-r--r-- 1 root root 259M Aug 19 16:02 neostore.transaction.db.1413
-rw-r--r-- 1 root root 259M Aug 19 16:02 neostore.transaction.db.1414
-rw-r--r-- 1 root root 259M Aug 19 16:03 neostore.transaction.db.1415
-rw-r--r-- 1 root root 259M Aug 19 16:04 neostore.transaction.db.1416
-rw-r--r-- 1 root root 259M Aug 19 16:04 neostore.transaction.db.1417
-rw-r--r-- 1 root root 259M Aug 19 16:05 neostore.transaction.db.1418
-rw-r--r-- 1 root root 259M Aug 19 16:05 neostore.transaction.db.1419
-rw-r--r-- 1 root root 259M Aug 19 16:06 neostore.transaction.db.1420
-rw-r--r-- 1 root root 273M Aug 19 16:06 neostore.transaction.db.1421
-rw-r--r-- 1 root root 259M Aug 19 16:07 neostore.transaction.db.1422
-rw-r--r-- 1 root root 259M Aug 19 16:08 neostore.transaction.db.1423
-rw-r--r-- 1 root root 259M Aug 19 16:08 neostore.transaction.db.1424
-rw-r--r-- 1 root root 259M Aug 19 16:08 neostore.transaction.db.1425
-rw-r--r-- 1 root root 259M Aug 19 16:09 neostore.transaction.db.1426
-rw-r--r-- 1 root root 259M Aug 19 16:10 neostore.transaction.db.1427
-rw-r--r-- 1 root root 259M Aug 19 16:10 neostore.transaction.db.1428
-rw-r--r-- 1 root root 259M Aug 19 16:11 neostore.transaction.db.1429
-rw-r--r-- 1 root root 259M Aug 19 16:12 neostore.transaction.db.1430
-rw-r--r-- 1 root root 259M Aug 19 16:12 neostore.transaction.db.1431
-rw-r--r-- 1 root root 259M Aug 19 16:13 neostore.transaction.db.1432
-rw-r--r-- 1 root root 259M Aug 19 16:14 neostore.transaction.db.1433
-rw-r--r-- 1 root root 259M Aug 19 16:14 neostore.transaction.db.1434
-rw-r--r-- 1 root root 259M Aug 19 16:15 neostore.transaction.db.1435
-rw-r--r-- 1 root root 259M Aug 19 16:16 neostore.transaction.db.1436
-rw-r--r-- 1 root root 259M Aug 19 16:16 neostore.transaction.db.1437
-rw-r--r-- 1 root root 259M Aug 19 16:17 neostore.transaction.db.1438
-rw-r--r-- 1 root root 259M Aug 19 16:17 neostore.transaction.db.1439
-rw-r--r-- 1 root root 259M Aug 19 16:18 neostore.transaction.db.1440
-rw-r--r-- 1 root root 259M Aug 19 16:19 neostore.transaction.db.1441
-rw-r--r-- 1 root root 259M Aug 19 16:19 neostore.transaction.db.1442
-rw-r--r-- 1 root root 273M Aug 19 16:20 neostore.transaction.db.1443
-rw-r--r-- 1 root root 259M Aug 19 16:21 neostore.transaction.db.1444
-rw-r--r-- 1 root root 259M Aug 19 16:21 neostore.transaction.db.1445
-rw-r--r-- 1 root root 259M Aug 19 16:23 neostore.transaction.db.1446
-rw-r--r-- 1 root root 259M Aug 19 16:23 neostore.transaction.db.1447
-rw-r--r-- 1 root root 259M Aug 19 16:24 neostore.transaction.db.1448
-rw-r--r-- 1 root root 259M Aug 19 16:25 neostore.transaction.db.1449
-rw-r--r-- 1 root root 259M Aug 19 16:25 neostore.transaction.db.1450
-rw-r--r-- 1 root root 259M Aug 19 16:26 neostore.transaction.db.1451
-rw-r--r-- 1 root root 259M Aug 19 16:27 neostore.transaction.db.1452
-rw-r--r-- 1 root root 259M Aug 19 16:28 neostore.transaction.db.1453
-rw-r--r-- 1 root root 259M Aug 19 16:28 neostore.transaction.db.1454
-rw-r--r-- 1 root root 259M Aug 19 16:29 neostore.transaction.db.1455
-rw-r--r-- 1 root root 259M Aug 19 16:30 neostore.transaction.db.1456
...
-rw-r--r-- 1 root root 258M Aug 19 18:47 neostore.transaction.db.1674
-rw-r--r-- 1 root root 258M Aug 19 18:48 neostore.transaction.db.1675
-rw-r--r-- 1 root root 258M Aug 19 18:48 neostore.transaction.db.1676
-rw-r--r-- 1 root root 258M Aug 19 18:48 neostore.transaction.db.1677
-rw-r--r-- 1 root root 258M Aug 19 18:49 neostore.transaction.db.1678
-rw-r--r-- 1 root root 258M Aug 19 18:49 neostore.transaction.db.1679
-rw-r--r-- 1 root root 258M Aug 19 18:50 neostore.transaction.db.1680
-rw-r--r-- 1 root root 258M Aug 19 18:51 neostore.transaction.db.1681
-rw-r--r-- 1 root root 258M Aug 19 18:51 neostore.transaction.db.1682
-rw-r--r-- 1 root root 258M Aug 19 18:52 neostore.transaction.db.1683
-rw-r--r-- 1 root root 258M Aug 19 18:52 neostore.transaction.db.1684
-rw-r--r-- 1 root root 258M Aug 19 18:53 neostore.transaction.db.1685
-rw-r--r-- 1 root root 271M Aug 19 18:54 neostore.transaction.db.1686
-rw-r--r-- 1 root root 271M Aug 19 18:54 neostore.transaction.db.1687
-rw-r--r-- 1 root root 258M Aug 19 18:55 neostore.transaction.db.1688
-rw-r--r-- 1 root root 258M Aug 19 18:55 neostore.transaction.db.1689
-rw-r--r-- 1 root root 258M Aug 19 18:56 neostore.transaction.db.1690
-rw-r--r-- 1 root root 258M Aug 19 18:56 neostore.transaction.db.1691
-rw-r--r-- 1 root root 258M Aug 19 18:57 neostore.transaction.db.1692
-rw-r--r-- 1 root root 258M Aug 19 18:57 neostore.transaction.db.1693
-rw-r--r-- 1 root root 258M Aug 19 18:58 neostore.transaction.db.1694
-rw-r--r-- 1 root root 258M Aug 19 18:59 neostore.transaction.db.1695
-rw-r--r-- 1 root root 258M Aug 19 18:59 neostore.transaction.db.1696
-rw-r--r-- 1 root root 258M Aug 19 18:59 neostore.transaction.db.1697
-rw-r--r-- 1 root root 258M Aug 19 19:00 neostore.transaction.db.1698
-rw-r--r-- 1 root root 258M Aug 19 19:01 neostore.transaction.db.1699
-rw-r--r-- 1 root root 258M Aug 19 19:02 neostore.transaction.db.1700
-rw-r--r-- 1 root root 258M Aug 19 19:02 neostore.transaction.db.1701
-rw-r--r-- 1 root root 258M Aug 19 19:02 neostore.transaction.db.1702
-rw-r--r-- 1 root root 258M Aug 19 19:03 neostore.transaction.db.1703
-rw-r--r-- 1 root root 258M Aug 19 19:03 neostore.transaction.db.1704
-rw-r--r-- 1 root root 258M Aug 19 19:04 neostore.transaction.db.1705
-rw-r--r-- 1 root root 258M Aug 19 19:04 neostore.transaction.db.1706
-rw-r--r-- 1 root root 258M Aug 19 19:05 neostore.transaction.db.1707
-rw-r--r-- 1 root root   16 Aug 19 19:05 neostore.transaction.db.1708

There are literally hundreds of transaction logs. Posting as a bug in Neo4j issues as well.

Thoughts?

[dana_canzano]

transaction logs are rotated when a checkoint is run.
Does you logs/debug.log report a checkpoint has been run?

With Neo4j 3.5.6 you can force a checkpoint via Cypher by running call dbms.checkpoint(); though you generally should not need to call this as checkpoints will automatically run every 100k transactions or every 15 minutes.

[benjamin.squire]

I see check points being run:
Rotated to transaction log [/mnt/lv1/data/databases/graph.db/neostore.transaction.db.1822] version=1822, last transaction in previous log=1733584
What I am saying is that the retention policy of 15 files and pruning unneeded transaction files is not being followed. I don't want 100's of GB of transaction logs

[dana_canzano]

given your initial description which included

-rw-r--r-- 1 root root 287M Aug 19 16:01 neostore.transaction.db.1412
-rw-r--r-- 1 root root 259M Aug 19 16:02 neostore.transaction.db.1413
-rw-r--r-- 1 root root 259M Aug 19 16:02 neostore.transaction.db.1414
-rw-r--r-- 1 root root 259M Aug 19 16:03 neostore.transaction.db.1415

so these 4 files, each some 259M large were created in a span of less than 2 minutes (i.e. 4 txn logs over 2 minutes)
So even if we checkpoint every 15 minutes, and if the same transaction activity happended then you would have 30 txn log files, thus exceeding your retention of 15 files. But at checkpoint we should then only keep the most recent 15 files.

also out of curiosity why only 15 files?
Generally txn log files are only necessary to support incremental backup.

[benjamin.squire]

Really I could be fine with not keeping any. Should I change the transaction retention policy to False? Should I change the checkpoint to check more often? I am ending up with recovery times that can run for hours which is a major thing I am trying to avoid.

[dana_canzano]

per https://neo4j.com/docs/operations-manual/3.5/configuration/transaction-logs/

dbms.tx_log.rotation.retention_policy=<true/false>

If this parameter is set to false, only the most recent non-empty log will be kept. This option is not recommended in production Enterprise Edition environments, as incremental backups rely on the presence of the transaction logs since the last backup.

[benjamin.squire]

I read that but would this only affect incremental back ups? What about recovery? There are times when unclean shutdowns happen. Would we still be able to recover in these scenarios?