"forces all dependent data to be materialized in main memory before proceeding" - Upload CSV file with 30K nodes

Hello, I'm trying to process a big file with data from multiple monitoring stations. I've found a solution, but the same thing that Neo4j's warning tells me "the execution plan for this query contains the Eager operator, which forces all dependent data to be materialized in main memory before proceeding", I am also not satisfied with this query. Any help is going to be great!

LOAD CSV WITH HEADERS FROM "file:///data.csv" as obj
MATCH (n:Station) WHERE (obj.SNIRH CONTAINS n.SNIRH) OR (n.EDIA=obj.EDIA)
WITH n, obj
CREATE (data:Data)
SET data += obj
MERGE (data)-[r:COLLECTED]->(n)
RETURN count(r) as Relations, count(data) as Data

Have you tried using PERIODIC COMMIT? I try to use it whenever loading anything more than a trivial dataset. I see some notes in there about Eager operator as well.

1 Like