I am suing LOAD CSV
and apoc.periodic.iterate
to import data. I have one .csv
that contains the folder and file names that contain the data to be imported. Below is the code I currently have:
CALL apoc.periodic.iterate("
LOAD CSV WITH HEADERS FROM 'file:///NBIFilesListFiles.csv' AS line1
WITH collect('file:///'+line1.Folder+'/'+line1.File) as Files
UNWIND Files AS file
LOAD CSV WITH HEADERS FROM file AS row RETURN row",
"
MERGE (state:State {id: row.STATE_CODE_001})
MERGE (state)<-[:OF_STATE]-(county:County {id: row.COUNTY_CODE_003})
MERGE (county)<-[:OF_COUNTY]-(place:Place {id: row.PLACE_CODE_004})
MERGE (place)<-[:OF_PLACE]-(bridge:Bridge {id: row.STRUCTURE_NUMBER_008})
ON CREATE SET bridge.name = row.STRUCTURE_NUMBER_008,
bridge.latitude = row.LAT_016,
bridge.longitude = row.LONG_017,
bridge.yearbuilt = row.YEAR_BUILT_027,
bridge.deck_cond = row.DECK_COND_058,
bridge.superstructure_cond = row.SUPERSTRUCTURE_COND_059,
bridge.substructure_cond = row.SUBSTRUCTURE_COND_060,
bridge.channel_cond = row.CHANNEL_COND_061,
bridge.culvert_cond = row.CULVERT_COND_062,
bridge.structural_eval = row.STRUCTURAL_EVAL_067,
bridge.deck_geometry_eval = row.DECK_GEOMETRY_EVAL_068,
bridge.undclrence_eval = row.UNDCLRENCE_EVAL_069,
bridge.posting_eval = row.POSTING_EVAL_070,
bridge.waterway_eval = row.WATERWAY_EVAL_071,
bridge.appr_road_eval = row.APPR_ROAD_EVAL_072,
place.name = row.PLACE_CODE_004,
county.name = row.COUNTY_CODE_003,
state.name = row.STATE_CODE_001
",
{batchSize:1000,iterateList:true})
My goal is to use LOAD CSV
to load the .csv
that contains all the other file names to import and pass that into apoc.periodic.iterate
. Is that possible?
Right now what I have in the code above tends to bog down my machine and cause Neo4j Desktop to crash. I have adjusted these settings to try and help
dbms.memory.heap.initial_size=1G
dbms.memory.heap.max_size=2G