I'm using the ETL tool to map my organization's relational database. Their database has decades of data across roughly 7000 tables.
I am currently doing the mapping step (not the importing) and it has been running for 6 hours.
I'm totally fine with waiting but just wanted to ask if the mapping step is potentially a long process? If not I can kill the process and investigate further.
neo4j browser: 3.2.17
ETL tool 1.3.6
Thanks very much,
Not sure what to expect, but it ran the mapping step for 2 days and it hadn't finished. I assumed something was wrong so I cancelled it and will investigate further.
Any advice as to whether or not the mapping stage should take minutes, hours, or days would be much appreciated.
Even for 1 table also It is not proceeding further after Carawling.. step.
Since this is an old post, I am not sure, if you question was already answered ....
(1) What is the database size ?
(2) What is the size of each tables ?
I am also having the same issue regarding the etl tool . Can anyone please answer that how long would it take to map around 1000 tables.
Hi @susmitadaskgp, there is no static value. It all depends on your Data Model with Entities and Relationships.
What database are you using ? Before mapping did you had a check with the Data Dictionary ?
I highly recommend using a small sampling of the existing data before doing a massive import.
It may take multiple tries to debug it, so there will be occasions where you'll be waiting for a long time, only to discover the data wasn't imported correctly.
Also, you can get an estimate for the total time. i.e. if you load 1% of the data, you can estimate the total time by multiplying by 100.
The other thing I recommend is to clone the DB at each step as a backup if you do multiple steps. That way, if a step messes up the data, then you can go back to the previously good DB.
Hi @susmitadaskgp, are you migrating a OLTP or a OLAP database ? Maybe not all the 1000 tables, you have to migrate. You can create intermediate views also in the source database. There are always so many options to choose from.
Always have discovery and identifications of entities process first. This will save a lot of time.
@susmitadaskgp, like I guess, I think you are migrating an OLAP database. Sure, I can help you. Before I can advise more on this, I need more in-depth knowledge on the data. If you want we can connect in skype dominicvivek06 , and discuss there further.