Paralle data loading

Hi,

It is my first time to use GridSQL.
I tried to load 200G data into a parititioned table of a 20-nodes GridSQL cluster using gs-loader. It seems that gs-loader will send data to the coordinator, then the coordinator partitions the data to each agent. So the coordinator is a bottleneck during data loading.

Is there any parallel data loading tools to load the data parallely?

Regards,

Samuel

Hi Samuel,

You can try and set up multiple coordinators with the same nodes and the same metadata database. Do not run createmddb.sh or createdb.sh for them! They can perform parallel loading.
You will have a problem if you loading indexed tables or tables with autoincrement fields, including "with rowid" case.
However this trick should work for plain tables.

Andrei

I will try this tricky method. Thanks Andrei.

It seems that an agent is configured to connect just one coordinator.
If I try to setup up multi coordinators, It seems that I need to start multiple agents in each slave node. Is it right?

If I try to setup up multi coordinators, It seems that I need to start multiple agents in each slave node. Is it right?

One Agent will report to One Co-ordinator.

Samuel,

You can configure extra coordinators without agents, just specify xdb.node.N.dbhost in the main config file

Andrei

Hi Andrei,

A couple of questions about this:

1. Do you mean we set up multiple gridsql.config file each with the coordinator on a different port and run gs-server.sh pointing to each specific config file to startup multiple processes?

2. In this scenario, on the client side, I guess we need to make some changes on our client side to perhaps have multiple URLs for the target coordinator and perhaps use some logic to distribute the client load across different coordinators.

Thanks,

Ranga

Hi Ranga,

This trick only applies for initial loading.
If you configure multiple coordinators and access them concurrently with generic clients you may end up with corrupted database.

Andrei