resource competetion when setting up multiple nodes in one machine?

Hi,

I use three Quad-Core machines,and I set up multiple nodes in each machine.Then I test tpc-h data in the system.

I create index in orderkey both in lineitem table and orders table.The join time of table lineitem and orders as below:
(the sql:select count(*) from lineitem_1g l join orders_1g o on l.orderkey=o.orderkey,three nodes means each machine just has one node,6nodes means each machine has two nodes,12nodes means each machine has four nodes)

Gridsql(3nodes) Gridsql(6nodes) Gridsql(12nodes)
1g 11s 683ms 5s 940ms 2s 117ms
5g 19s 117ms 10s 130ms 6s 676ms
10g 37s 301ms 20s 303ms 11s 820ms
15g 2m 21s 330ms 7m 5s 989ms 5m 44s 988ms

when the data is less tan 15g, it works good.But when the data is up to 15g,high degree of parallelism work worse.

Maybe it is caused by resouce competetion in postgresql?

Thankyou~

Xiao

when the data is less tan 15g, it works good.But when the data is up to 15g,high degree of parallelism work worse.

It seems that you don't have proper configuration of Underlying PostgreSQL database. You may require to tune the underlying databases too.

Also, you may require to tune co-ordinator/agent process too(i.e. allocating more resources for processing)

People with >1TB of data are using GridSQL and gridSQL is performing well in those environment.

Hi Xiao,

I am not sure. One often observes better performance on larger data sets because of more caching. Did you partition orders on orderkey and lineitem on orderkey? Are you sure the 15 g test was setup properly?

Regards,

Mason

I am sorry, I did not read your original post carefully. You are using 3 servers and increasing the number of logical nodes on each.

Again, I would be curious to know if you specified a partitioning column for each and which column. My next question is then if each logical node has its own dedicated storage. If they are sharing the same device, there could be some IO contention, especially at 15 GB if the data no longer fits in memory.

Regards,

Mason