XDBServerException : Can not send data to Nodes

Hi,
when executing the following query in GridSQL i'm getting below error, can anyone suggest what went wrong

GridSQL -> select count(*) from TDR_MSC_060807;
SQLException: ERROR: Failed To Get Results For ( SQL , NodeURL) : ( SELECT COUNT(*) AS XCOL1 FROM TDR_MSC_060807 ) eQS Node 3 has aborted execution, cause is: com.edb.gridsql.exception.XDBServerException : Can not send data to Nodes

Thankx
Venkat

Hi Venkat,

This is usually a permissions error, where one node cannot access another. You may have to change the pg_hba.conf file on each node and configuring .pgpass; please see the Admin Guide. You can test this out by logging in on one node and trying to access the other directly and see if you get prompted for a password. If you do get prompted, then the permissions are not set up properly.

Regards,

Mason

Mason_S wrote:Hi Venkat,

This is usually a permissions error, where one node cannot access another. You may have to change the pg_hba.conf file on each node and configuring .pgpass; please see the Admin Guide. You can test this out by logging in on one node and trying to access the other directly and see if you get prompted for a password. If you do get prompted, then the permissions are not set up properly.

Regards,

Mason

Hi Mason,

I am having the same problem. I have all nodes configured on the same box. As a matter of fact, it looks like all queries execute fine, except for those that contain aggregate functions: those all fail with the above error.

Thanks,
Vlad.

Vlad_O wrote:Hi Mason,

I am having the same problem. I have all nodes configured on the same box. As a matter of fact, it looks like all queries execute fine, except for those that contain aggregate functions: those all fail with the above error.

Thanks,
Vlad.

Sorry for the complaint, I figured it out

Just FYI, perhaps it will help somebody else: the underlying database's log file contained "FATAL: sorry, too many clients already" error. So the problem was that the max_connections parameter in the postgresql.conf was not high enough. I ignored your recommendations regarding this parameter in the Admin Guide, sorry.

This message was edited 1 time. Last update was at 3 Apr 2009 07:15:17

Hi Mason,

I'm still getting the same problem. I'm able to connect to the node database from all other locations and servers.

May be permissions is one issue but coming to my scenario i think that's not problem with permissions.

And also there is no problem from SQL end. CREATE table AS is one situation where i'm getting this problem.

Thanks,
Venkat.

Hi Venkat,

If you get the error everytime for queries like SELECT COUNT(*)..., then it is almost definitely a permissions problem.

If it just happens from time to time for really large queries, then it could be a resource issue.

Does it happen everytime for SELECT COUNT(*)? If so, try changing pg_hba.conf to allow connections from everywhere, and restarting Postgres/Advanced Server. You can always tighten up permissions later.

Thanks,

Mason

Hi Mason,

I'm not getting the error everytime.

Most of the time i see the error with Create Table AS statement. As i have 20 to 30k records in my test tables, while fetching the data from those tables also i'm getting the same error. I'm trying to trace the problem but it's like once in a while. But still i'll just come with clear scenario where i'm facing this problem for maximum number of times.

Thanks,
Venkat.

Hi Venkat,

Which version are you using? Are you using the latest from CVS?

Thanks,

Mason

Hi Mason,

Yes we are using the latest ones.
(PostgresPlus 8.3.3 GridSQL 1.0)

Thanks,
Venkat

Hi Venkat,

I think there was a subtle timing issue or two that were fixed in 1.1. Please try the latest code from CVS and see if it helps.

Thanks,

Mason

This message was edited 1 time. Last update was at 9 Apr 2009 09:58:51

Hi Mason,

We are using the very recent taken from CVS.

And here is one case where we got the same error:

GridSQL -> create table csdtn_test (col1 varchar(100),col2 varchar(20),col3 date,col4 NUMERIC(17,2),col5 integer,col6 VARCHAR(5)) partitioning key col1 on all tablespace tbs_grid;
OK
Response time: 0m 0s 164ms Total time: 0m 0s 164ms

GridSQL -> copy csdtn_test from '/home/postgress/test.csv' delimiter ',' CSV;
OK
Response time: 0m 0s 429ms Total time: 0m 0s 429ms

GridSQL -> select count(*) from csdtn_test;
+----------+
| count(*) |
+----------+
| 18000 |
+----------+
1 row(s).
Response time: 0m 0s 133ms Total time: 0m 0s 133ms

GridSQL -> create table test_csdtn partitioning key col1 on all tablespace tbs_chennai_grid as select col1,case when ltrim(rtrim(col6))='4' then 'ABC' else 'DEF' end ST,sum(case when col4=0 then 0 else col5 end) from csdtn_test where col2='NW' group by col1,case when ltrim(rtrim(col6))='4' then 'ABC' else 'DEF' end;
SQLException: ERROR: Failed To Get Results For ( SQL , NodeURL) : ( SELECT .....................

And in console log the error is:

eQS Node 6 has aborted execution, cause is: com.edb.gridsql.exception.XDBServerException : Can not send data to Nodes
at com.edb.gridsql.queryproc.QueryProcessor.executeQueryStep(Unknown Source)
at com.edb.gridsql.queryproc.QueryProcessor.executeQueryExecPlan(Unknown Source)
at com.edb.gridsql.queryproc.QueryProcessor.execute(Unknown Source)
at com.edb.gridsql.parser.SqlCreateTable.execute(Unknown Source)
at com.edb.gridsql.engine.ExecutableRequest.execute(Unknown Source)
at com.edb.gridsql.engine.XDBSessionContext.executeRequest(Unknown Source)
at com.edb.gridsql.engine.ServerStatement.execute(Unknown Source)
at com.edb.gridsql.engine.ServerStatement.describe(Unknown Source)
at com.edb.gridsql.engine.XDBSessionContext.describeStatement(Unknown Source)
at com.edb.gridsql.protocol.PgProtocolSession.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: com.edb.gridsql.exception.XDBMessageMonitorException: Node 6 has aborted execution, cause is: com.edb.gridsql.exception.XDBServerException : Can not send data to Nodes
at com.edb.gridsql.engine.MessageMonitor.checkMessages(Unknown Source)
at com.edb.gridsql.engine.MultinodeExecutor.executeStep(Unknown Source)
... 13 more
Caused by: com.edb.gridsql.exception.XDBWrappedException: com.edb.gridsql.exception.XDBServerException : Can not send data to Nodes
at com.edb.gridsql.engine.NodeProducerThread.processStep(Unknown Source)
at com.edb.gridsql.engine.NodeProducerThread.run(Unknown Source)
... 1 more

2009-04-09 18:57:38,763 - ERROR Throwing throwable:
com.edb.gridsql.exception.XDBServerException: Can not send data to Nodes
at com.edb.gridsql.engine.ProducerSender.sendToNodes(Unknown Source)
at com.edb.gridsql.engine.NodeProducerThread.processStep(Unknown Source)
at com.edb.gridsql.engine.NodeProducerThread.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: testdbN3@127.0.0.1: loader failed to load data: com.edb.util.PSQLException: SERRORCXX000Mcould not open relation with OID 11004649Fheapam.cL869Rrelation_open
at com.edb.gridsql.engine.loader.EDBWriter.stopEDBCopyThread(Unknown Source)
at com.edb.gridsql.engine.loader.EDBWriter.finish(Unknown Source)
at com.edb.gridsql.engine.loader.DataProcessorThread.finishLoad(Unknown Source)
at com.edb.gridsql.engine.loader.Loader.finishLoad(Unknown Source)
... 4 more

Thanks,
Venkat.

Hi Venkat,

We are having problems reproducing this problem.

Please see this thread:
http://forums.enterprisedb.com/posts/list/615.page

That is consistent with the error you are reporting, especially when you have multiple logical nodes sharing the same database server instance.

For your error, it only happens occasionally?

What would be interesting to do would be to run a different postmaster on a different port for each logical node, to ensure that there are no schema change concurrency issues, and see if the problem disappears.

Mason