YF instance slow and unresponsive
Resolved
Hi Guys,
We have a customer where there YF instance becomes unresponsive, they are running v7.4 build 20180515.
They previously updated to v7.4.6 and was told to rollback to the v7.4 build 20180515 build due to a bug in v7.4.6
There are no errors in the log files, apart from the memory leak warnings (see info pages and logs attached).
Previously I logged a similar call on the same version and I was told that the memory leaks warnings is not related.
Regards,
Stephen
Hi Stephen,
Apologies for not getting back you on this as yet, we will get to you in the next 24hrs.
Thanks,
David
Hi Stephen,
Apologies for not getting back you on this as yet, we will get to you in the next 24hrs.
Thanks,
David
Hi David
Thanks for letting me know
Regards,
Stephen
Hi David
Thanks for letting me know
Regards,
Stephen
Hi Stephen,
Thanks for getting in touch. I have reviewed the previous tickets and I had mentioned that we should firstly consider upgrading the JVM size if there is still an issue with performance. Can you advise if this has been done as yet to see if this improves the performance?
Regards,
Paul
Hi Stephen,
Thanks for getting in touch. I have reviewed the previous tickets and I had mentioned that we should firstly consider upgrading the JVM size if there is still an issue with performance. Can you advise if this has been done as yet to see if this improves the performance?
Regards,
Paul
Hi Paul,
This is 2 different enviroments, I just pointed out the similarities.
As you can see in this instance (by the included info.jsp page) there is almost 9GB of ram available to the JVM.
Regards,
Stephen
Hi Paul,
This is 2 different enviroments, I just pointed out the similarities.
As you can see in this instance (by the included info.jsp page) there is almost 9GB of ram available to the JVM.
Regards,
Stephen
Hi Stephen,
I have just been looking through your source jdbc logs and notice the following, which seems to point to connection issue with the JDBC database where it does not have sufficient connections available which I would suggest looking at, but perhaps increasing the number of connections to this data source. If Yellowfin is waiting for connections it will appear to look like it has hung.
2018-08-13 07:08:00 WARN: [70025] [352546] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:00 WARN: [70025] [352542] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:01 WARN: [70025] [352547] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:03 WARN: [70025] [352543] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:05 WARN: [70025] [352545] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:28 WARN: [70025] [352543] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:28 WARN: [70025] [352570] No connections available. Waiting for a connection to be returned.
2018-08-13 07:09:00 WARN: [70025] All connections are checked out and 1 threads are waiting!
2018-08-13 07:09:00 INFO: [70025] Connection[0] was checked out at 2018-08-13 07:08:29
2018-08-13 07:09:00 INFO: [70025] at com.hof.pool.DBConnectionPool.checkOutConnection(DBConnectionPool.java:1414)
2018-08-13 07:09:00 INFO: [70025] at com.hof.pool.DBConnectionManager.getConnection(DBConnectionManager.java:312)
2018-08-13 07:09:00 INFO: [70025] at com.hof.util.DBAction.<init>(DBAction.java:352)
2018-08-13 07:09:00 INFO: [70025] at com.hof.util.DBAction.<init>(DBAction.java:465)
2018-04-17 06:52:10 NOTICE: [70025] sourceName: Redshift (testdb)
2018-04-17 06:52:10 NOTICE: [70025] connectionMethod: JDBC
2018-04-17 06:52:10 NOTICE: [70025] min connections: 1
2018-04-17 06:52:10 NOTICE: [70025] max connections: 5
I have also noticed the following in the same source log, for which we have a KB on which you can find here.
2018-08-10 16:34:59 ERROR: [70025] Connection[3] is closed
2018-08-10 16:35:00 INFO: [70025] Connection[3] successfully reopened
2018-08-10 16:35:59 WARN: [70025] Execution problem occurred when clearing warnings: java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: This connection has been closed.
java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: This connection has been closed.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.hof.pool.JDBCConnection.C(JDBCConnection.java:391)
at com.hof.pool.JDBCConnection.D(JDBCConnection.java:371)
at com.hof.pool.DBConnectionPool.E(DBConnectionPool.java:1680)
at com.hof.pool.DBConnectionPool.A(DBConnectionPool.java:238)
at com.hof.pool.DBConnectionPool$_A.run(DBConnectionPool.java:2104)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:803)
at org.postgresql.jdbc.PgConnection.clearWarnings(PgConnection.java:721)
at com.hof.pool.JDBCConnection$3.call(JDBCConnection.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Can I suggest you look at this jdbc data source and look at increasing the maximum connections and go through the JDBC Verify link provided above to see if this fixes the problem.
Thanks you,
Paul
Hi Stephen,
I have just been looking through your source jdbc logs and notice the following, which seems to point to connection issue with the JDBC database where it does not have sufficient connections available which I would suggest looking at, but perhaps increasing the number of connections to this data source. If Yellowfin is waiting for connections it will appear to look like it has hung.
2018-08-13 07:08:00 WARN: [70025] [352546] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:00 WARN: [70025] [352542] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:01 WARN: [70025] [352547] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:03 WARN: [70025] [352543] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:05 WARN: [70025] [352545] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:28 WARN: [70025] [352543] No connections available. Waiting for a connection to be returned.
2018-08-13 07:08:28 WARN: [70025] [352570] No connections available. Waiting for a connection to be returned.
2018-08-13 07:09:00 WARN: [70025] All connections are checked out and 1 threads are waiting!
2018-08-13 07:09:00 INFO: [70025] Connection[0] was checked out at 2018-08-13 07:08:29
2018-08-13 07:09:00 INFO: [70025] at com.hof.pool.DBConnectionPool.checkOutConnection(DBConnectionPool.java:1414)
2018-08-13 07:09:00 INFO: [70025] at com.hof.pool.DBConnectionManager.getConnection(DBConnectionManager.java:312)
2018-08-13 07:09:00 INFO: [70025] at com.hof.util.DBAction.<init>(DBAction.java:352)
2018-08-13 07:09:00 INFO: [70025] at com.hof.util.DBAction.<init>(DBAction.java:465)
2018-04-17 06:52:10 NOTICE: [70025] sourceName: Redshift (testdb)
2018-04-17 06:52:10 NOTICE: [70025] connectionMethod: JDBC
2018-04-17 06:52:10 NOTICE: [70025] min connections: 1
2018-04-17 06:52:10 NOTICE: [70025] max connections: 5
I have also noticed the following in the same source log, for which we have a KB on which you can find here.
2018-08-10 16:34:59 ERROR: [70025] Connection[3] is closed
2018-08-10 16:35:00 INFO: [70025] Connection[3] successfully reopened
2018-08-10 16:35:59 WARN: [70025] Execution problem occurred when clearing warnings: java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: This connection has been closed.
java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: This connection has been closed.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.hof.pool.JDBCConnection.C(JDBCConnection.java:391)
at com.hof.pool.JDBCConnection.D(JDBCConnection.java:371)
at com.hof.pool.DBConnectionPool.E(DBConnectionPool.java:1680)
at com.hof.pool.DBConnectionPool.A(DBConnectionPool.java:238)
at com.hof.pool.DBConnectionPool$_A.run(DBConnectionPool.java:2104)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:803)
at org.postgresql.jdbc.PgConnection.clearWarnings(PgConnection.java:721)
at com.hof.pool.JDBCConnection$3.call(JDBCConnection.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Can I suggest you look at this jdbc data source and look at increasing the maximum connections and go through the JDBC Verify link provided above to see if this fixes the problem.
Thanks you,
Paul
Hi Paul,
Thank for the feedback and suggestions, I have forwarded it on to the client.
Regards,
Stephen
Hi Paul,
Thank for the feedback and suggestions, I have forwarded it on to the client.
Regards,
Stephen
Hi Stephen,
Just touching base again as I have not heard back from you in regards to this and checking in to see how things are going and if you need any further help?
Regards,
Paul
Hi Stephen,
Just touching base again as I have not heard back from you in regards to this and checking in to see how things are going and if you need any further help?
Regards,
Paul
Hi Paul,
I think you can close it for now.
I sent through your suggestions to the customer, and have not heard and further complaints
Regards,
Stephen
Hi Paul,
I think you can close it for now.
I sent through your suggestions to the customer, and have not heard and further complaints
Regards,
Stephen
Thanks Stephen for getting back to me. Closing for now.
Thanks Stephen for getting back to me. Closing for now.
Replies have been locked on this page!