YF Instance unresponsive
Resolved
Hi Guys,
We are running v7.4 build 20180515
Yesterday our eval instance became unresponsive and had to be restarted .
According to the log "yellowfin_eval-stderr.2018-07-11.log" at around 16:25 on the 2018-07-26 we had 5 memory leak warnings before it shut down
Regards,
Stephen
Files:
logs.rar
The same problem 
Hi Stephen,
Looking at the logs, the memory leak notifications are related directly to the shutdown of Yellowfin and do not appear to be the cause of the actual problem so I have a few questions.
Is the system now stable?
Is this the first time this has happened or has it happened before?
Did you happen to take a snapshot of the JVM size using info.jsp - URL://info.jsp to see how the JVM size was at the time or run any thread checks?
Generally there are a few reasons YF could appear to be unresponsive, which could range from a lack of Memory being available to use, server specs or possible DB connection queueing due to multiple long queries being run.
Can you send me the info.jsp snapshot now please so I can have a quick look although it may not shed any light on the problem at present but it would be a good idea to know how much memory is being allocated.
If this happens again, can I also recommend you do the following before shutting it down.
Snapshot the info.jsp - Refer Here
Snapshot the threads in use (Preferably around 4-6 every 30-60 seconds - Refer Here - or if you can our threadrunner tool will do this for you.
Can I also recommend you look at our Performance guide to see if anything can be fine tuned.
It might also be worth looking to ensure you have sufficient jdbc connections to the DB, which can be looked at using the above Performance link.
Regards,
Paul
Hi Stephen,
Looking at the logs, the memory leak notifications are related directly to the shutdown of Yellowfin and do not appear to be the cause of the actual problem so I have a few questions.
Is the system now stable?
Is this the first time this has happened or has it happened before?
Did you happen to take a snapshot of the JVM size using info.jsp - URL://info.jsp to see how the JVM size was at the time or run any thread checks?
Generally there are a few reasons YF could appear to be unresponsive, which could range from a lack of Memory being available to use, server specs or possible DB connection queueing due to multiple long queries being run.
Can you send me the info.jsp snapshot now please so I can have a quick look although it may not shed any light on the problem at present but it would be a good idea to know how much memory is being allocated.
If this happens again, can I also recommend you do the following before shutting it down.
Snapshot the info.jsp - Refer Here
Snapshot the threads in use (Preferably around 4-6 every 30-60 seconds - Refer Here - or if you can our threadrunner tool will do this for you.
Can I also recommend you look at our Performance guide to see if anything can be fine tuned.
It might also be worth looking to ensure you have sufficient jdbc connections to the DB, which can be looked at using the above Performance link.
Regards,
Paul
Hi Paul,
System seems to be stable for now, I think this is the first time it happened.
Unfortunately I wasn't aware that I was suppose to check the info.jsp.
That being said the whole instance was unresponsive, could not even get to the login page.
Since the port was shutdown how would I have been able to view the info.jsp page?
I will try your suggestions the next time it happens, but in the interim are you saying that ther is no way for you to identify the problem based on the log files?
Regards,
Stephen
Hi Paul,
System seems to be stable for now, I think this is the first time it happened.
Unfortunately I wasn't aware that I was suppose to check the info.jsp.
That being said the whole instance was unresponsive, could not even get to the login page.
Since the port was shutdown how would I have been able to view the info.jsp page?
I will try your suggestions the next time it happens, but in the interim are you saying that ther is no way for you to identify the problem based on the log files?
Regards,
Stephen
Hi Stephen,
I believe the info.jsp should still work, however if you had shut Yellowfin down then obviously this would not. There was nothing specific in the logs showing a reason for the hanging, which brings my attention to possible JVM size allocation or even DB connection queuing. Neither of these would show as an issue in the log which is why it is good to run at the time to get a picture of the environment.
Can I suggest that you might even want to run this periodically (perhaps during peak times of YF usage) and see if there are any max limits being reached or getting close to. I.E. In use compared to Max allocated.
Do you know the current JVM allocation at the moment. You can get this from the info.jsp and you might even consider increasing this if it has never been changed. As the system becomes more heavily used, this is certainly an item which should be looked at.
Regards,
Paul
Hi Stephen,
I believe the info.jsp should still work, however if you had shut Yellowfin down then obviously this would not. There was nothing specific in the logs showing a reason for the hanging, which brings my attention to possible JVM size allocation or even DB connection queuing. Neither of these would show as an issue in the log which is why it is good to run at the time to get a picture of the environment.
Can I suggest that you might even want to run this periodically (perhaps during peak times of YF usage) and see if there are any max limits being reached or getting close to. I.E. In use compared to Max allocated.
Do you know the current JVM allocation at the moment. You can get this from the info.jsp and you might even consider increasing this if it has never been changed. As the system becomes more heavily used, this is certainly an item which should be looked at.
Regards,
Paul
Hi Paul,
As previously stated and as evident in the log files, the instance was stopped automatically.
I could not get it to respond, I could not access any of the screens, not even the login screen, which led me to restart the instance.
Currently the JVM has access to 1.1 GB min and 3.6 GB max memory allocated.
Regards,
Stephen
Hi Paul,
As previously stated and as evident in the log files, the instance was stopped automatically.
I could not get it to respond, I could not access any of the screens, not even the login screen, which led me to restart the instance.
Currently the JVM has access to 1.1 GB min and 3.6 GB max memory allocated.
Regards,
Stephen
Hi Stephen,
No problems, so let's keep an eye on it and as mentioned, it would be a good idea to have a look at the JVM heap size over the next little while. 3.6GB is fairly large but it might be worth increasing it, if the in use starts to touch the Maximum allocation. We do have some customers with of 20GB allocations due to the large amounts of reports and data being processed, so increasing this slightly (if possible) would not hurt. Let me know if you need any more information or further help.
Regards,
Paul
Hi Stephen,
No problems, so let's keep an eye on it and as mentioned, it would be a good idea to have a look at the JVM heap size over the next little while. 3.6GB is fairly large but it might be worth increasing it, if the in use starts to touch the Maximum allocation. We do have some customers with of 20GB allocations due to the large amounts of reports and data being processed, so increasing this slightly (if possible) would not hurt. Let me know if you need any more information or further help.
Regards,
Paul
Hi Stephen,
I have not heard back from you in regards to this ticket. Do you still need any further help with this or would you like me to close this off for now.?
Thank you,
Paul
Hi Stephen,
I have not heard back from you in regards to this ticket. Do you still need any further help with this or would you like me to close this off for now.?
Thank you,
Paul
Hi Stephen,
I have not heard back from for a little while on this ticket. Is there anything else you need or I can help with or are you okay for me to close this ticket off for now as you can always open it as required?
Regards,
Paul
Hi Stephen,
I have not heard back from for a little while on this ticket. Is there anything else you need or I can help with or are you okay for me to close this ticket off for now as you can always open it as required?
Regards,
Paul
Hi Paul,
I have been taking snapshots of the info pages as requested.
The server has not since had any problems, will reopen the call if the problem reoccurs.
Regards,
Stephen
Hi Paul,
I have been taking snapshots of the info pages as requested.
The server has not since had any problems, will reopen the call if the problem reoccurs.
Regards,
Stephen
Hi Stephen,
Thats great news and yes let me know if you need further help.
Regards,
Paul
Hi Stephen,
Thats great news and yes let me know if you need further help.
Regards,
Paul
Replies have been locked on this page!