Tomcat does not shut down cleanly

Bharath Kumar shared this question 3 weeks ago
Awaiting Reply

Hi Team,

I am facing an issue with Yellowfin tomcat. It does not shutdown, possibly the threads are running at the background. From the catalina logs, I see the below snipper:


1-Dec-2018 09:35:22.725 INFORMATION [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.

11-Dec-2018 09:35:22.726 INFORMATION [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]

11-Dec-2018 09:35:22.778 INFORMATION [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]

11-Dec-2018 09:35:22.865 WARNUNG [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [AdminConsole] appears to have started a thread named [pool-132707-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage

java.lang.ClassCastException: java.io.ObjectStreamClass cannot be cast to java.lang.String

at java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1650)

at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:803)

at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:891)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1857)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.catalina.session.StandardSession.doReadObject(StandardSession.java:1611)

at org.apache.catalina.session.StandardSession.readObjectData(StandardSession.java:1077)

at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:217)

at org.apache.catalina.session.StandardManager.load(StandardManager.java:161)

at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:351)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)

at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1140)

at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)


We are using YF build 7.3 August 2018 build and tomcat 8.5.3 version.

Appreciate your assistance.

Thanks,

Bharath

Comments (12)

photo
1

Hi Bharath,

Thanks for reaching out. When you say Yellowfin is not shutting down cleanly. Is there an issue starting it back up again or is it just the WARNING you are seeing in the logs? The WARNING's can be ignored as these can be quite normal and Yellowfin should shutdown correctly. Please let me know if this is not the case.

Thank you,

Paul

photo
1

Hi Paul,

There is no issue starting it up, but it takes time to shut down. Possibly, the threads still running at the back-end or tomcat is still busy. If the warnings can be ignored, can you let us know how to identify the process which is still running though we shutdown the service.

Regards,

Bharath

photo
1

Hi Bharath,

Once the JVM has stopped, all threads associated to the JVM thread pool will be terminated. Everything is running within the JVM thread pool, so it is not a simple method of working out what process is still running. If Yellowfin does indeed hang (which it should not), the simplest way to fix the issue is to kill the Java PID running Yellowfin. You could possible perform a thread / heap dump on the JVM during shutdown, but I am not sure how accurate the information would be.

Regards,

Paul

photo
1

Hi Paul,

I agree to kill the Java Pid, but our client does not agree. When they shutdown the service, it hangs for about 15-20 mins and we have to kill the PID.

I did some investigation and copied the jdbc database drivers from root/web-inf/lib to tomcat/lib, but that did not help.

Regards,

Bharath

photo
1

HI Bharath,

So does it take 15-20 minutes everytime you shut down Yellowfin as It should not be taking this long?

photo
1

Hi Paul ,

Yes, it should not take much time for stopping the services . Please find the attached thread dump file .

Let me know if you need more details on this.


Regards

Pratiksha

photo
photo
1

Hi Bharath,

The only obvious things which seems to present itself is the JVM is going through a GC during this time, but even then a GC should only take a very short amount of time.

The logs you have sent through are from December. Do you have anything more recent when they are getting this problem?

Thanks,

Paul

photo
1

Hi Paul ,

Attached are the latest logs.


Regards

Pratiksha

photo
1

Hi Pratiksha,

Can you advise how long these GC stats were generated for please. I cannot see any issues directly with the GC times and on average they are taking less than 100ms to complete.


c4cefcf3392b708b5e47db0c725eee23

Thank you,

Paul

photo
1

Hi Pratiksha,

I have not been able to determine whether the shutdown is taking 15 plus minutes, to perform at ths stage. Are you able to show this hapenning with a screen recording or screen share by any chance? Are you running the latest BMC build and/or Java versions? This might be worth looking at in-case there is an issue here. From what I can see at the moment (via the logs), is that the shutdown looks okay. I can see several INFO and WARNINGS but no SEVERE, CRITICAL or ERRORS going on and they seem to be only related to open jdbc connections.

Thanks,

Paul

photo
1

Hi Paul ,

Thanks for your response.

Application has been deployed with external tomcat : apache-tomcat-8.5.34

Build Version: 20171012

Yellowfin version is 7.3


Thanks

Pratiksha

photo
1

Hi Pratiksha,

Thanks for the information. I think it would be worth upgrading the client to the latest version of Yellowfin as this build is now quite old. Do you have the option to do this?

Thanks,

Paul