Broadcast Not Working

Jasper Dan Estareja shared this problem 2 months ago
Resolved

Hi Support,


Our client has multiple broadcasts that are scheduled to run daily, weird thing is starting last June 22, 2022, almost all broadcasts stopped working. Please refer to the screenshots below.

/b0faa2f57b15aaf850fa58db57ef6d6e

/fad6a9a340cb29a1add9511a118d15bb

/988b5ec2118ca836bc86470f9a513f11

/3fabc418d68800f7c9afe4cbcd3982a3


As per our system admin, the server is has a schedule restart everyday at 3:45AM. But I don't think this is causing the skipping broadcast, as it looks like it's only skipping specifically Wednesday.

I uploaded the logs and sysinfo.xml via FTP. Zip filename is Jasper_skippingbroadcast.zip.


Hope you can help us resolve this issue.


Thanks,


Jasper

Comments (14)

photo
1

Hi Jasper,

Unfortunately, the logs shared doesn't include jdbc and yf logs to verify if there are any errors. Could you please send through the entire compressed logs folder around the time of broadcast failure.

And could you please help me with the following:

1. Are there any changes made to the reports/YF on 22/June or on 21/June (since that's when the issue started)

2. Could you please verify the ds connections, change in recipient list, report security changes (if any) etc, on those reports.

3. Can we create a new test broadcast and see if that works fine.


Best Regards,

Deepak

photo
1

Apologies, I have just realized its windows service logs. Going through yellowfin-stdout files.

photo
1

Hi Jasper,

From the stdout logs:

06:45:09.119 [TaskRunner[Filter Cache Task for Report 9723478]] ERROR com.hof.mi.util.background.TaskRunner - [3891] [Background] [TASK_RUNNER] Oh no! An error occurred saving your details to the database. Please try again later

06:45:09.221 [TaskRunner[Report Broadcast Task for Broadcast 8205609]] ERROR com.hof.mi.process.ReportBroadcastProcess - [3889] [Background] [TASK_RUNNER] com.hof.util.ActionErrorsException

com.hof.util.ActionErrorsException: null

at com.hof.mi.process.ReportBroadcastProcess.broadcastReport(ReportBroadcastProcess.java:2103) [i4-mi.jar:8.0 (20211215)]

at com.hof.mi.servlet.ReportBroadcastTask.runBroadcastTask(ReportBroadcastTask.java:146) [i4-mi.jar:8.0 (20211215)]

at com.hof.mi.util.background.TaskRunner.A(TaskRunner.java:85) [i4-mi.jar:8.0 (20211215)]

at com.hof.mi.util.background.TaskRunner.run(TaskRunner.java:172) [i4-mi.jar:8.0 (20211215)]

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_271]

at java.lang.Thread.run(Unknown Source) [?:1.8.0_271]


I believe, we may have to enable debug logging at the time of report broadcast failure to understand the issue better. You may enable debug logging as outlined here.

Kindly uploaded the compressed entire logs folder to FTP and provide the name of the file uploaded along with timestamp of the failure.

And would be still interested to know what has changed on 21/22 june, new test broadcast results.

Thanks,

Deepak

photo
1

Hi Deepak,


Thanks for investigating this. I've forwarded you message to our system admin to request for the debug logging.

I'll get back to you soon.


With regards to June 21-22, there has been no changes as per system admin. The broadcast failure only happens on Wednesdays, and works perfectly on other days, which is really weird.


Regards,


Jasper

photo
1

Hi Deepak,


Below is the reply from our system admin.

/b55c34df0fa34c2d5c073cb6fd290cc3

Thanks,

Jasper

photo
1

Hi Jasper,

If we have log4j.properties file, then it looks like the DEBUG mode is enabled.

Could you please share the entire compressed logs folder on the day which broadcast failed along with timestamp of broadcast?

I also think that it should be an issue with the broadcast scheduled on Wednesday.

1. Does the broadcast scheduled on tuesday(or any other day) is different to the one scheduled on Wednesdays? (May be like additional user groups receiving the broadcast or different report altogether etc.,?)

2. Can we create a new broadcast of same report for testing and see if that fails?

Thanks,

Deepak

photo
1

Hi Deepak,


Most of their broadcasts are scheduled to run Daily, all of those don't work on Wednesdays. An exception is the broadcast that runs weekly, specifically on Wednesday, this works just fine.

I've created two testing broadcasts on the client's report. These are set to run daily and every Wednesday. I received the broadcast email for today, I'll let you know if it runs tomorrow.

I've uploaded the entire logs folder to FTP. Filename is 206860_logs.zip.


Thanks,

Jasper

photo
1

Hi Deepak,


Looks like the broadcasts worked today, we didn't do any changes on the server.

Attached is the zip logs.


Thanks,

Jasper

photo
1

Hi Jasper,

Apologies for the delay here. Thanks for the update(So, all the boradcasts including daily and weekly worked fine?)

From the logs shared, I have checked for issues on wednesdays (06/07/2022, 29/06/2022, 22/06/2022): It looks like there JDBC connections have checked out while broadcasting. An example of one of the message below:


06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - Connection[4] checkout timeout exceeded

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - Connection[4] was checked out at 2022-07-06 05:15:13

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.pool.DBConnectionPool.checkOutConnection(DBConnectionPool.java:1393)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.pool.DBConnectionManager.getConnection(DBConnectionManager.java:310)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:334)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:447)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:402)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.servlet.ReportBroadcastTask.runBroadcastTask(ReportBroadcastTask.java:133)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.util.background.TaskRunner.A(TaskRunner.java:85)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.util.background.TaskRunner.run(TaskRunner.java:172)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.FutureTask.run(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.lang.Thread.run(Unknown Source)


This would mean that broadcast ran for more time than checkout time. May be because it was a big report or multiple things running at that time, which makes the process slower, causing it to take longer, and exceed the checkout time. I believe it would be good to check the background tasks running around that time or try adjusting the schedule by few mins.


Thanks,

Deepak

photo
1

Hi Deepak,


All broadcasts seemed to work yesterday. I checked with our system admin, he said that this same issue happened month ago.

Here's the ticket raised before:

Yellowfin Broadcast Email (yellowfinbi.com)

There are quite a number of reports sending broadcasts at the same time, and I think some of them has sub-queries which makes it run slower.

If we increase the session timeout would that help to resolve this?


Thanks,

Jasper

photo
1

Hi Jasper,

In our case, the issue seems to be happening only on wednesday?(Please correct me if broadcast fails on other days) which means that some additional activity is being carried on this day. And as you have mentioned, sub query reports may take longer time, can we change the schedule to +- few mins to see if that gives sometime to complete processing the previous tasks.

Increasing the connection timeout may sometimes introduce additional issues. Like, if we a higher connection timeout setting and if we have used up all the connections, then YF wouldn't have any more connections to make until one of them is released(The connections will only be released after completing the job/at connection timeout). So, that's one caveat of increasing the timeout.

Regards,

Deepak

photo
1

Hi Deepak,


It only skips the Wednesday broadcasts, but works fine on other days.

But it's weird that it suddenly worked this week's Wednesday.


Thanks,

Jasper

photo
1

Hi Deepak,


Looks like it got resolved again, I'm not sure what happened but it's been working fine since last week.

I'll let you know if the issue returns.

You may close this ticket for now.


Thanks,

Jasper

photo
1

Hi Jasper.

Thanks for the update. I will mark this as complete.

Warm Regards,

Deepak