Broadcast Not Working

Hi Jasper,

Unfortunately, the logs shared doesn't include jdbc and yf logs to verify if there are any errors. Could you please send through the entire compressed logs folder around the time of broadcast failure.

And could you please help me with the following:

1. Are there any changes made to the reports/YF on 22/June or on 21/June (since that's when the issue started)

2. Could you please verify the ds connections, change in recipient list, report security changes (if any) etc, on those reports.

3. Can we create a new test broadcast and see if that works fine.

Best Regards,

Deepak

Apologies, I have just realized its windows service logs. Going through yellowfin-stdout files.

Hi Jasper,

From the stdout logs:

06:45:09.119 [TaskRunner[Filter Cache Task for Report 9723478]] ERROR com.hof.mi.util.background.TaskRunner - [3891] [Background] [TASK_RUNNER] Oh no! An error occurred saving your details to the database. Please try again later

06:45:09.221 [TaskRunner[Report Broadcast Task for Broadcast 8205609]] ERROR com.hof.mi.process.ReportBroadcastProcess - [3889] [Background] [TASK_RUNNER] com.hof.util.ActionErrorsException

com.hof.util.ActionErrorsException: null

at com.hof.mi.process.ReportBroadcastProcess.broadcastReport(ReportBroadcastProcess.java:2103) [i4-mi.8.0 (20211215)]

at com.hof.mi.servlet.ReportBroadcastTask.runBroadcastTask(ReportBroadcastTask.java:146) [i4-mi.8.0 (20211215)]

at com.hof.mi.util.background.TaskRunner.A(TaskRunner.java:85) [i4-mi.8.0 (20211215)]

at com.hof.mi.util.background.TaskRunner.run(TaskRunner.java:172) [i4-mi.8.0 (20211215)]

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_271]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_271]

at java.lang.Thread.run(Unknown Source) [?:1.8.0_271]

I believe, we may have to enable debug logging at the time of report broadcast failure to understand the issue better. You may enable debug logging as outlined here.

Kindly uploaded the compressed entire logs folder to FTP and provide the name of the file uploaded along with timestamp of the failure.

And would be still interested to know what has changed on 21/22 june, new test broadcast results.

Thanks,

Deepak

Hi Deepak,

Thanks for investigating this. I've forwarded you message to our system admin to request for the debug logging.

I'll get back to you soon.

With regards to June 21-22, there has been no changes as per system admin. The broadcast failure only happens on Wednesdays, and works perfectly on other days, which is really weird.

Regards,

Jasper

Hi Deepak,

Below is the reply from our system admin.

Thanks,

Jasper

Hi Jasper,

If we have log4j.properties file, then it looks like the DEBUG mode is enabled.

Could you please share the entire compressed logs folder on the day which broadcast failed along with timestamp of broadcast?

I also think that it should be an issue with the broadcast scheduled on Wednesday.

1. Does the broadcast scheduled on tuesday(or any other day) is different to the one scheduled on Wednesdays? (May be like additional user groups receiving the broadcast or different report altogether etc.,?)

2. Can we create a new broadcast of same report for testing and see if that fails?

Thanks,

Deepak

Hi Deepak,

Most of their broadcasts are scheduled to run Daily, all of those don't work on Wednesdays. An exception is the broadcast that runs weekly, specifically on Wednesday, this works just fine.

I've created two testing broadcasts on the client's report. These are set to run daily and every Wednesday. I received the broadcast email for today, I'll let you know if it runs tomorrow.

I've uploaded the entire logs folder to FTP. Filename is 206860_logs.zip.

Thanks,

Jasper

Hi Deepak,

Looks like the broadcasts worked today, we didn't do any changes on the server.

Attached is the zip logs.

Thanks,

Jasper

Hi Jasper,

Apologies for the delay here. Thanks for the update(So, all the boradcasts including daily and weekly worked fine?)

From the logs shared, I have checked for issues on wednesdays (06/07/2022, 29/06/2022, 22/06/2022): It looks like there JDBC connections have checked out while broadcasting. An example of one of the message below:

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - Connection[4] checkout timeout exceeded

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - Connection[4] was checked out at 2022-07-06 05:15:13

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.pool.DBConnectionPool.checkOutConnection(DBConnectionPool.java:1393)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.pool.DBConnectionManager.getConnection(DBConnectionManager.java:310)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:334)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:447)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.util.DBAction.<init>(DBAction.java:402)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.servlet.ReportBroadcastTask.runBroadcastTask(ReportBroadcastTask.java:133)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.util.background.TaskRunner.A(TaskRunner.java:85)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at com.hof.mi.util.background.TaskRunner.run(TaskRunner.java:172)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.FutureTask.run(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

06:46:04.807 [DBConnectionManagerMaintenanceTimer] ERROR com.hof.pool.DBConnectionPool.jdbclog - at java.lang.Thread.run(Unknown Source)

This would mean that broadcast ran for more time than checkout time. May be because it was a big report or multiple things running at that time, which makes the process slower, causing it to take longer, and exceed the checkout time. I believe it would be good to check the background tasks running around that time or try adjusting the schedule by few mins.

Thanks,

Deepak

Hi Deepak,

All broadcasts seemed to work yesterday. I checked with our system admin, he said that this same issue happened month ago.

Here's the ticket raised before:

Yellowfin Broadcast Email (yellowfinbi.com)

There are quite a number of reports sending broadcasts at the same time, and I think some of them has sub-queries which makes it run slower.

If we increase the session timeout would that help to resolve this?

Thanks,

Jasper

Hi Jasper,

In our case, the issue seems to be happening only on wednesday?(Please correct me if broadcast fails on other days) which means that some additional activity is being carried on this day. And as you have mentioned, sub query reports may take longer time, can we change the schedule to +- few mins to see if that gives sometime to complete processing the previous tasks.

Increasing the connection timeout may sometimes introduce additional issues. Like, if we a higher connection timeout setting and if we have used up all the connections, then YF wouldn't have any more connections to make until one of them is released(The connections will only be released after completing the job/at connection timeout). So, that's one caveat of increasing the timeout.

Regards,

Deepak

Hi Deepak,

It only skips the Wednesday broadcasts, but works fine on other days.

But it's weird that it suddenly worked this week's Wednesday.

Thanks,

Jasper

Hi Deepak,

Looks like it got resolved again, I'm not sure what happened but it's been working fine since last week.

I'll let you know if the issue returns.

You may close this ticket for now.

Thanks,

Jasper

Hi Jasper.

Thanks for the update. I will mark this as complete.

Warm Regards,

Deepak