Task Scheduler Issue

tara shared this question 9 days ago
In Progress

Hello,

We appear to have an issue with the task scheduler - there are millions of warnings that look like they started 2/23 after something got deleted. What is the best way to remove these so the system doesn't keep trying to run them?

YF:2022-02-23 17:58:28: WARN (ReportBroadcastTask) - [33] [Background] [TASK_SCHEDULER] ReportBroadcast record missing for TaskSchedule with ScheduleUnitId: 123972
YF:2022-02-23 17:58:28: WARN (ReportBroadcastTask) - [33] [Background] [TASK_SCHEDULER] ReportBroadcast record missing for TaskSchedule with ScheduleUnitId: 124038
YF:2022-02-23 17:58:28: WARN (ReportBroadcastTask) - [33] [Background] [TASK_SCHEDULER] ReportBroadcast record missing for TaskSchedule with ScheduleUnitId: 134181

Comments (7)

photo
1

Hi Tara,

Thanks for reaching out to support with your issue.

It sounds like some reference(s) may have be missing from the configuration DB.

Yellowfin checks at the beginning of every new minute whether there are any background tasks it should run. In your case it looks like it is finding some TaskSchedules but somehow they are missing their ReportBroadcast entries.


You could theoretically look this information up in the DB to identify the broadcast.

The tables that you will want to be looking at for broadcasts and their relational columns should be as follows:

  • ReportTask - TaskID (this should be = to broadcastID)
  • TaskSchedule - ScheduleUnitID (this should be = to broadcastID)

At that point you could try to remove the references


The following query will get rid of any childless TaskSchedules:


DELETE  FROM TaskSchedule  WHERE ScheduleUnitId  NOT IN (    SELECT BroadcastId     FROM ReportBroadcast )  AND ScheduleSubjectCode = 'BROADCAST'


When running any queries directly against the Yellowfin Configuration Database it's possible to cause irrevocable data loss. This process isn't directly supported nor recommended unless otherwise advised to do so. Before running any queries against your Yellowfin Configuration Database, please make sure you have full, up-to-date backups of both your Yellowfin Configuration Database and your Yellowfin Installation folder. This way, if something goes wrong you can revert back to your current state.

Hope htis helps! let me know if you have any additional questions.

Thanks,

Eric

photo
1

It appears that this did the trick, thanks! Just out of curiosity, how does this typically get corrupted to begin with? Could this possibly be related to our issue with broadcast counts being wrong?

photo
1

Hi Tara,

Are you seeing the task count be way off, like showing millions of entries? Could I ask what version you're running if so? I think something like this was fixed in a fairly recent version and an upgrade may help resolve. I'll check.

Thanks,

Eric

photo
1

Hi Eric,

It generally shows around ~5K. I did see that it was in a fix but we're on 9.7.2 build 20220602 so I thought we would have been past the fix. Maybe i'm wrong though.


Tara

photo
1

Hi Tara,

Not sure what happened to make the orphaned task schedules, but I'm thinking something similar could have possibly impacted the broadcast count. Otherwise this is from the old related ticket prior to the fix - the cause was a faulty query in the event table, that could be bypassed by increasing the event cache or shortening the lookup period. Maybe this would work for you?


1. Truncate the event table (backup it up prior to truncating)

2. Increase the event cache. Sagar also has an example web.xml with all possible parameters with examples.

3. Configure the broadcast events to only look at last 30 days or less.

4. Check info_cache.jsp to ensure cache is not full (so it doesn't hit the DB). You may need to configure your page security in order to allow access. (CSRF)

5. Check broadcast count.

Thanks,

Eric

photo
1

Thanks, I'll try that out. On the topic of Task Scheduler I'm also seeing a lot of these errors. What do these mean?

YF:2022-09-23 20:12:05: WARN (TaskScheduler) - [28] [Background] [TASK_SCHEDULER] Execution submission rejected, tasks Running: 5, Waiting Tasks: 20, Wait Queue Capacity: 0

It's generally followed by:WARN (JDBCConnection) - [19] [Unknown] [Unknown] Timeout occurred when clearing warnings: java.util.concurrent.TimeoutException

photo
1

Hi Eric,

I've gone through the steps of truncating Event table. Max Event cach is set to 500,000, looking at last 30 days and is not full. I still see a high count for broadcasts. Once the first broadcast runs, then it corrects. That seems to be the only trigger to get it to have the correct counts. Images attached, is something off here?