This article will provide some insight as to how broadcasts are run behind the scenes. For the basics on what Broadcasts are, and how to set them up, please check our wiki here.
Overview
Broadcasts are background tasks that run on a schedule and are used to send content to users (or FTP) in a file type/display of your choice. There are essentially 4 components to this; Content, Recipient, Schedule, Background Task Runner.
Content
This is simply the content being sent to the user, e.g. a report, report link, dashboards coming soon.
What is important to understand with the content is that it can differ based on the content recipient. Yellowfin will group the result sets as much as possible, except when using access filters. When access filters are enabled no grouping of result sets are done.
If you're sending the content to a FTP site, then the values are based on system defaults, or the user creating the broadcast, as you essentially don't have a recipient user (so no user information can be obtained).
Recipient
Self-explanatory. Who is the broadcast going to? To a user groups, individual external email address, or possibly even a FTP site.
Schedule
This is the schedule itself, what is meant to run at what time. It's what you configure when creating your broadcast; time, rules etc..
Some behind the scenes details on the scheduler itself can be found here. The schedule starts with a background poll process which runs every 30secs, if it finds a task that should be running at that time it will add it to the schedule queue. There are 2 queues it can be added to; the concurrent queue, or the waiting queue.
Background Task Runner
This is the actual running of the tasks itself, and yes it happens in the background (and not only used for broadcasts). By default 5 tasks can be run concurrently, while another 20 can be sitting in the waiting to run queue. The 5 and 20 are the default settings, though can be changed. If the concurrent task queue is full, it goes to the waiting queue, if the waiting queue is full it does not get added, and will try to be picked up 30secs later. If the poll identifies a schedule as missed, it will attempt to re-run it (this is a case where YF was not running during that scheduled time), however, when it runs next depends on the frequency. A failed daily report that did not run today would attempt to run again, unless it's within 12 hrs of the next run time, then it will just wait for the next scheduled run. It's also worth noting that the task runner uses it's own thread, and can also use it's very own connection to the data source if using the secondary connection pool option.
How it all ties to together
- The scheduler runs, finds a task that is meant to run, then adds it to the background queue. If there is a spot it the queue it starts running. At this point Yellowfin looks at the content and recipient list.
- If sending to a single user or FTP;
The report sql is run and result set is pulled from the DB.
The result set is stored in memory.
The email content is generated. This would be the email itself, so if you're exporting as PDF, the content contains the text in the email and the PDF file.
The Result set is then cleared
The email is sent. - If sending to multiple users (with no source filters);
The recipient list is returned, and Yellowfin will identify result set groups to bundle up.
The report sql is run and result set is pulled from the DB.
The result set is stored in memory.
The email content is generated for that group/s.
The email is sent. - If sending a report that contains source filters;
The recipient list is returned.
The report sql is run and result set is pulled from the DB.
The result set is stored in memory.The email content is generated for that user.
If that was the only recipient the email is sent.
If there are more recipients, the report sql is run again, and it loops until that final result set, and it's all sent.
Note: You can change the behavior when broadcasting to multiple users (steps 3 & 4), see the following articles for more info;
Broadcast using CC recipients, Broadcast reports individually instead of bulk sending
If you have any questions, or running into issues relating to schedules, please raise a ticket with our support team.
Regards,
The Yellowfin Support Team.