Category
Assisted Insights
Product Version
9.0
What impact does Yellowfin signals have on the database?
Answered
I've been going through various documentation about how Signals work and its still not entirely clear to me about how YF initially grabs the baseline data that it stores in memory. Is it just a standard SQL query that will hit the database once per signal on the scheduled time and then store all of that in memory? My understanding is that after that initial large query it should run all sub-tasks on the in memory stored data?
Just trying to fully assess the potential impact on the database of enabling signals on our systems.
Thanks for the help!
The same question
Hello Kat Brownlee,
Thank you for reaching out to Yellowfin Support
I just want to let you know that in order to understand the answer to your question, you must know how does Yellowfin Signals work?
Firstly, you need a view to base your Signals off. You choose your date, metric and dimension fields, and then all of this data is pulled from that view and stored in memory, where it will be used for the magic of signals. We can think of this data set as the baseline for signals. Data outside of this view cannot be used.
So now that we have a baseline dataset in memory, the real magic starts to happen.
Sub-tasks are then created which will generate their own data set (based on the baseline) and store these in memory. These subtasks will be displayed in Scheduled Management. The number of sub-tasks generated is based on a few factors;
-Signal configuration
-Row level security (e.g Source Filters, Data source substitution)
Analysis is then performed on each sub-task's dataset, and when Signals are detected the data for that Signal is cached in Report Instance (unless Signals caching has been turned off, in which case it will have to hit the data set directly). The metadata about the signal is also saved to the database, along with some extra data fragments stored for some Signal types.
The data for each sub-task is held in memory until analysis for that sub-task is completed, at which point that memory is freed. There is no overlap between sub-tasks for this data, so it is not shared.
Once all sub-tasks are completed, the baseline data set is removed from memory.
I've also included a couple links below for your reference.
https://wiki.yellowfinbi.com/display/user80/Exploring+a+Yellowfin+Signal
https://wiki.yellowfinbi.com/display/yfcurrent/Signals+Analysis+Configuration
https://wiki.yellowfinbi.com/display/yfcurrent/Yellowfin+Signals
https://wiki.yellowfinbi.com/display/user80/Schedule+Management
I hope this was helpful, but please let me know if it didn't answer your question.
Regards,
Yamini Naidu
Hello Kat Brownlee,
Thank you for reaching out to Yellowfin Support
I just want to let you know that in order to understand the answer to your question, you must know how does Yellowfin Signals work?
Firstly, you need a view to base your Signals off. You choose your date, metric and dimension fields, and then all of this data is pulled from that view and stored in memory, where it will be used for the magic of signals. We can think of this data set as the baseline for signals. Data outside of this view cannot be used.
So now that we have a baseline dataset in memory, the real magic starts to happen.
Sub-tasks are then created which will generate their own data set (based on the baseline) and store these in memory. These subtasks will be displayed in Scheduled Management. The number of sub-tasks generated is based on a few factors;
-Signal configuration
-Row level security (e.g Source Filters, Data source substitution)
Analysis is then performed on each sub-task's dataset, and when Signals are detected the data for that Signal is cached in Report Instance (unless Signals caching has been turned off, in which case it will have to hit the data set directly). The metadata about the signal is also saved to the database, along with some extra data fragments stored for some Signal types.
The data for each sub-task is held in memory until analysis for that sub-task is completed, at which point that memory is freed. There is no overlap between sub-tasks for this data, so it is not shared.
Once all sub-tasks are completed, the baseline data set is removed from memory.
I've also included a couple links below for your reference.
https://wiki.yellowfinbi.com/display/user80/Exploring+a+Yellowfin+Signal
https://wiki.yellowfinbi.com/display/yfcurrent/Signals+Analysis+Configuration
https://wiki.yellowfinbi.com/display/yfcurrent/Yellowfin+Signals
https://wiki.yellowfinbi.com/display/user80/Schedule+Management
I hope this was helpful, but please let me know if it didn't answer your question.
Regards,
Yamini Naidu
Hi Yamini,
I think I found these documents that referenced the above as well but it sort of doesnt fully address my question. Which is mostly, will this lock up the database/view when the initial baseline is being pulled? We have seen some issues with it table locking and memory constraints occurring with chart production so I want to be sure which points can cause failure. I gather its safe for the subtasks but just to confirm the only hit to the database itself is the initial baseline collection?
Hi Yamini,
I think I found these documents that referenced the above as well but it sort of doesnt fully address my question. Which is mostly, will this lock up the database/view when the initial baseline is being pulled? We have seen some issues with it table locking and memory constraints occurring with chart production so I want to be sure which points can cause failure. I gather its safe for the subtasks but just to confirm the only hit to the database itself is the initial baseline collection?
Hello Kat,
I apologize for the extreme delay on my part and appreciate your patience immensely.
Just letting you know that the Yellowfin signal will undoubtedly affect the performance. At certain interval, bad jobs are executed. Bad jobs retrieve information from a database and compare old data. I have provided one wiki link that may be of use to you; please review that link and let me know if it was of any assistance.
https://wiki.yellowfinbi.com/display/yfcurrent/Server+Requirements#ServerRequirements-signals_server
Regards,
Yamini Naidu
Hello Kat,
I apologize for the extreme delay on my part and appreciate your patience immensely.
Just letting you know that the Yellowfin signal will undoubtedly affect the performance. At certain interval, bad jobs are executed. Bad jobs retrieve information from a database and compare old data. I have provided one wiki link that may be of use to you; please review that link and let me know if it was of any assistance.
https://wiki.yellowfinbi.com/display/yfcurrent/Server+Requirements#ServerRequirements-signals_server
Regards,
Yamini Naidu
Hello Kat,
Hope you are doing well.
I just want to touch base to see if you had chance to read through my response. If you can let me know that would be great.
Thank you!
Regards,
Yamini Naidu
Hello Kat,
Hope you are doing well.
I just want to touch base to see if you had chance to read through my response. If you can let me know that would be great.
Thank you!
Regards,
Yamini Naidu
Hello Kat,
Hope you are doing well.
As there is no revert back from you and because of age and inactivity I am going to go ahead and mark this ticket as closed for now. If you wish to re-open and discuss further I will be more than happy to do so.
Thanks!
Regards,
Yamini Naidu
Hello Kat,
Hope you are doing well.
As there is no revert back from you and because of age and inactivity I am going to go ahead and mark this ticket as closed for now. If you wish to re-open and discuss further I will be more than happy to do so.
Thanks!
Regards,
Yamini Naidu
Replies have been locked on this page!