Top-N report on very large result set
Hi,
We have a problem with a specific report where we want to list the top-n product by some metric. The problem is that the result set can contains more than 100K rows. This makes everything very slow.
I found this article, which offers a solution but is quite limited in terms of filters and also very old:
https://community.yellowfinbi.com/topic/how-to-put-an-order-by-into-the-sql-of-a-report
Our first problem, the standard Amazon Redshift Connector does not support Stored Procedures in the view builder process, although Redshift does support them. Is there a way around it?
The next problem might be Access Filters, which need to work, too. Do they work with stored procedure based filters?
To avoid all those problems, is there a way to push down the ranking and sorting to the data base other than using a stored procedure?
Best regards,
Ingo
Hi Ingo,
Thanks for reaching out.
I will test the stored procedure view connection with Redshift connector. Could you please confirm if you have already tried the CData Redshift connector and it wouldn;t allow stored proc to pass through?
Cdata Redshift connector: https://www.cdata.com/download/download.aspx?tag=yellowfin&sku=FRRJ-V&type=demo&file=full/FRRJ-V/setup.zip
And regarding 2. Stored procedure filters are automatically applied to every report that is built on the view, and access filters would further restrict data.
3. I think the sorting would only happen post-processing but unfortunately, I don't think we can change this.
Thanks,
Deepak
Hi Ingo,
Thanks for reaching out.
I will test the stored procedure view connection with Redshift connector. Could you please confirm if you have already tried the CData Redshift connector and it wouldn;t allow stored proc to pass through?
Cdata Redshift connector: https://www.cdata.com/download/download.aspx?tag=yellowfin&sku=FRRJ-V&type=demo&file=full/FRRJ-V/setup.zip
And regarding 2. Stored procedure filters are automatically applied to every report that is built on the view, and access filters would further restrict data.
3. I think the sorting would only happen post-processing but unfortunately, I don't think we can change this.
Thanks,
Deepak
Hi Deepak,
No I have not tried to use a cdata connector...I would hope that Yellowfin to put feature support into their native connectors. If you know for sure that stored procedures works with cdata connectors, I might give it a shot.
Best regards,
Ingo
Hi Deepak,
No I have not tried to use a cdata connector...I would hope that Yellowfin to put feature support into their native connectors. If you know for sure that stored procedures works with cdata connectors, I might give it a shot.
Best regards,
Ingo
Hi Ingo,
Sure, please let me know how that goes because YF has officially launched CDATA connectors: https://community.yellowfinbi.com/announcement/yellowfin-launches-certified-cdata-connectors and the existing connectors will still be available for usage but they won't be modified/supported so I think the only way forward would be try CDATA connectors.
More information: https://community.yellowfinbi.com/announcement/discontinue-support-for-marketplace-content
Thanks,
Deepak
Hi Ingo,
Sure, please let me know how that goes because YF has officially launched CDATA connectors: https://community.yellowfinbi.com/announcement/yellowfin-launches-certified-cdata-connectors and the existing connectors will still be available for usage but they won't be modified/supported so I think the only way forward would be try CDATA connectors.
More information: https://community.yellowfinbi.com/announcement/discontinue-support-for-marketplace-content
Thanks,
Deepak
Hi Deepak,
I think there is a bit of a misunderstanding. Amazon Redshift is a fully YF supported JDBC Data Source and not a market place connetor:
Therefore please find out with the Dev team or whoever, if the Redshift Data Source will be able to support stored procedures in the near future as Redshift is capable, but it is not recognized in the Yellowfin UI.
Many thanks and best regards,
Ingo
Hi Deepak,
I think there is a bit of a misunderstanding. Amazon Redshift is a fully YF supported JDBC Data Source and not a market place connetor:
Therefore please find out with the Dev team or whoever, if the Redshift Data Source will be able to support stored procedures in the near future as Redshift is capable, but it is not recognized in the Yellowfin UI.
Many thanks and best regards,
Ingo
Hi Ingo,
Apologies for the delay.
We will get in touch with the wider team with your query and get back to you with an update. Just wanted to let you know that the team is currently on Christmas break.
Thanks,
Deepak
Hi Ingo,
Apologies for the delay.
We will get in touch with the wider team with your query and get back to you with an update. Just wanted to let you know that the team is currently on Christmas break.
Thanks,
Deepak
Hi Ingo,
I haven't had a chance to discuss this with the wider team yet. In the meanwhile, I managed to find connection details to an existing redshift instance and tried to create a Stored proc view using it:
But when I couldn't finish creating a view because I do not have valid stored procedures that I can use (Unfortunately, I do not have access to redshift instance itself to create stored procs in it) :
I have used the standard out of the box redshift that uses postgres driver to create the connection:
I have also tried connecting to the same redshift instance as a generic ds source with a redshift driver (attached):
Thanks,
Deepak
Hi Ingo,
I haven't had a chance to discuss this with the wider team yet. In the meanwhile, I managed to find connection details to an existing redshift instance and tried to create a Stored proc view using it:
But when I couldn't finish creating a view because I do not have valid stored procedures that I can use (Unfortunately, I do not have access to redshift instance itself to create stored procs in it) :
I have used the standard out of the box redshift that uses postgres driver to create the connection:
I have also tried connecting to the same redshift instance as a generic ds source with a redshift driver (attached):
Thanks,
Deepak
Hi Deepak,
I'll have a look at those data source and driver combinations and get back to you
Best regards,
Ingo
Hi Deepak,
I'll have a look at those data source and driver combinations and get back to you
Best regards,
Ingo
Sure Ingo. Please let me know how that goes.
Regards,
Deepak
Sure Ingo. Please let me know how that goes.
Regards,
Deepak
Hi Ingo,
Hope you're having a good week.
Just wanted to check-in and see how it's all going.
Kind regards,
Deepak Chaganti
Hi Ingo,
Hope you're having a good week.
Just wanted to check-in and see how it's all going.
Kind regards,
Deepak Chaganti
Hi Deepak,
I did some testing today and a stored procedure does not seem to work as Redshift only allows the return of a cursor and that does not seem to be supported in Yellowfin. But I found a different solution, using a Freehand SQL Report. (https://community.yellowfinbi.com/topic/parameters-in-freehand-sql-reports) That does almost work like I need it to.
With Common Table Expression I'm able to write a fitting query and even use Access Filter. But I found two issues with the Freehand SQL Report, one a bug (handling of using same access filter multiple times in a query) in my opinion and the other maybe solvable. So I would close this topic and create two tickets for the Freehand SQL Report.
Best regards,
Ingo
Hi Deepak,
I did some testing today and a stored procedure does not seem to work as Redshift only allows the return of a cursor and that does not seem to be supported in Yellowfin. But I found a different solution, using a Freehand SQL Report. (https://community.yellowfinbi.com/topic/parameters-in-freehand-sql-reports) That does almost work like I need it to.
With Common Table Expression I'm able to write a fitting query and even use Access Filter. But I found two issues with the Freehand SQL Report, one a bug (handling of using same access filter multiple times in a query) in my opinion and the other maybe solvable. So I would close this topic and create two tickets for the Freehand SQL Report.
Best regards,
Ingo
Hi Ingo,
Sure, thanks for the update.
I can close this case on your behalf. The other issue related to Freehand SQL with access filters used multiple times in same query can be discussed with a new ticket.
Regards,
Deepak
Hi Ingo,
Sure, thanks for the update.
I can close this case on your behalf. The other issue related to Freehand SQL with access filters used multiple times in same query can be discussed with a new ticket.
Regards,
Deepak
Replies have been locked on this page!