Top-N report on very large result set

Ingo Klose shared this question 4 months ago
Completed

Hi,

We have a problem with a specific report where we want to list the top-n product by some metric. The problem is that the result set can contains more than 100K rows. This makes everything very slow.

I found this article, which offers a solution but is quite limited in terms of filters and also very old:

https://community.yellowfinbi.com/topic/how-to-put-an-order-by-into-the-sql-of-a-report

Our first problem, the standard Amazon Redshift Connector does not support Stored Procedures in the view builder process, although Redshift does support them. Is there a way around it?

The next problem might be Access Filters, which need to work, too. Do they work with stored procedure based filters?

To avoid all those problems, is there a way to push down the ranking and sorting to the data base other than using a stored procedure?


Best regards,

Ingo

Replies (11)

photo
1

Hi Ingo,

Thanks for reaching out.

I will test the stored procedure view connection with Redshift connector. Could you please confirm if you have already tried the CData Redshift connector and it wouldn;t allow stored proc to pass through?

Cdata Redshift connector: https://www.cdata.com/download/download.aspx?tag=yellowfin&sku=FRRJ-V&type=demo&file=full/FRRJ-V/setup.zip

And regarding 2. Stored procedure filters are automatically applied to every report that is built on the view, and access filters would further restrict data.

3. I think the sorting would only happen post-processing but unfortunately, I don't think we can change this.

Thanks,

Deepak

photo
1

Hi Deepak,

No I have not tried to use a cdata connector...I would hope that Yellowfin to put feature support into their native connectors. If you know for sure that stored procedures works with cdata connectors, I might give it a shot.

Best regards,

Ingo

photo
1

Hi Ingo,

Sure, please let me know how that goes because YF has officially launched CDATA connectors: https://community.yellowfinbi.com/announcement/yellowfin-launches-certified-cdata-connectors and the existing connectors will still be available for usage but they won't be modified/supported so I think the only way forward would be try CDATA connectors.

More information: https://community.yellowfinbi.com/announcement/discontinue-support-for-marketplace-content

Thanks,

Deepak

photo
1

Hi Deepak,

I think there is a bit of a misunderstanding. Amazon Redshift is a fully YF supported JDBC Data Source and not a market place connetor:

cea5b73a97f3c7382b8fb79cb1672b58


Therefore please find out with the Dev team or whoever, if the Redshift Data Source will be able to support stored procedures in the near future as Redshift is capable, but it is not recognized in the Yellowfin UI.

Many thanks and best regards,

Ingo

photo
1

Hi Ingo,

Apologies for the delay.

We will get in touch with the wider team with your query and get back to you with an update. Just wanted to let you know that the team is currently on Christmas break.

Thanks,

Deepak

photo
1

Hi Ingo,

I haven't had a chance to discuss this with the wider team yet. In the meanwhile, I managed to find connection details to an existing redshift instance and tried to create a Stored proc view using it:

434b3251949e314e2497ae81181adfe5


But when I couldn't finish creating a view because I do not have valid stored procedures that I can use (Unfortunately, I do not have access to redshift instance itself to create stored procs in it) :

15d5e95d44c288a7ae623d0934e65570


I have used the standard out of the box redshift that uses postgres driver to create the connection:

6b2027ca4f774f923dc2c272c161b36d


I have also tried connecting to the same redshift instance as a generic ds source with a redshift driver (attached):

01004526b7db73167704613662ee1994


Thanks,

Deepak

photo
1

Hi Deepak,

I'll have a look at those data source and driver combinations and get back to you

Best regards,

Ingo

photo
1

Sure Ingo. Please let me know how that goes.

Regards,

Deepak

photo
1

Hi Ingo,

Hope you're having a good week.

Just wanted to check-in and see how it's all going.

Kind regards,

Deepak Chaganti

photo
1

Hi Deepak,

I did some testing today and a stored procedure does not seem to work as Redshift only allows the return of a cursor and that does not seem to be supported in Yellowfin. But I found a different solution, using a Freehand SQL Report. (https://community.yellowfinbi.com/topic/parameters-in-freehand-sql-reports) That does almost work like I need it to.

With Common Table Expression I'm able to write a fitting query and even use Access Filter. But I found two issues with the Freehand SQL Report, one a bug (handling of using same access filter multiple times in a query) in my opinion and the other maybe solvable. So I would close this topic and create two tickets for the Freehand SQL Report.

Best regards,

Ingo

photo
1

Hi Ingo,

Sure, thanks for the update.

I can close this case on your behalf. The other issue related to Freehand SQL with access filters used multiple times in same query can be discussed with a new ticket.

Regards,

Deepak

Leave a Comment
 
Attach a file