For Snowflake datasources, allow warehouse to be defined at report level

Dean Flinter shared this idea 2 years ago
Idea Logged

Hi,

I know this is DWH vendor specific but I think this would be incredibly useful

Snowflake allows elastic compute power with what they call virtual warehouses. These range from X-Small (1 node, 8 CPU cores, 12GB RAM) all the way up to 4X-Large (128 nodes). These can be dynamically changed on a whim depending on your requirements, either through some SQL commands or by appending the warehouse name to the connection string

In Yellowfin, when defining a Snowflake datasource connection, there is a space to specify a named warehouse. This means all connections on this source will use this warehouse.

For static workloads, this is fine but for ultimate flexibility we do not define this and use pass through authentication which means that Snowflake will use the default warehouse attached to the users' account.

For most of our use cases, this suits us perfectly as we can assign different warehouses to different users depending on their department/need etc. This also means we can monitor cost and performance for each department and adjust as needed

It also means that to achieve this level of flexibility, we do not have to create different datasources for each department and hence otherwise identical views and suites of reports that would all need to be maintained separately, leading to a lot of duplication of work.

However, we have had some use cases where a particular report is quite heavy and would benefit from a more powerful warehouse. We could adjust all users' default warehouse but that would incur extra cost for power that may not be needed in all cases

A solution to this would be to be able to define a warehouse per report. I imagine it would work similarly to the same field in the datasource properties where if left blank, it would default to the datasource one (or the user one if not defined there also) but if populated, it would override the datasource one and use the named warehouse. This way the report will get the extra power it needs but users would still be on their default warehouse for other reports where it is not defined

For dashboards this might mean several reports, all with different connection properties. While this is ideal in the sense that it respects the power levels we want for the report, I realise it might be alot. A suitable compromise might be to define this per dashboard also

The heirarchy then being Dashboard->Report->Datasource

Apologies for the essay but I thought it best to give the thinking behind this suggestion

If you need any more details, please let me know


Thanks

Dean

Replies (4)

photo
1

Hi Dean,

Thanks for reaching out to support. I'm running a little late and didn't get a chance to investigate thoroughly, but I THINK we might already have a developer task for this. I'll get back to you tomorrow.

Thanks,

Eric

photo
1

Hi Dean,

I found the task I remembered, for "additional parameters for Snowflake connection" which was used to add a "default schema" option in YF back in a 7.4 build. I'm not 100% sure if this idea would fall under that umbrella, but just to confirm your objective, the task for devs here would basically be:

"ability to define a default Snowflake data warehouse per report / dashboard"

Does that sound accurate? If so, I should be able to get this submitted / updated today depending on how we decide to handle the existing task.

Thanks,

Eric

photo
1

Hi Eric,

Yeah that previous task probably isn't applicable here

Not sure how succinct your task names have to be but perhaps the below is a little more comprehensive?

"ability to define a Snowflake virtual warehouse per report / dashboard that when defined is used instead of the one defined in the datasource"

I would also stress that the ability to leave both blank is also preserved


Thanks

Dean

photo
1

Hi Dean,

Thanks much for the reply here. I've gone ahead and created a new developer task to address this. Updated to the task will be provided here as they are available. I will in turn mark this ticket as Idea Logged for now; feel welcome to reply here with any related inquiries.

Thanks,

Eric

Leave a Comment
 
Attach a file