Database storage and database sizing

Guest ●

Hi Bruce,

Thanks for getting in touch. First, I noticed a couple of things in your question. Product Version was marked as 6.3 & older. I want to give you a heads up that 6.3 has been out of support for quite some time. Our current supported versions are 7.2 and 7.3. Also, since you're using Smart Reporting, I assume that you're a BMC client? I would get in touch with them about getting upgraded to a supported version.

Now on to your actual questions:

1. The database size question is always tricky, especially if you don't have the flexibility to auto-grow. Can you change the DB size ceiling manually as needed? It all depends on how large your instance is going to be. How many users, reports, unique logins, dashboards, etc. On one end of the spectrum, the largest DB I have on my personal laptop that Yellowfin runs out of is about 100 megabytes with me being the only user. On the other side, we have clients that administer to tens of thousands of users who have hundreds of gigabytes of storage allocated to the Yellowfin database. If you could share some estimations about the expected size of the instance we should be able to give you a better recommendation.

2. The Yellowfin configuration database really only stores application configuration and report metadata. There are only a couple of exceptions, and these are related to content added by the user. This includes the files for Connectors, images, or other materials uploaded through the plugin manager. Other than those and a small handful of similar files, everything else in there is metadata

3. There aren't any report files that get stored in the Yellowfin file structure. Since just about everything is generated on demand and would likely be outdated as soon as the report was run again, there is little point on hanging on to old stuff. However, I'm going to double check with the rest of my team on this just to make sure that I'm not forgetting about any strange corner cases. Please stay tuned for an update!

Thanks,

-Conner

Hi Bruce,

Thanks for getting in touch. First, I noticed a couple of things in your question. Product Version was marked as 6.3 & older. I want to give you a heads up that 6.3 has been out of support for quite some time. Our current supported versions are 7.2 and 7.3. Also, since you're using Smart Reporting, I assume that you're a BMC client? I would get in touch with them about getting upgraded to a supported version.

Now on to your actual questions:

1. The database size question is always tricky, especially if you don't have the flexibility to auto-grow. Can you change the DB size ceiling manually as needed? It all depends on how large your instance is going to be. How many users, reports, unique logins, dashboards, etc. On one end of the spectrum, the largest DB I have on my personal laptop that Yellowfin runs out of is about 100 megabytes with me being the only user. On the other side, we have clients that administer to tens of thousands of users who have hundreds of gigabytes of storage allocated to the Yellowfin database. If you could share some estimations about the expected size of the instance we should be able to give you a better recommendation.

2. The Yellowfin configuration database really only stores application configuration and report metadata. There are only a couple of exceptions, and these are related to content added by the user. This includes the files for Connectors, images, or other materials uploaded through the plugin manager. Other than those and a small handful of similar files, everything else in there is metadata

3. There aren't any report files that get stored in the Yellowfin file structure. Since just about everything is generated on demand and would likely be outdated as soon as the report was run again, there is little point on hanging on to old stuff. However, I'm going to double check with the rest of my team on this just to make sure that I'm not forgetting about any strange corner cases. Please stay tuned for an update!

Thanks,

-Conner

Reply URL

1

Bruce Steinberg ●

Awesome! thank you for getting back so quickly.

you are correct, we are BMC customers, but the support is not very helpful...yet.

Also, we must have a more current version so I don't think that is an issue.

We will have about 50 report writers with MAYBE 15k consumers. We are migrating off of Analytics, which has a large consumer base, but they let anyone create a report. So its not easy to query the metadata in Analytics, so getting good usage numbers is not easy.

If I read a little between the lines, if we are first configuring our lab, with only 2-3 users, we could probably start comfortably with 20-30gb...then adjust as we move towards production.

Do you know by any chance the size of the installation?

Thanks again Conner

Bruce Steinberg

206 902 7431

From: Yellowfin Support <support@yellowfin.bi>

Sent: Friday, September 8, 2017 9:42:02 AM

To: Bruce Steinberg

Subject: New Comment in "Database storage and database sizing"

Reply URL

1

Guest ●

Hey Bruce,

No problem! Apologies for the weekend delay.

Okay, good to hear that you're on a more current version. Ahh, understood about the 'fuzzy' projected numbers. I think that you're definitely reading between the lines correctly and that 20-30GB would be a good place to start for just the 2 or 3 users. Off the top of my head I am not sure what the size of the your installation will be, as I assume that it will also contain separate BMC components, but I can tell you that the size of a standalone installed 'stock' Yellowfin 7.3+ folder is in the ballpark of 400MB.

So while I think that a 20-30GB allocation for the metadata database is a good place to start, there are a couple of other thinks that I want to share with you so that you can scale your instance correctly when the time comes. We are happy to help you scale things as needed, but having a little extra background ahead of time can't hurt.

1) Yellowfin configuration database

As already mentioned, 20-30GB sounds like a good place to start for just a couple of users, but you will want to keep a close eye on how much storage the database actually uses. We've seen some really goofy things happen to instances that run out of RDBMS space on disk. A combination of due diligence, expanding the DB manually as necessary, and frequent backups should set you up for success here. If you notice parts of the database growing unexpectedly, here are a couple of notes on what may be going on:

https://community.yellowfinbi.com/knowledge-base/article/why-is-my-yellowfin-database-so-large-and-what-can-i-do-about-it

2) Java Virtual Machine (JVM) Memory used by Yellowfin

Yellowfin pulls memory resources from the pool of JVM memory that it is allocated at application startup. The default value is set relatively low, and is usually enough to serve smaller instances, but this is definitely a parameter that you will want to scale appropriately as your instance grows. A brief primer on how to that can be found here:

https://community.yellowfinbi.com/knowledge-base/article/what-is-jvm-max-memory-and-why-should-i-care?

3) Data Source connection settings

We've seen a handful of cases in the past where clients that have huge instances running on turbocharged hardware haven't changed any of the default settings for their data sources. As with many of the other settings, the default values should be fine, but as things start to grow, upping things like the number of connections, JDBC fetch size, etc. should be considered.

4) Caching

Yellowfin supports a variety of caching mechanisms to boost performance in parts of the application. Again, the defaults should be fine for starting out, but this is an area that we can look into when things grow. Here are a couple of notes on caching:

https://community.yellowfinbi.com/knowledge-base/article/what-is-event-caching-and-how-can-i-adjust-it

https://community.yellowfinbi.com/knowledge-base/article/yellowfin-internal-cache-options

Also, I have clarification from the rest of the team that we shouldn'tbe storing any whole reports or anything like that in the configuration database.

Whew! I know that this is a lot of info, but I figure that it's best to overshare on these topics, especially as a new instance is about to brought on line. Please don't hesitate to let us know if we can clear any of this up further, or if you have any other questions.

Thanks!

-Conner

Hey Bruce,

No problem! Apologies for the weekend delay.

Okay, good to hear that you're on a more current version. Ahh, understood about the 'fuzzy' projected numbers. I think that you're definitely reading between the lines correctly and that 20-30GB would be a good place to start for just the 2 or 3 users. Off the top of my head I am not sure what the size of the your installation will be, as I assume that it will also contain separate BMC components, but I can tell you that the size of a standalone installed 'stock' Yellowfin 7.3+ folder is in the ballpark of 400MB.

So while I think that a 20-30GB allocation for the metadata database is a good place to start, there are a couple of other thinks that I want to share with you so that you can scale your instance correctly when the time comes. We are happy to help you scale things as needed, but having a little extra background ahead of time can't hurt.

1) Yellowfin configuration database

As already mentioned, 20-30GB sounds like a good place to start for just a couple of users, but you will want to keep a close eye on how much storage the database actually uses. We've seen some really goofy things happen to instances that run out of RDBMS space on disk. A combination of due diligence, expanding the DB manually as necessary, and frequent backups should set you up for success here. If you notice parts of the database growing unexpectedly, here are a couple of notes on what may be going on:

https://community.yellowfinbi.com/knowledge-base/article/why-is-my-yellowfin-database-so-large-and-what-can-i-do-about-it

2) Java Virtual Machine (JVM) Memory used by Yellowfin

Yellowfin pulls memory resources from the pool of JVM memory that it is allocated at application startup. The default value is set relatively low, and is usually enough to serve smaller instances, but this is definitely a parameter that you will want to scale appropriately as your instance grows. A brief primer on how to that can be found here:

https://community.yellowfinbi.com/knowledge-base/article/what-is-jvm-max-memory-and-why-should-i-care?

3) Data Source connection settings

We've seen a handful of cases in the past where clients that have huge instances running on turbocharged hardware haven't changed any of the default settings for their data sources. As with many of the other settings, the default values should be fine, but as things start to grow, upping things like the number of connections, JDBC fetch size, etc. should be considered.

4) Caching

Yellowfin supports a variety of caching mechanisms to boost performance in parts of the application. Again, the defaults should be fine for starting out, but this is an area that we can look into when things grow. Here are a couple of notes on caching:

https://community.yellowfinbi.com/knowledge-base/article/what-is-event-caching-and-how-can-i-adjust-it

https://community.yellowfinbi.com/knowledge-base/article/yellowfin-internal-cache-options

Also, I have clarification from the rest of the team that we shouldn'tbe storing any whole reports or anything like that in the configuration database.

Whew! I know that this is a lot of info, but I figure that it's best to overshare on these topics, especially as a new instance is about to brought on line. Please don't hesitate to let us know if we can clear any of this up further, or if you have any other questions.

Thanks!

-Conner

Reply URL

1

Bruce Steinberg ●

Conner...you are AMAZING! Thank you so much!

Bruce Steinberg

USMC

DoD Development team

From: Yellowfin Support <support@yellowfin.bi>

Sent: Monday, September 11, 2017 9:03:02 AM

To: Bruce Steinberg

Subject: New Comment in "Database storage and database sizing"

Reply URL

1

Guest ●

Thanks Bruce!

Are you comfortable with me marking this one as 'Answered'?

You know where to find us, so don't hesitate to shoot any further questions or issues our way!

Best,

-Conner

Reply URL

1

Bruce Steinberg ●

Absolutely. Do we have a chance to rate it or just that its "Answered"?

If rate, then 5 star (assuming max 5 :))

I have no doubt I will see you all again.

Bruce.

From: Yellowfin Support <support@yellowfin.bi>

Sent: Monday, September 11, 2017 12:06:01 PM

To: Bruce Steinberg

Subject: New Comment in "Database storage and database sizing"

Reply URL

1

Guest ●

Thanks Bruce,

I'll go ahead and mark this answered then!

We do have a closed case survey that we're currently running, and it would be great if you wouldn't mind filling it out. (It'll only take you 30 seconds, promise!) Closed Request Survey.

Sounds good, we'll be here!

-Conner

Reply URL

1

Bruce Steinberg ●

Conner, awesome response and clear!! Zero'd right in on the answer to my question. THANK YOU!

Reply URL

Database storage and database sizing

Replies (8)