New Topic

Community Topic Question Data Sources

Lee Roquet

NA Business Team

David Registro

APAC SD Team

Yuu Asakawa

Agents

Category Data Sources

Product Version 8.0

Is AWS S3 supported with the Yellowfin JSON Connector?

Justin Hewitt shared this question ago

Answered

When will you support reading JSON files from an AWS S3 bucket ?

The same question

Best Answer

Justin Hewitt ●

AWS S3 support is coming in V2 of the JSON Connector. Check the Marketplace for the latest updated version.

Replies (5)

Justin Hewitt ● ●

AWS S3 support is coming in V2 of the JSON Connector. Check the Marketplace for the latest updated version.

Reply URL

Lee Roquet ●

We have developed S3 connectivity for the JSON connector

This allows you to connect to an S3 bucket with JSON files and read those files (much like the existing FTP interface)

The connector update is BETA right now.

/AmBEq8M9wjzJAAAAAElFTkSuQmCCAA==

Thanks,

Lee

Reply URL

Yuu Asakawa ●

Is it possible to provide me with document of connecting to S3?It doesn't seem to be written on the wiki, and I can't find any YouTube videos.

Reply URL

David Registro ●

Hi Yuu,

The original JSON connector can read JSON files from a S3 Bucket it can do so in either date/time read only order or process/delete ordering. However, the problem with the original JSON connector is that it stored the results in an internal BLOB JSON file, so will slow down over time.

You will need to do it via REST, but I don't have any detailed documentation on how to do this.

If there is a way for us to gain access to your S3 bucket (or a test one) we can play with it and figure it out. I understand this may need to be switched to a private ticket to provide connection details.

Thanks,David

Reply URL

Yuu Asakawa ●

Hi David,

Thank you for your Information.

I am sad that there is no documentation...

Can you consider posting it on the wiki?

As for the our test bucket, please see this private ticket.

I can't access this private ticket, but Toshimitsu should have posted information of connecting to the our S3 bucket.

Regards,

Yuu.

Reply URL

David Registro ●

Thanks Yuu! I can see the ticket and have given you access, so you can now access any ticket :).Looks like Mike has been working with Toshi on this. so I'll add these notes there.

But yes, you're correct.. if this really is possible we need to document how.

The problem is trying to figure out how first .

Thanks,

David

URL

Mike Sheehan ●

Hi Yuu,

After speaking with our global consulting team it was pointed out that you can use Amazon Athena to query S3 buckets, and we officially support Athena, so the consensus is that Athena is our officially recommended method of querying S3 buckets at this time. Please let me know if you have any other questions on this.

Regards,

Mike

URL

Yuu Asakawa ●

Hi Mike,

Thanks for any useful information. But that doesn't answer my question...Justin gave me the answer.

Reagrds,

Yuu.

URL

Mike Sheehan ●

Hi Yuu,

Thanks for letting me know. Would it be possible to share that info here?

Regards,

Mike

URL

Yuu Asakawa ●

Mi Mike,

I'm sorry, I only write Japanese.

I guess Toshimitsu has emailed the mailing list.

Regards,

Yuu.

URL

Mike Sheehan ●

Hi Yuu,

No problem, yes he did.

After speaking with Justin about the JSON connector, he found that if we wanted to update the data, the filenames placed in S3 had to be unique, and the step in ETL was only available for URLs and REST.

This is good because we have verified that the customer can use it, but basically we recommend using Amazon Athena.

Because Yellowfin keeps the files and filenames it reads in using the JSON connector. There is no way to delete or update retained data. As a result, Yellowfin performance can degrade with each use.

The results of our verification are summarized below. ・If you load a file that does not have the Variable name specified in PathBuilder, the value of that attribute is loaded as Null.

・Attribute data types can be cast (TEXT → NUMERIC, NUMERIC → TEXT) and read if possible. If casting is not possible (for example, 'word' → NUMERIC), the value is null.

・If you select ScheduleType: Marker, the imported file remains.

・If you select ScheduleType: Delete, the imported file is deleted.

Hopefully this can help others! These points considered, I'll go ahead and close this back out for now then.

Regards,

Mike

URL

Yuu Asakawa ●

Hi Mike,

Thank you for sharing.

That's the email he sent.

I received the exact same email.

Regards,

Yuu.

URL

Mike Sheehan ●

Hi Yuu,

Thanks for confirming!

Regards,

Mike

URL