Geopack for Netherlands

Hello Bram,

Thanks for reaching out with your question. I've sent you, in a separate email, an invite to our FTP Server. You can use this to upload your geopack files. Please let me know here once you have done so and I'll create the appropriate requests for your Geopack.

Thanks,

Ryan

Hi Ryan,

Files have just been sent succesfully!

Regards,

Bram

Hello Bram,

Thanks for the update. I've received your files and provided them with the Enhancement request. I've also converted this ticket over to an Idea so that it can be tracked appropriately. You can access that on our Community here. I'll keep this post updated with any further information regarding an ETA on this.

Thanks,

Ryan

Hey Ryan,

Would you be so kind to give an update with regard to the Geopack?

Thanks a lot!

Regards,

Bram

Hi Bram,

I'll respond on Ryan's behalf while he is away.

GeoPacks can take up to 6 weeks for completion, so it could be a while before you here an update on this.

I've made a point to check in on the task in 2 weeks time to see how it's all going and will ensure you are kept up-to-date.

Please let us know if there was anything else you were after in the meantime.

Thanks,

David

Hi David,

Thanks for the reply. I have to say that 6 weeks is more then I had expected. Is it possible for you guys to verify that the input is correct for the geopack? It would be a pity when the progress is delayed due to such trivial problems.

Additionally, I believe that the pack of CSV's is currently about 350 MB in size. I believe that the Geopack of the USA that is installed with YF is about 200MB in size (and covers a slightly bigger part of the world). Is there any reason to believe that the Dutch Geopack might not perform due to its larger size?

I'd rather reduce the uncertainty now and know that the outcome will be good than find out in 6 weeks that there might be issues.

Regards,

Bram

Hello Bram,

I've reviewed your provided files and it appears that you've configured them all as guided in our Geopack Format Wiki Page. While I don't create them myself, your CSV's are formatted nicely in accordance to this page. Well done! Once there's more information on the development of your Geopack, I'll update this Idea post so you can track the progress.

On the note of the size, this is likely due to a higher density of Provinces or Regions than there is in the States, excuse my ignorance on the proper terminology for the area. I feel like any performance impacts seen from this would be easily mitigated with a small amount of memory increase to Yellowfin, if needed at all.

In short, your provided data is formatted well and I don't foresee there being any issues with the data format as it progresses through development. I'll keep my eye on this and update you as it progresses.

Thanks,

Ryan

Hello Bram,

The Development Team has begun reviewing the provided components for your Geopack. There's only two small things that need updated to finish the process.

First, the 'VisionBI_Geopack_NL.csv file contains only two letters "cc". Can you verify what this file is used for?

Second, I seem to have overlooked the absence of a Metadata file, which describes the links between each CSV. This will define your hierarchy between the CSV files. An example is available under 'Metadata File' here.

Finally, one small change is required to the data in your files. We require the string data to be quoted in double quotes, i.e. "Victoria", "VIC".

These last few things will be the last piece of the puzzle to creating the Geopack.

Thanks,

Ryan

Hi Ryan,

Thanks for your notification, I believe the VisionBI_Geopack_NL.csv was supposed to be the metadata file (it follows the naming example of the example geopack), but I guess i must have puched fn+c instad of ctrl+c when I tried to paste the data into the file.

The double quote is an easy adjustment. I further noted that my VisionBI_Geopack_NL3 file has 2 entries for every row so i can correct that as well.

Shall I recreate the files? Can i upload them here or do i post them to the FTP again?

Hi Bram,

Thanks for the quick reply. Please use the FTP to upload them once more. I'll update your request as soon as I receive them.

Thanks,

Ryan

Hi Ryan,

The new files have been uploaded and emailed to support@yellowfin.bi

Hello Bram,

Thanks for the quick turn around on updating your provided files. I've included them on our internal task for the Geopack creation. I did notice that the Metadata File describing the relationships between your CSV's only contains the following value: "1,3" and wanted to verify that's correct. If not, feel free to attach just that CSV here and I'll update the rest.

Thanks,

Ryan

Hi Ryan,

If i created the file correctly the file correctly it contains 2 columns with 3 rows.

Row 1: 1,3

Row 2: 1,3

Row 3: 1,3

If I have understood the documentation on http://wiki.yellowfin.com.au/display/USER73Plus/GeoPack+Format correctly this means that:

Row 1 links the VisionBI_Geopack_NL1 column 1 with VisionBI_Geopack_NL2 column 3

Row 2 links the VisionBI_Geopack_NL2 column 1 with VisionBI_Geopack_NL3 column 3

Row 3 links the VisionBI_Geopack_NL3 column 1 with VisionBI_Geopack_NL4 column 3

Since the pack does not include any metadata for the regions it resulted in a consistent column numbering of the 'foreign keys' between the documents.

Regards Bram

Hello Bram,

Thanks for the reply. I'll keep you updated on the progress of your Geopack creation.

Thanks,

Ryan

Hi Ryan,

Would you be so kind to provide me with an update regarding the status/progres? Need to report to my boss soon.

Thanks!

Bram

Hello Bram,

Apologies on the delayed response here. I've requested an update on the internal task and will update this post as soon as I have further information.

Thanks,

Ryan

Hi Ryan,

No news thusfar?

Regards,

Bram

Hi Bram,

After generating the Geopack, our Developer wasn't able to upload it to Yellowfin due to errors. There were some observations made about the new data set at that time:

1) The new version is about 25MB smaller. Is there any particular reason for the reduction in size?

2) It would appear that the data is given not in Lat / Long but in some type of projection. We wanted to ensure you're aware of this and that you're ok with that.

3) There are some redundant spaces in the data: "PV_CODE", "PV_NAAM" should be replaced with "PV_CODE","PV_NAAM". Or "POINT (245731.87103209776 581962.83074227453)" should be replaced with "POINT(245731.87103209776 581962.83074227453)". This may be a cause of the errors we're seeing when uploading.

As far as an ETA, this process can take up to 6 weeks. As we're still dialing in the format of your data, we can't say for sure at this time. Can you address the items above and re-upload the data with the mentioned changes on point 3 if everything else looks good?

Thanks,

Ryan

Hi Ryan,

1) I believe this has to do with the VisionBI_Geopack_NL.csv file. In my initial version, every neighbourhood(wijk) was inserted twice into the file. I corrected this in the second version. But since there is a lot less data in this file I suppose the size reduction can be explained. Thanks for pointing this out though.

2)/3) This will require some programming on my side. I plan to finish this tomorrow and will upload the files to the FTPserver as soon as I can.

With regard to the process, the first upload of the file dates from 28-06-2017. In my book that is more than 6 weeks ago. I understand that my files have been slightly different from the requested format, however that is also the reason that I asked to verify the format early on. In addition there seems to be a lot of time lost as a result of long feedback-loops.

I hope we can work together in order to complete the geopack a.s.a.p.

Kind regards

Bram

Hi Bram,

I look forward to the updated files.

Thanks,

Ryan

Hi Ryan,

I have spent quite some time now to adjust the projection. However sql always creates the wkt format as Long /Lat. Is this an issue for you?

Kind regards

Hello Bram,

Thanks for the update. Unfortunately, I don't have an answer for that question. We can either post the question for review by the Developer in AUS who is tasked with creating the Geopack (their day starts towards the end of mine) and await his reply, or we can provide the data along with the question and determine whether this is an issue. I recommend the latter, as if this isn't an issue the Developer will have the requested data.

Let me know how you'd like to proceed.

Thanks,

Ryan

Hi Ryan,

Today after some tweaking i succeeded in changing the order of Latitude and longitude.

I send 2 mails containg the files to support@yellowfin.com The third version contains one last mistake (I removed to many spaces) the fourth version should be exactly as you asked.

Please let me know wether these new sourcefiles are in the right format

Hi Bram,

Thanks for the quick turn around on these files. I've updated the Geopack task accordingly with the fourth version you provided. I'll keep informed as soon as I have an update.

Thank you,

Ryan

Hi Ryan,

Any news?

Thanks!

Hi Bram,

We have generated the first version of the geopack. At present, only the first two levels of data are being recognized. It's possible that it's because of some long names (i.e. "Aalsmeerderbrug/ Oude Meer/ Rozenburg / Schiphol Rijk"). We're in the process of determining the next step now, I'll keep you updated.

Thanks,

Ryan

Hi Ryan,

Thanks for the update, good to hear you are making progress! If the cause of the problem is content related and you need new input just give a shout.

Regards Bram

Hi Bram,

Will do! I'll get an update as soon as I can.

Thank you,

Ryan

Hi Bram,

Can you verify whether there exist any short names for the names of Wijkss and Buurts? We'd like to try shorter names to verify whether this is causing the issue at hand.

Thanks,

Ryan

Hi Bram,

I wanted to touch base on this and verify whether you can provide any shorter names for Wijkss and Burrts entries.

Thanks,

Ryan

Hi Ryan,

Apologies for the late reply, had a week off. How short woud they have tot be? Do you need proper names tot verify this to be the issue? Can't you Just try without the long names or on a subset?

Hi Bram,

To communicate from development, when they left only one row in the third level of data, the script which generates the geopack started recognizing the file properly. The problem with trying without the long names is that would require programmatically or manually editing the provided data to try to isolate the issue. I'll touch base with Development again and see about the subset idea.

Thanks,

Ryan

Hello Ryan,

I'm a collegae of Bram, he is out on an assignment for the coming period. Could you let me know the status of the Geopack?

Regards,

Jurgen

Hi Jurgen,

Thanks for checking in on this. Apologies for the delay in response, as I was out for holiday. I'll chase this up and let you know the status!

Thanks,

Ryan

Hi Bram,

Sorry for the delay in response on this one. At present we do not have plans to complete this Geopack.

In saying this, you should be able to grab this data yourself and put it into a DB for reporting.This is where we have directed clients to in the past: https://search.data.gov.au/

Now in terms of converting data to Shapes and WKT formats, the following article (while quite old) should still be relevant.

https://www.yellowfinbi.com/resources/forum/yfforum-how-does-mapping-work-in-yellowfin-thread-148904

If you need anything further on this let me know as I should be able to get a consultant to assist further.

Thanks,

Ryan