Duplicate entry found error (Signals)

Vefa Gulecyuz shared this question 6 months ago
Answered

Hi guys, I receive below error multiple times for several dates while a Signal is running.

Duplicate entry found for 2020-01-12T00:00. In-memory aggregation is required. This can affect system performance, consider using a date field with correct granularity.

My date field is daily data in the attached format. I am trying to detect Signals on a weekly basis. I had similar Signals before on different datasets without this error.

My signal settings are also attached. All other parameters are default.

Why do you think this is happening?

Regards

Comments (7)

photo
1

And even though it completed the run successfully (but rather slowly) for the first two times, I received the attached fail note on Monitor Signals page.

In the logs it says: YF:2020-01-28 13:28:44: WARN (DBAction:safeDisconnect) - Null Connection - couldn't terminate...

photo
1

Hi Vefa,


Thanks for providing those screenshots.


Would you mind sending through a copy of your log folder to the https://ftp.yellowfin.bi portal? Please let me know the name of the file.


Thanks,

Simon

photo
1

Hi Simon, thanks for the reply. Logs are uploaded: logs_vefa_30-01.zip

photo
1

Hi Vefa,


Thanks for sending through those logs.


The problem is with date granularity and the way Yellowfin selects data. The calculation needs to be done so that it is applied once to each row/date, but when your date field's intrinsic granularity does not match the required granularity set in the job, that requires in-memory reaggregation. This would mean your calculation is split across multiple rows, and reaggregating them will produce an incorrect result.


From your logs, I can see that the issue you are experiencing is for what looks like to be the whole year of 2019. You will be able to see the error stack trace in your yellowfin.log file.


The solution is to use a date field that matches the required granularity. You can set up a date function field in the view and then use that to coerce the data into the correct granularity.


Let me know how you go.


Kind regards,

Simon

photo
1

Hi Vefa,


I'm just checking in with you, to see how you went with the previous recommendations. If you still require assistance, please respond to this message and I will get back to you shortly.


Kind regards,

Simon

photo
1

Hi Simon, thanks for the explanation.

I will try to use a date function with the granularity used in Signal settings. I hope this will improve performance.

You can close the ticket, thanks.

photo
1

Hi Vefa,


If you continue to have trouble with the error mentioned earlier, please don't hesitate to respond to the ticket, even when marked as answered and I will get back to you.


Good luck!

Simon