Forums » Bugs & Problems Search

song/identity inconsistencies? New Reply

Author Post
Posts: 5
Registered: Jun 03, 2012

Hi,

I'm working on a research project where we need to evaluate a number of fingerprinting technologies.

I've been running a series of tests using ENMFP on Linux, with both the precompiled codegen.Linux-i686 and also with the C++ test.cxx file (with content prepare using ffmpeg as in the code comments). I have a set of 50 test music files, and am taking slices of 30s, 25s, 20s, 15s, 10s and 5s duration at in-points of 0s, 15s, 30s, 45s and 60s. I am also running fingerprints on the whole file.

I am confused by some of the results. Many of the tracks haven't been identified with the full file, or for example with a 0s in-point and 30s duration, but then (surprisingly?) have correctly identified with a 0s in-point and 20s or 15s duration. Any ideas why this would be the case?

Also, comparing the results of codegen.Linux-i686 and test.cxx is showing some variations. I've not fully analysed this, but there are differences in the scores as well as differences in some cases where a match was / was not made.

Any help would be appreciated, as we'd like to continue working with ENMFP if possible.

Thank you.

Posts: 197
Registered: Sep 05, 2008

HI s.thompson --

we don't test full file code matching much, so it's possible you're experiencing a bug we haven't uncovered yet. We internally use a time of roughly 20s for almost all of our ENMFP matches. We can take a look at this soon but our advice for now is to always query us with 20-30s of audio starting from roughly 30s into the file.

Regarding your second point, do you mean that test.cxx with libcodegen vs the compiled codegen.Linux-i686? If so, what are you using for the decoder in the test.cxx case? It should be exactly the same otherwise.

Posts: 197
Registered: Sep 05, 2008

Also, it would be great if you could send us an example file that fails at full file but works at 20 or 30s duration. Send it to brian@echonest.com . Thanks!

Posts: 5
Registered: Jun 03, 2012

Hopefully you should have an email in your inbox now - bounced the first time because of its size. Thanks.

Posts: 197
Registered: Sep 05, 2008

Hey s.thompson, thanks for the mails. We did find an issue that your tests exposed, I'll follow up here when we know more.

Posts: 5
Registered: Jun 03, 2012

Hi, thanks for looking into them too. Look forward to hearing from you.

Posts: 197
Registered: Sep 05, 2008

Hey s.thompson, here's a brief report: there's two problems at the moment:

  • Our public ENMFP resolving server is more out of date than we would like due to a big infrastructure move. What's happening a lot in your case is your queries match a track in our DB that has been re-assigned to another song via our internal ingestion process and we can't return the out-of-date metadata so we return no match. We have a plan to put this back in sync but it will be roughly a week or two before that happens. I'll update here when it does.
  • For short or weak matches (in that second dump you sent us) our reference inverted index that retrieves the candidates can sometimes return many possible matches all with the same score in an unordered fashion. Since that database is sharded and our API hits it from different worker endpoints (as in a load balancer) the sorting can be non-deterministic. This is a bug in our implementation and we'll be fixing it ASAP, but again note this should only happen when the matches are "weak."

Once we fix these two issues you should have a lot better accuracy with ENMFP. Thanks very much for the detailed report and data, it's very appreciated.

Posts: 5
Registered: Jun 03, 2012

Hi Brian, thanks for the update. We're going to be increasing the size and variety of our sample set and continuing to run evaluations. I'll update you with new results when your fixes are in place. Stephen.

Posts: 197
Registered: Sep 05, 2008

Please see our update on ENMFP.

Posts: 197
Registered: Sep 05, 2008

s.thompson -- try again if you can? We just rolled out a fix for your issues.

Posts: 5
Registered: Jun 03, 2012

Will do, thanks. I've an extended test I'll run too. Will email you the results.

Reply to this Thread

You must log in to post a reply.