Software Open Access

ekzhu/datasketch: Improved performance for MinHash and MinHashLSH

Eric Zhu; Vadim Markovtsev; aastafiev; Wojciech Łukasiewicz; ae-foster; Jordan Martin; Ekevoo; Kevin Mann; Keyur Joshi; Spandan Thakur; Stefano Ortolani; Titusz; Vojtech Letal; Zac Bentley; fpug


MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nmm##2200000uu#4500</leader>
  <controlfield tag="005">20201216002712.0</controlfield>
  <controlfield tag="001">4323502</controlfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">@athenianco</subfield>
    <subfield code="a">Vadim Markovtsev</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">aastafiev</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Wojciech Łukasiewicz</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">ae-foster</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Jordan Martin</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Ekevoo</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Six Five Design</subfield>
    <subfield code="a">Kevin Mann</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">University of Illinois, Urbana-Champaign</subfield>
    <subfield code="a">Keyur Joshi</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Adobe</subfield>
    <subfield code="a">Spandan Thakur</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Stefano Ortolani</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Titusz</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">@blindspot-ai</subfield>
    <subfield code="a">Vojtech Letal</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Klaviyo</subfield>
    <subfield code="a">Zac Bentley</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">fpug</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">795687</subfield>
    <subfield code="z">md5:a3d3bce4aa309dab4bcd0ed08870cbc8</subfield>
    <subfield code="u">https://zenodo.org/record/4323502/files/ekzhu/datasketch-1.5.2.zip</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2020-12-15</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">software</subfield>
    <subfield code="o">oai:zenodo.org:4323502</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="a">Eric Zhu</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">ekzhu/datasketch: Improved performance for MinHash and MinHashLSH</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="a">Other (Open)</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;ul&gt;
&lt;li&gt;Performance improvement for MinHash's update method.&lt;/li&gt;
&lt;li&gt;Make MinHash updates 4.5X faster by using &lt;code&gt;update_batch&lt;/code&gt; method for bulk update on MinHash. [See API doc].(&lt;a href="http://ekzhu.com/datasketch/documentation.html#datasketch.MinHash.update_batch"&gt;http://ekzhu.com/datasketch/documentation.html#datasketch.MinHash.update_batch&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Further performance gain by using bulk generation of MinHash using &lt;code&gt;MinHash.bulk&lt;/code&gt; or &lt;code&gt;MinHash.generator&lt;/code&gt;. See &lt;a href="http://ekzhu.com/datasketch/documentation.html#datasketch.MinHash.bulk"&gt;API doc&lt;/a&gt; and &lt;a href="https://github.com/ekzhu/datasketch/pull/142"&gt;pull request&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Optional compression for MinHash LSH index by hashing the bucket key produced by &lt;code&gt;MinHashLSH._H&lt;/code&gt;. See &lt;a href="https://github.com/ekzhu/datasketch/pull/143"&gt;pull request&lt;/a&gt;. This leads to saving of memory/storage space used by the index.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thank you @Sinusoidal36!&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">url</subfield>
    <subfield code="i">isSupplementTo</subfield>
    <subfield code="a">https://github.com/ekzhu/datasketch/tree/1.5.2</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.598238</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.4323502</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">software</subfield>
  </datafield>
</record>
1,453
228
views
downloads
All versions This version
Views 1,45331
Downloads 2281
Data volume 276.3 MB795.7 kB
Unique views 1,26130
Unique downloads 1111

Share

Cite as