<feed xmlns='http://www.w3.org/2005/Atom'>
<title>user/iwakeh/collector, branch task-25522</title>
<subtitle>iwakeh's personal collector repository</subtitle>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/'/>
<entry>
<title>Add lines according to their count.</title>
<updated>2018-03-07T13:46:55+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-03-07T13:46:54+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=2c00f28ab7eaba0310d2d96eda7f81f2a301a4a5'/>
<id>2c00f28ab7eaba0310d2d96eda7f81f2a301a4a5</id>
<content type='text'>
Making the test from the previous commit pass and fixing task-25522.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Making the test from the previous commit pass and fixing task-25522.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add a failing test.</title>
<updated>2018-03-07T13:46:53+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-03-07T13:46:51+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=2e6fa506b3c9634837e6da4c8ad3dbf477936232'/>
<id>2e6fa506b3c9634837e6da4c8ad3dbf477936232</id>
<content type='text'>
Making a static method easier accessible for tests.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Making a static method easier accessible for tests.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add bastet to directory authorities to download votes for.</title>
<updated>2018-03-07T13:46:50+00:00</updated>
<author>
<name>Karsten Loesing</name>
<email>karsten.loesing@gmx.net</email>
</author>
<published>2018-03-07T13:46:50+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=727d4e54ef450204bb9b23404ae2cc9157fea12c'/>
<id>727d4e54ef450204bb9b23404ae2cc9157fea12c</id>
<content type='text'>
We did download bastet's votes in the past after reading the
consensus. But there could have been situations when there was no
consensus that we did not explicitly ask for bastet's vote.

Found per chance while setting up a CollecTor instance with a webstats
module.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We did download bastet's votes in the past after reading the
consensus. But there could have been situations when there was no
consensus that we did not explicitly ask for bastet's vote.

Found per chance while setting up a CollecTor instance with a webstats
module.
</pre>
</div>
</content>
</entry>
<entry>
<title>Describe file protocol for Tor web server logs.</title>
<updated>2018-02-26T15:24:50+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-02-26T15:24:50+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=190d90a800394cbe7e494b486e6ac7eac7377783'/>
<id>190d90a800394cbe7e494b486e6ac7eac7377783</id>
<content type='text'>
Part of task-20234.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Part of task-20234.
</pre>
</div>
</content>
</entry>
<entry>
<title>Bump version to 1.5.0-dev.</title>
<updated>2018-02-26T15:24:49+00:00</updated>
<author>
<name>Karsten Loesing</name>
<email>karsten.loesing@gmx.net</email>
</author>
<published>2018-02-26T15:24:49+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=ef1dfb6d32d5bb42a95052cd44790b6c586f38c9'/>
<id>ef1dfb6d32d5bb42a95052cd44790b6c586f38c9</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Prepare for 1.5.0 release.</title>
<updated>2018-02-26T14:06:51+00:00</updated>
<author>
<name>Karsten Loesing</name>
<email>karsten.loesing@gmx.net</email>
</author>
<published>2018-02-26T13:23:05+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=ddfa7bad243dc8e0a3105b14418794c87786e88f'/>
<id>ddfa7bad243dc8e0a3105b14418794c87786e88f</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Circumvent Collection (integer) size limit.</title>
<updated>2018-02-26T13:16:07+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-02-20T16:30:14+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=d05b4e4aee3bc15c3e4d5bac660dfcee5bc26279'/>
<id>d05b4e4aee3bc15c3e4d5bac660dfcee5bc26279</id>
<content type='text'>
Clean log lines immediately when they are read and also make use of sanitized
log's high redundancy immediately, i.e., continue with maps of
&lt;LocalDate, &lt;Map&lt;String, Long&gt;&gt;.

Rename method(s) to reflect what they do.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Clean log lines immediately when they are read and also make use of sanitized
log's high redundancy immediately, i.e., continue with maps of
&lt;LocalDate, &lt;Map&lt;String, Long&gt;&gt;.

Rename method(s) to reflect what they do.
</pre>
</div>
</content>
</entry>
<entry>
<title>Reduce memory footprint and wall time.</title>
<updated>2018-02-20T16:30:13+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-02-20T16:30:09+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=8557bf6255e6e3745088033e8e7bad7801421686'/>
<id>8557bf6255e6e3745088033e8e7bad7801421686</id>
<content type='text'>
Adapt to latest changes of metrics-lib (task-25329) and make use of the high
redundancy of logs (e.g. a 3G file might only contain 350 different lines).
This avoids OOM and array out of bounds exceptions for large files (&gt;2G) and
gives a speed-up of roughly 50%. (The earlier 66min are down to 34min for
meronense&amp;weschniakowii files plus two larger files.)

There is a BATCH constant, which could be tuned for processing speed. It is
logged for each webstats module run.  Currently, it is set to 100k.  This
was more or less arbitrarily chosen and used for all the tests.  A test run
using 500k didn't show significant differences.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Adapt to latest changes of metrics-lib (task-25329) and make use of the high
redundancy of logs (e.g. a 3G file might only contain 350 different lines).
This avoids OOM and array out of bounds exceptions for large files (&gt;2G) and
gives a speed-up of roughly 50%. (The earlier 66min are down to 34min for
meronense&amp;weschniakowii files plus two larger files.)

There is a BATCH constant, which could be tuned for processing speed. It is
logged for each webstats module run.  Currently, it is set to 100k.  This
was more or less arbitrarily chosen and used for all the tests.  A test run
using 500k didn't show significant differences.
</pre>
</div>
</content>
</entry>
<entry>
<title>Adapt CollecTor to latest metrics-lib master branch.</title>
<updated>2018-02-20T16:30:08+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2018-02-20T16:30:08+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=fbb35f75da022a23912b937b1825d8f216abad07'/>
<id>fbb35f75da022a23912b937b1825d8f216abad07</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add hasContent method to make even more use of DescriptorBuilder.</title>
<updated>2018-02-20T16:30:07+00:00</updated>
<author>
<name>iwakeh</name>
<email>iwakeh@torproject.org</email>
</author>
<published>2017-10-27T17:35:19+00:00</published>
<link rel='alternate' type='text/html' href='https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?id=5b68aaf8aa7c5f3769544061344e75f7884e87ef'/>
<id>5b68aaf8aa7c5f3769544061344e75f7884e87ef</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
