summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
...
* Merge branch 'metrics-lib'Damian Johnson2012-03-25
|\
| * Integer truncation could break reader's last modified checkDamian Johnson2012-03-25
| | | | | | | | | | | | When saving the last modified timestamps they're saved as ints. However, the timestamp itself is a float, so the following check for an unchanging timestamp would fail.
| * Fixing reader's pydoc exampleDamian Johnson2012-03-25
| |
| * Unit tests for server descriptor parsingDamian Johnson2012-03-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Borrowing heavily from Karsten's metrics-lib unit tests [1], plus a few of our own. This doesn't include everything, some are covered by util tests and others are effectively covered by parsing metrics and cached-descriptor files. That said, I wouldn't be against including more - my fingers are just getting numb... While looking through those tests I discovered these deltas... * When parsing the descriptor he validated that we had the correct first and last entry but we didn't. This was a bug, now fixed. * He checks that the 'protocols' line only contains numberic protocols but we accept any space separated content. The spec doesn't specifify what the protocols can be so I'm leaving this alone. * He had checks that a 'published' line with a year of '3012' or '1912' would fail. I can understand having a sanity check but the spec does not say that those dates are invalid so again leaving it as-is. * Metrics lib has validation for the exit policy but we don't. This is because we haven't yet implemented an ExitPolicy class - gsathya is currently working on that in... https://trac.torproject.org/projects/tor/ticket/5454 * Karsten is parsing and validating read/write-history lines. We, on the other hand, log an INFO level warning when these appear and assert in the integ tests that they (along with 'eventdns') do not exist in our cached-descriptor file. My understanding is that these are deprecated entries and should not appear outside of the extra-info descriptors. Maybe I'm misunderstanding something here... On everything else we either match or are maybe slightly more strict about only allowing content conforming to the spec. [1] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java
| * Skipping auth message check when cookie missingDamian Johnson2012-03-25
| | | | | | | | | | | | | | When we first run our tests we don't have an authentication cookie unless we've run with the RUN_COOKIE target. We have special handling for this, but it did not account for running with an open connection (the default) in which case we'd encounter a second exception while trying to check the failure message.
| * Miswired 'ONLINE' and 'RELATIVE' targetsDamian Johnson2012-03-25
| | | | | | | | | | | | The test runner still used the 'test.' prefixed configuration keys rather than 'integ.'. This caused the 'ONLINE' and 'RELATIVE' testing targets to be ignored.
| * Skipping integ test on cached descriptors if not availableDamian Johnson2012-03-25
| | | | | | | | | | | | The first time we run our integration tests our data directory is not yet fully populated. This causes our test that runs over the cached descriptor file to fail. Adding a check that the cached descriptors exist before running it.
| * Including deprecated server descriptor attributesDamian Johnson2012-03-25
| | | | | | | | | | | | | | | | I have mixed feelings about cluttering the server descriptor with dead fields, but since we're already this close to being perfectly spec conformant we might as well include this last bit. Adding the fields but logging an info level warning if we see it, and checking in our cached descriptor that they do not exist.
| * Integ test to validate our cached descriptor fileDamian Johnson2012-03-24
| | | | | | | | | | | | | | | | | | Parsing all of the descriptors in our cached descriptors file to see if it has anything that'll make us choke (and by extension indicate either a tor or stem bug). This is an important yet lengthy test, taking around four second. Targets do not effect this test so only running it once for the test run.
| * Dropping attribute for block typesDamian Johnson2012-03-24
| | | | | | | | | | | | We already have enough attributes, and parsing out the armor type is redundant since it's within the armor itself. Besides, it's not likely to change and I doubt callers will ever care about it.
| * Missing version 3 attributesDamian Johnson2012-03-24
| | | | | | | | | | | | Adding the server descriptor attributes that are new as of version 3. The only attributes that I'm still excluding are eventdns (because it's dead) and read/write-history (since they're extra-info now).
| * Allowing for extra args on lines with multipleDamian Johnson2012-03-24
| | | | | | | | | | | | | | | | | | | | Well, crap. I've been developing against the version 2 spec on the assumption that was both the majority use case and what my test instance was running. However, that's evidently pretty ancient and I've been running against version 3 all along. First change I'm spotting in the spec is that lines with multiple arguments should ignore extras so doing that.
| * Server descriptor test for metrics contentDamian Johnson2012-03-24
| | | | | | | | | | | | | | | | Test for parsing a single server descriptor and checking its content. This includes a few fixes and improvements for the ServerDescriptorV2 class, the most intersting of which is that declaring instance variables prior to init makes a single reference. Hence all server descriptors were using the same exit policy. Oops. :)
| * Integration test for reading non-descriptor filesDamian Johnson2012-03-24
| | | | | | | | Integ test for reading non-descriptor plaintext and binary data.
| * Cleaning up prior descriptor implementationsDamian Johnson2012-03-24
| | | | | | | | Adding header pydocs and made a variety of small fixes and naming improvements.
| * Server descriptor parsing fixesDamian Johnson2012-03-23
| | | | | | | | | | | | | | Resolving the variety of issues that were breaking the integ tests. We still don't have any tests to actually verify the server descriptor parsing, but at least we're now exercising it on some real data... and not dieing in a fire! Progress!
| * Resolving circular dependency with server_descriptorDamian Johnson2012-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ... damn this was a pita to debug. A long explanation of the issue is: http://stackoverflow.com/questions/8991520/python-conditional-module-object-has-no-attribute-error- with-personal-package Short story is that the stem.descriptor module imported server_descriptor prior to being fully initialized which caused a server_descriptor use of stem.descriptor.Descriptor to fail. The confusing was that it was actually reporting failure in using "stem.descriptor" (the module itself) which made no sense because it was listed within sys.modules. It turns out that the python importer uses module attributes rather than sys.modules mappings to resolve statements (ie, it was doing getattr(stem, "descriptor")). However, a module's added as an attribute of its parent when its *finished* importing. This fixes the unit tests, but integ tests are still getting stuck when they use the new server_descriptor parsing.
| * Cleaning up minor import issuesDamian Johnson2012-03-23
| |
| * Merging descriptor.py with module initDamian Johnson2012-03-23
| | | | | | | | | | | | | | | | | | | | The syntax for using the descriptor module was stupid ("import stem.descriptor.descriptor"). Using the json module as an example for using __init__.py for base functions. Common ones will live in 'stem.descritor' but the contents, like 'stem.descriptor.reader', are also accessable. This is similar to how the os module works, though in that case they're using a lot more python magic to also support multiple operating systems.
| * Using server descriptor when parsingDamian Johnson2012-03-23
| | | | | | | | | | Parsing server descriptors via the new server descriptor class. This breaks the tests and will need quite a bit of work. :)
| * Pylint corrections for server descriptorDamian Johnson2012-03-23
| | | | | | | | | | Corrections from running pylint over the untested server descriptor class (all basic problems that would keep it from running).
| * Parsing descriptor files and annotationsDamian Johnson2012-03-23
| | | | | | | | | | | | | | | | Completely untested change to finish the implementation of server descriptors. This parses a cached descriptor file into individual descriptors and annotations. Next comes unit and integ tests to start exercising it in an automated way.
| * Including exit policy among the attributesDamian Johnson2012-03-20
| | | | | | | | | | | | We'll later have a proper ExitPolicy class but that's a small project of its own. Until then this will just provide the list of strings that make up the exit policy.
| * Making server descriptor validation optionalDamian Johnson2012-03-20
| | | | | | | | | | | | Adding constructor pydocs and an argument to skip validation. This should have performance advantages and allow for best-effort parsing of malformed descriptor content.
| * Handling the server descriptor's contact and family linesDamian Johnson2012-03-19
| |
| * Handling the server descriptor's router-signature lineDamian Johnson2012-03-19
| |
| * Karsten already had a ticket for opt removalDamian Johnson2012-03-19
| |
| * Generalizing PGP style block parsingDamian Johnson2012-03-19
| | | | | | | | | | | | | | Descriptors include blocks for both "RSA PUBLIC KEY" and "SIGNATURE" so generalizing that parser (the spec around how to handle these blocks is wrong and I'll fix that later). I'm also including the block type since callers will want that to be prepared for encryption migrations.
| * Handling the server descriptor's onion-key and signing-key linesDamian Johnson2012-03-18
| |
| * Parsing descriptor signature blocksDamian Johnson2012-03-18
| |
| * Handling the server descriptor's uptime lineDamian Johnson2012-03-18
| |
| * Handling the server descriptor's hibernating lineDamian Johnson2012-03-18
| |
| * Handling the server descriptor's fingerprint lineDamian Johnson2012-03-18
| |
| * Stripping off the 'opt ' keyline prefixDamian Johnson2012-03-18
| |
| * Handling the server descriptor's published lineDamian Johnson2012-03-17
| | | | | | | | | | | | | | Ugggg, I'm always filled with hate when I need to touch the datetime module. I'm not entirely sure if I'm supposed to be associating timezone information with this datetime object or not - and if I am how to get the f*ing thing to realize that it's GMT.
| * Handling the server descriptor's platform lineDamian Johnson2012-03-17
| |
| * Handling the server descriptor's bandwidth lineDamian Johnson2012-03-17
| |
| * Validation for server descriptor router lineDamian Johnson2012-03-17
| | | | | | | | | | Checking that there aren't any surprises from a server descriptor's router line.
| * Function to check validity of a portDamian Johnson2012-03-17
| |
| * Funcions to check validity of nicknames and fingerprintsDamian Johnson2012-03-17
| | | | | | | | | | | | Couple functions to check if a string is a valid nickname or fingerprint, plus tests. I needed to guess at the definition of a HEXDIG but otherwise nothing very interesting.
| * Implementing and testing is_valid_ip_address()Damian Johnson2012-03-17
| | | | | | | | | | | | Moving over an arm function for testing if a string is a valid IPv4 address. Also throwing in some unit tests and a fix for entries with leading zeros (for instance "1.2.3.01").
| * Completely untested start for descriptor parsingDamian Johnson2012-03-17
| | | | | | | | | | | | Starting on server descriptor parsing. This is gonna need some other classes and helper functions (such as an ExitPolicy and is_valid_ip_address()) but otherwise should be pretty straight forward.
| * Stub for identifying descriptor typesDamian Johnson2012-03-17
| | | | | | | | | | | | The various tor descriptor types were not made to be readily identifyable, so we'll probably need to guess their type based on their first line. Ideally that first line would explicitly say the type and version, but oh well...
| * Descriptor module with common functionalityDamian Johnson2012-03-14
| | | | | | | | | | | | | | | | | | | | Adding the descriptor module which has the Descriptor class (the common parent for all descriptor types) and the function for parsing descriptors from a file. This parser will later do type recognition to figure out how descriptors should be parsed. This is identical to how metrics-lib does this. Nice advantage is that we can now move all of the remaining 'TODO' notes out of the reader. It's done. :)
| * Making _get_raw_tar_descriptors() thread safeDamian Johnson2012-03-13
| | | | | | | | | | | | At present integ tests are single threaded, but there's a project proposal to change that and spawn threads for each target. Fixing this helper function so it won't be sad when that day comes.
| * Condensing basic example a tiny bitDamian Johnson2012-03-13
| | | | | | | | | | Combining the constructor and 'with' lines since that's probably the common case. The second example already shows a standalone constructor anyway.
| * Inverting save_processed_files arg orderDamian Johnson2012-03-13
| | | | | | | | Little nicer when the usually-static argument is first imho.
| * Adding test for the DescriptorReader's buffer_sizeDamian Johnson2012-03-13
| | | | | | | | | | Simple test that we don't read ahead more than the DescriptorReader's buffer size.
| * Adding tarball support to the DescriptorReaderDamian Johnson2012-03-12
| | | | | | | | | | | | | | Adding support for reading directly from tarballs (which is how metrics are commonly fetched). This supports all forms of compression that the tarfile module does (gzip and bz2 among other). Including some tests and archives to read against.
| * Minor pylint correctionsDamian Johnson2012-03-12
| |
generated by cgit v1.2.1 (git 2.18.0) at 2024-02-08 06:29:59 +0000