Stem Docs

CollecTor

CollecTor

Descriptor archives are available from CollecTor. If you need Tor's topology at a prior point in time this is the place to go!

With CollecTor you can either read descriptors directly...

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)

# provide yesterday's exits

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    print('  %s (%s)' % (desc.nickname, desc.fingerprint))

... or download the descriptors to disk and read them later.

import datetime
import stem.descriptor
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)
cache_dir = '~/descriptor_cache/server_desc_today'

collector = stem.descriptor.collector.CollecTor()

for f in collector.files('server-descriptor', start = yesterday):
  f.download(cache_dir)

# then later...

for f in collector.files('server-descriptor', start = yesterday):
  for desc in f.read(cache_dir):
    if desc.exit_policy.is_exiting_allowed():
      print('  %s (%s)' % (desc.nickname, desc.fingerprint))
get_instance - Provides a singleton CollecTor used for...
  |- get_server_descriptors - published server descriptors
  |- get_extrainfo_descriptors - published extrainfo descriptors
  |- get_microdescriptors - published microdescriptors
  |- get_consensus - published router status entries
  |
  |- get_key_certificates - authority key certificates
  |- get_bandwidth_files - bandwidth authority heuristics
  +- get_exit_lists - TorDNSEL exit list

File - Individual file residing within CollecTor
  |- read - provides descriptors from this file
  +- download - download this file to disk

CollecTor - Downloader for descriptors from CollecTor
  |- get_server_descriptors - published server descriptors
  |- get_extrainfo_descriptors - published extrainfo descriptors
  |- get_microdescriptors - published microdescriptors
  |- get_consensus - published router status entries
  |
  |- get_key_certificates - authority key certificates
  |- get_bandwidth_files - bandwidth authority heuristics
  |- get_exit_lists - TorDNSEL exit list
  |
  |- index - metadata for content available from CollecTor
  +- files - files available from CollecTor

New in version 1.8.0.

stem.descriptor.collector.get_instance()[source]

Provides the singleton CollecTor used for this module's shorthand functions.

Returns:singleton CollecTor instance
stem.descriptor.collector.get_server_descriptors(start=None, end=None, cache_to=None, bridge=False, timeout=None, retries=3)[source]

Shorthand for get_server_descriptors() on our singleton instance.

stem.descriptor.collector.get_extrainfo_descriptors(start=None, end=None, cache_to=None, bridge=False, timeout=None, retries=3)[source]

Shorthand for get_extrainfo_descriptors() on our singleton instance.

stem.descriptor.collector.get_microdescriptors(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Shorthand for get_microdescriptors() on our singleton instance.

stem.descriptor.collector.get_consensus(start=None, end=None, cache_to=None, document_handler='ENTRIES', version=3, microdescriptor=False, bridge=False, timeout=None, retries=3)[source]

Shorthand for get_consensus() on our singleton instance.

stem.descriptor.collector.get_key_certificates(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Shorthand for get_key_certificates() on our singleton instance.

stem.descriptor.collector.get_bandwidth_files(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Shorthand for get_bandwidth_files() on our singleton instance.

stem.descriptor.collector.get_exit_lists(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Shorthand for get_exit_lists() on our singleton instance.

class stem.descriptor.collector.File(path, size, last_modified)[source]

Bases: object

File within CollecTor.

Variables:
  • path (str) -- file path within collector
  • compression (stem.descriptor.Compression) -- file compression, None if this cannot be determined
  • size (int) -- size of the file
  • start (datetime) -- beginning of the time range descriptors are for, None if this cannot be determined
  • end (datetime) -- ending of the time range descriptors are for, None if this cannot be determined
  • last_modified (datetime) -- when the file was last modified
read(directory=None, descriptor_type=None, document_handler='ENTRIES', timeout=None, retries=3)[source]

Provides descriptors from this archive. Descriptors are downloaded or read from disk as follows...

  • If this file has already been downloaded through :func:`~stem.descriptor.collector.CollecTor.download' these descriptors are read from disk.
  • If a directory argument is provided and the file is already present these descriptors are read from disk.
  • If a directory argument is provided and the file is not present the file is downloaded this location then read.
  • If the file has neither been downloaded and no directory argument is provided then the file is downloaded to a temporary directory that's deleted after it is read.
Parameters:
Returns:

iterator for Descriptor instances in the file

Raises :
  • ValueError if unable to determine the descirptor type
  • TypeError if we cannot parse this descriptor type
  • DownloadFailed if the download fails
download(directory, decompress=True, timeout=None, retries=3)[source]

Downloads this file to the given location. If a file already exists this is a no-op.

Parameters:
  • directory (str) -- destination to download into
  • decompress (bool) -- decompress written file
  • timeout (int) -- timeout when connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose
Returns:

str with the path we downloaded to

Raises :

DownloadFailed if the download fails

class stem.descriptor.collector.CollecTor(retries=2, timeout=None)[source]

Bases: object

Downloader for descriptors from CollecTor. The contents of CollecTor are provided in an index that's fetched as required.

Variables:
  • retries (int) -- number of times to attempt the request if downloading it fails
  • timeout (float) -- duration before we'll time out our request
get_server_descriptors(start=None, end=None, cache_to=None, bridge=False, timeout=None, retries=3)[source]

Provides server descriptors published during the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • bridge (bool) -- standard descriptors if False, bridge if True
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of ServerDescriptor for the given time range

Raises :

DownloadFailed if the download fails

get_extrainfo_descriptors(start=None, end=None, cache_to=None, bridge=False, timeout=None, retries=3)[source]

Provides extrainfo descriptors published during the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • bridge (bool) -- standard descriptors if False, bridge if True
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of RelayExtraInfoDescriptor for the given time range

Raises :

DownloadFailed if the download fails

get_microdescriptors(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Provides microdescriptors published during the given time range, sorted oldest to newest. Unlike server/extrainfo descriptors, microdescriptors change very infrequently...

"Microdescriptors are expected to be relatively static and only change
about once per week." -dir-spec section 3.3

CollecTor archives only contain microdescriptors that change, so hourly tarballs often contain very few.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of :class:`~stem.descriptor.microdescriptor.Microdescriptor for the given time range

Raises :

DownloadFailed if the download fails

get_consensus(start=None, end=None, cache_to=None, document_handler='ENTRIES', version=3, microdescriptor=False, bridge=False, timeout=None, retries=3)[source]

Provides consensus router status entries published during the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • document_handler (stem.descriptor.__init__.DocumentHandler) -- method in which to parse a NetworkStatusDocument
  • version (int) -- consensus variant to retrieve (versions 2 or 3)
  • microdescriptor (bool) -- provides the microdescriptor consensus if True, standard consensus otherwise
  • bridge (bool) -- standard descriptors if False, bridge if True
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of RouterStatusEntry for the given time range

Raises :

DownloadFailed if the download fails

get_key_certificates(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Directory authority key certificates for the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of :class:`~stem.descriptor.networkstatus.KeyCertificate for the given time range

Raises :

DownloadFailed if the download fails

get_bandwidth_files(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

Bandwidth authority heuristics for the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of :class:`~stem.descriptor.bandwidth_file.BandwidthFile for the given time range

Raises :

DownloadFailed if the download fails

get_exit_lists(start=None, end=None, cache_to=None, timeout=None, retries=3)[source]

TorDNSEL exit lists for the given time range, sorted oldest to newest.

Parameters:
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
  • cache_to (str) -- directory to cache archives into, if an archive is available here it is not downloaded
  • timeout (int) -- timeout for downloading each individual archive when the connection becomes idle, no timeout applied if None
  • retries (int) -- maximum attempts to impose on a per-archive basis
Returns:

iterator of :class:`~stem.descriptor.tordnsel.TorDNSEL for the given time range

Raises :

DownloadFailed if the download fails

index(compression='best')[source]

Provides the archives available in CollecTor.

Parameters:

compression (descriptor.Compression) -- compression type to download from, if undefiled we'll use the best decompression available

Returns:

Index with the archive contents

Raises :

If unable to retrieve the index this provide...

  • ValueError if json is malformed
  • IOError if unable to decompress
  • DownloadFailed if the download fails
files(descriptor_type=None, start=None, end=None)[source]

Provides files CollecTor presently has, sorted oldest to newest.

Parameters:
  • descriptor_type (str) -- descriptor type or prefix to retrieve
  • start (datetime.datetime) -- time range to begin with
  • end (datetime.datetime) -- time range to end with
Returns:

list of File

Raises :

If unable to retrieve the index this provide...

  • ValueError if json is malformed
  • IOError if unable to decompress
  • DownloadFailed if the download fails