Stem Docs

Descriptor Remote

Descriptor Remote

Module for remotely retrieving descriptors from directory authorities and mirrors. This is the simplest method for getting current tor descriptor information...

import stem.descriptor.remote

for desc in stem.descriptor.remote.get_server_descriptors():
  if desc.exit_policy.is_exiting_allowed():
    print('  %s (%s)' % (desc.nickname, desc.fingerprint))

More custom downloading behavior can be done through the DescriptorDownloader class, which issues Query instances to get you descriptor content. For example...

from stem.descriptor.remote import DescriptorDownloader

downloader = DescriptorDownloader(
  use_mirrors = True,
  timeout = 10,
)

query = downloader.get_server_descriptors()

print('Exit Relays:')

try:
  for desc in query.run():
    if desc.exit_policy.is_exiting_allowed():
      print('  %s (%s)' % (desc.nickname, desc.fingerprint))

  print
  print('Query took %0.2f seconds' % query.runtime)
except Exception as exc:
  print('Unable to retrieve the server descriptors: %s' % exc)
get_instance - Provides a singleton DescriptorDownloader used for...
  |- their_server_descriptor - provides the server descriptor of the relay we download from
  |- get_server_descriptors - provides present server descriptors
  |- get_extrainfo_descriptors - provides present extrainfo descriptors
  +- get_consensus - provides the present consensus or router status entries

Query - Asynchronous request to download tor descriptors
  |- start - issues the query if it isn't already running
  +- run - blocks until the request is finished and provides the results

DescriptorDownloader - Configurable class for issuing queries
  |- use_directory_mirrors - use directory mirrors to download future descriptors
  |- their_server_descriptor - provides the server descriptor of the relay we download from
  |- get_server_descriptors - provides present server descriptors
  |- get_extrainfo_descriptors - provides present extrainfo descriptors
  |- get_microdescriptors - provides present microdescriptors with the given digests
  |- get_consensus - provides the present consensus or router status entries
  |- get_key_certificates - provides present authority key certificates
  +- query - request an arbitrary descriptor resource

New in version 1.1.0.

stem.descriptor.remote.MAX_FINGERPRINTS

Maximum number of descriptors that can requested at a time by their fingerprints.

stem.descriptor.remote.MAX_MICRODESCRIPTOR_HASHES

Maximum number of microdescriptors that can requested at a time by their hashes.

stem.descriptor.remote.Compression(enum)

Compression when downloading descriptors.

New in version 1.7.0.

Compression Description
PLAINTEXT Uncompressed data.
GZIP GZip compression.
ZSTD Zstandard compression, this requires the zstandard module.
LZMA LZMA compression, this requires the 'lzma module <https://docs.python.org/3/library/lzma.html>`_.
stem.descriptor.remote.get_instance()[source]

Provides the singleton DescriptorDownloader used for the following functions...

New in version 1.5.0.

Returns:singleton DescriptorDownloader instance
stem.descriptor.remote.their_server_descriptor(**query_args)[source]

Provides the server descriptor of the relay we're downloading from.

New in version 1.7.0.

Parameters:query_args -- additional arguments for the Query constructor
Returns:Query for the server descriptors
stem.descriptor.remote.get_server_descriptors(fingerprints=None, **query_args)[source]

Shorthand for get_server_descriptors() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_extrainfo_descriptors(fingerprints=None, **query_args)[source]

Shorthand for get_extrainfo_descriptors() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_microdescriptors(hashes, **query_args)[source]

Shorthand for get_microdescriptors() on our singleton instance.

New in version 1.8.0.

stem.descriptor.remote.get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)[source]

Shorthand for get_consensus() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_detached_signatures(**query_args)[source]

Shorthand for get_detached_signatures() on our singleton instance.

New in version 1.8.0.

class stem.descriptor.remote.Query(resource, descriptor_type=None, endpoints=None, compression=None, retries=2, fall_back_to_authority=False, timeout=None, start=True, block=False, validate=False, document_handler='ENTRIES', **kwargs)[source]

Bases: object

Asynchronous request for descriptor content from a directory authority or mirror. These can either be made through the DescriptorDownloader or directly for more advanced usage.

To block on the response and get results either call run() or iterate over the Query. The run() method pass along any errors that arise...

from stem.descriptor.remote import Query

query = Query(
  '/tor/server/all',
  timeout = 30,
)

print('Current relays:')

try:
  for desc in Query('/tor/server/all', 'server-descriptor 1.0').run():
    print(desc.fingerprint)
except Exception as exc:
  print('Unable to retrieve the server descriptors: %s' % exc)

... while iterating fails silently...

print('Current relays:')

for desc in Query('/tor/server/all', 'server-descriptor 1.0'):
  print(desc.fingerprint)

In either case exceptions are available via our 'error' attribute.

Tor provides quite a few different descriptor resources via its directory protocol (see section 4.2 and later of the dir-spec). Commonly useful ones include...

Resource Description
/tor/server/all all present server descriptors
/tor/server/fp/<fp1>+<fp2>+<fp3> server descriptors with the given fingerprints
/tor/extra/all all present extrainfo descriptors
/tor/extra/fp/<fp1>+<fp2>+<fp3> extrainfo descriptors with the given fingerprints
/tor/micro/d/<hash1>-<hash2> microdescriptors with the given hashes
/tor/status-vote/current/consensus present consensus
/tor/status-vote/current/consensus-microdesc present microdescriptor consensus
/tor/status-vote/next/consensus-signatures detached signature, used for making the next consenus
/tor/keys/all key certificates for the authorities
/tor/keys/fp/<v3ident1>+<v3ident2> key certificates for specific authorities

ZSTD compression requires zstandard, and LZMA requires the lzma module.

For legacy reasons if our resource has a '.z' suffix then our compression argument is overwritten with Compression.GZIP.

Changed in version 1.7.0: Added support for downloading from ORPorts.

Changed in version 1.7.0: Added the compression argument.

Changed in version 1.7.0: Added the reply_headers attribute.

The class this provides changed between Python versions. In python2 this was called httplib.HTTPMessage, whereas in python3 the class was renamed to http.client.HTTPMessage.

Changed in version 1.7.0: Endpoints are now expected to be DirPort or ORPort instances. Usage of tuples for this argument is deprecated and will be removed in the future.

Changed in version 1.7.0: Avoid downloading from tor26. This directory authority throttles its DirPort to such an extent that requests either time out or take on the order of minutes.

Changed in version 1.7.0: Avoid downloading from Bifroest. This is the bridge authority so it doesn't vote in the consensus, and apparently times out frequently.

Changed in version 1.8.0: Serge has replaced Bifroest as our bridge authority. Avoiding descriptor downloads from it instead.

Variables:
  • resource (str) -- resource being fetched, such as '/tor/server/all'
  • descriptor_type (str) -- type of descriptors being fetched (for options see parse_file()), this is guessed from the resource if None
  • endpoints (list) -- DirPort or ORPort of the authority or mirror we're querying, this uses authorities if undefined
  • compression (list) -- list of stem.descriptor.remote.Compression we're willing to accept, when none are mutually supported downloads fall back to Compression.PLAINTEXT
  • retries (int) -- number of times to attempt the request if downloading it fails
  • fall_back_to_authority (bool) -- when retrying request issues the last request to a directory authority if True
  • content (str) -- downloaded descriptor content
  • error (Exception) -- exception if a problem occured
  • is_done (bool) -- flag that indicates if our request has finished
  • start_time (float) -- unix timestamp when we first started running
  • reply_headers (http.client.HTTPMessage) -- headers provided in the response, None if we haven't yet made our request
  • runtime (float) -- time our query took, this is None if it's not yet finished
  • validate (bool) -- checks the validity of the descriptor's content if True, skips these checks otherwise
  • document_handler (stem.descriptor.__init__.DocumentHandler) -- method in which to parse a NetworkStatusDocument
  • kwargs (dict) -- additional arguments for the descriptor constructor

Following are only applicable when downloading from a DirPort...

Variables:
  • timeout (float) -- duration before we'll time out our request
  • download_url (str) -- last url used to download the descriptor, this is unset until we've actually made a download attempt
Parameters:
  • start (bool) -- start making the request when constructed (default is True)
  • block (bool) -- only return after the request has been completed, this is the same as running query.run(True) (default is False)
start()[source]

Starts downloading the scriptors if we haven't started already.

run(suppress=False)[source]

Blocks until our request is complete then provides the descriptors. If we haven't yet started our request then this does so.

Parameters:

suppress (bool) -- avoids raising exceptions if True

Returns:

list for the requested Descriptor instances

Raises :

Using the iterator can fail with the following if suppress is False...

  • ValueError if the descriptor contents is malformed
  • socket.timeout if our request timed out
  • urllib2.URLError for most request failures

Note that the urllib2 module may fail with other exception types, in which case we'll pass it along.

class stem.descriptor.remote.DescriptorDownloader(use_mirrors=False, **default_args)[source]

Bases: object

Configurable class that issues Query instances on your behalf.

Parameters:
  • use_mirrors (bool) -- downloads the present consensus and uses the directory mirrors to fetch future requests, this fails silently if the consensus cannot be downloaded
  • default_args -- default arguments for the Query constructor
use_directory_mirrors()[source]

Downloads the present consensus and configures ourselves to use directory mirrors, in addition to authorities.

Returns:NetworkStatusDocumentV3 from which we got the directory mirrors
Raises :Exception if unable to determine the directory mirrors
their_server_descriptor(**query_args)[source]

Provides the server descriptor of the relay we're downloading from.

New in version 1.7.0.

Parameters:query_args -- additional arguments for the Query constructor
Returns:Query for the server descriptors
get_server_descriptors(fingerprints=None, **query_args)[source]

Provides the server descriptors with the given fingerprints. If no fingerprints are provided then this returns all descriptors known by the relay.

Parameters:
  • fingerprints (str,list) -- fingerprint or list of fingerprints to be retrieved, gets all descriptors if None
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the server descriptors

Raises :

ValueError if we request more than 96 descriptors by their fingerprints (this is due to a limit on the url length by squid proxies).

get_extrainfo_descriptors(fingerprints=None, **query_args)[source]

Provides the extrainfo descriptors with the given fingerprints. If no fingerprints are provided then this returns all descriptors in the present consensus.

Parameters:
  • fingerprints (str,list) -- fingerprint or list of fingerprints to be retrieved, gets all descriptors if None
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the extrainfo descriptors

Raises :

ValueError if we request more than 96 descriptors by their fingerprints (this is due to a limit on the url length by squid proxies).

get_microdescriptors(hashes, **query_args)[source]

Provides the microdescriptors with the given hashes. To get these see the microdescriptor_digest attribute of RouterStatusEntryMicroV3. Note that these are only provided via the microdescriptor consensus. For exampe...

>>> import stem.descriptor.remote
>>> consensus = stem.descriptor.remote.get_consensus(microdescriptor = True).run()
>>> my_router_status_entry = list(filter(lambda desc: desc.nickname == 'caersidi', consensus))[0]
>>> print(my_router_status_entry.microdescriptor_digest)
IQI5X2A5p0WVN/MgwncqOaHF2f0HEGFEaxSON+uKRhU

>>> my_microdescriptor = stem.descriptor.remote.get_microdescriptors([my_router_status_entry.microdescriptor_digest]).run()[0]
>>> print(my_microdescriptor)
onion-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBAOJo9yyVgG8ksEHQibqPIEbLieI6rh1EACRPiDiV21YObb+9QEHaR3Cf
FNAzDbGhbvADLBB7EzuViL8w+eXQUOaIsJRdymh/wuUJ78bv5oEIJhthKq/Uqa4P
wKHXSZixwAHfy8NASTX3kxu9dAHWU3Owb+4W4lR2hYM0ZpoYYkThAgMBAAE=
-----END RSA PUBLIC KEY-----
ntor-onion-key kWOHNd+2uBlMpcIUbbpFLiq/rry66Ep6MlwmNpwzcBg=
id ed25519 xE/GeYImYAIB0RbzJXFL8kDLpDrj/ydCuCdvOgC4F/4
Parameters:
  • hashes (str,list) -- microdescriptor hash or list of hashes to be retrieved
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the microdescriptors

Raises :

ValueError if we request more than 92 microdescriptors by their hashes (this is due to a limit on the url length by squid proxies).

get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)[source]

Provides the present router status entries.

Changed in version 1.5.0: Added the microdescriptor argument.

Parameters:
  • authority_v3ident (str) -- fingerprint of the authority key for which to get the consensus, see 'v3ident' in tor's config.c for the values.
  • microdescriptor (bool) -- provides the microdescriptor consensus if True, standard consensus otherwise
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the router status entries

get_vote(authority, **query_args)[source]

Provides the present vote for a given directory authority.

Parameters:
  • authority (stem.directory.Authority) -- authority for which to retrieve a vote for
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the router status entries

get_key_certificates(authority_v3idents=None, **query_args)[source]

Provides the key certificates for authorities with the given fingerprints. If no fingerprints are provided then this returns all present key certificates.

Parameters:
  • authority_v3idents (str) --

    fingerprint or list of fingerprints of the authority keys, see 'v3ident' in tor's config.c for the values.

  • query_args -- additional arguments for the Query constructor
Returns:

Query for the key certificates

Raises :

ValueError if we request more than 96 key certificates by their identity fingerprints (this is due to a limit on the url length by squid proxies).

get_detached_signatures(**query_args)[source]

Provides the detached signatures that will be used to make the next consensus. Please note that these are only available during minutes 55-60 each hour. If requested during minutes 0-55 tor will not service these requests, and this will fail with a 404.

New in version 1.8.0.

Parameters:query_args -- additional arguments for the Query constructor
Returns:Query for the detached signatures
query(resource, **query_args)[source]

Issues a request for the given resource.

Changed in version 1.7.0: The fall_back_to_authority default when using this method is now False, like the Query class.

Parameters:
  • resource (str) -- resource being fetched, such as '/tor/server/all'
  • query_args -- additional arguments for the Query constructor
Returns:

Query for the descriptors

Raises :

ValueError if resource is clearly invalid or the descriptor type can't be determined when 'descriptor_type' is None

stem.descriptor.remote.get_authorities()[source]

Provides cached Tor directory authority information. The directory information hardcoded into Tor and occasionally changes, so the information this provides might not necessarily match your version of tor.

Deprecated since version 1.7.0: Use stem.directory.Authority.from_cache() instead.

Returns:dict of str nicknames to Authority instances