Descriptor Remote¶

Module for remotely retrieving descriptors from directory authorities and mirrors. This is the simplest method for getting current tor descriptor information...

import stem.descriptor.remote

for desc in stem.descriptor.remote.get_server_descriptors():
  if desc.exit_policy.is_exiting_allowed():
    print('  %s (%s)' % (desc.nickname, desc.fingerprint))

More custom downloading behavior can be done through the DescriptorDownloader class, which issues Query instances to get you descriptor content. For example...

from stem.descriptor.remote import DescriptorDownloader

downloader = DescriptorDownloader(
  use_mirrors = True,
  timeout = 10,
)

query = downloader.get_server_descriptors()

print('Exit Relays:')

try:
  for desc in query.run():
    if desc.exit_policy.is_exiting_allowed():
      print('  %s (%s)' % (desc.nickname, desc.fingerprint))

  print
  print('Query took %0.2f seconds' % query.runtime)
except Exception as exc:
  print('Unable to retrieve the server descriptors: %s' % exc)

get_instance - Provides a singleton DescriptorDownloader used for...
  |- their_server_descriptor - provides the server descriptor of the relay we download from
  |- get_server_descriptors - provides present server descriptors
  |- get_extrainfo_descriptors - provides present extrainfo descriptors
  |- get_microdescriptors - provides present microdescriptors with the given digests
  |- get_consensus - provides the present consensus or router status entries
  |- get_bandwidth_file - provides bandwidth heuristics used to make the next consensus
  +- get_detached_signatures - authority signatures used to make the next consensus

Query - Asynchronous request to download tor descriptors
  |- start - issues the query if it isn't already running
  +- run - blocks until the request is finished and provides the results

DescriptorDownloader - Configurable class for issuing queries
  |- use_directory_mirrors - use directory mirrors to download future descriptors
  |- their_server_descriptor - provides the server descriptor of the relay we download from
  |- get_server_descriptors - provides present server descriptors
  |- get_extrainfo_descriptors - provides present extrainfo descriptors
  |- get_microdescriptors - provides present microdescriptors with the given digests
  |- get_consensus - provides the present consensus or router status entries
  |- get_vote - provides an authority's vote for the next consensus
  |- get_key_certificates - provides present authority key certificates
  |- get_bandwidth_file - provides bandwidth heuristics used to make the next consensus
  |- get_detached_signatures - authority signatures used to make the next consensus
  +- query - request an arbitrary descriptor resource

New in version 1.1.0.

stem.descriptor.remote.MAX_FINGERPRINTS¶

Maximum number of descriptors that can requested at a time by their fingerprints.

stem.descriptor.remote.MAX_MICRODESCRIPTOR_HASHES¶

Maximum number of microdescriptors that can requested at a time by their hashes.

stem.descriptor.remote.Compression(enum)¶

Compression when downloading descriptors.

New in version 1.7.0.

Compression	Description
PLAINTEXT	Uncompressed data.
GZIP	GZip compression.
ZSTD	Zstandard compression, this requires the zstandard module.
LZMA	LZMA compression, this requires the 'lzma module <https://docs.python.org/3/library/lzma.html>`_.

stem.descriptor.remote.get_instance()[source]¶

Provides the singleton DescriptorDownloader used for this module's shorthand functions.

New in version 1.5.0.

Returns:	singleton DescriptorDownloader instance

stem.descriptor.remote.their_server_descriptor(**query_args)[source]¶

Provides the server descriptor of the relay we're downloading from.

New in version 1.7.0.

Parameters:	query_args -- additional arguments for the Query constructor
Returns:	Query for the server descriptors

stem.descriptor.remote.get_server_descriptors(fingerprints=None, **query_args)[source]¶

Shorthand for get_server_descriptors() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_extrainfo_descriptors(fingerprints=None, **query_args)[source]¶

Shorthand for get_extrainfo_descriptors() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_microdescriptors(hashes, **query_args)[source]¶

Shorthand for get_microdescriptors() on our singleton instance.

New in version 1.8.0.

stem.descriptor.remote.get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)[source]¶

Shorthand for get_consensus() on our singleton instance.

New in version 1.5.0.

stem.descriptor.remote.get_bandwidth_file(**query_args)[source]¶

Shorthand for get_bandwidth_file() on our singleton instance.

New in version 1.8.0.

stem.descriptor.remote.get_detached_signatures(**query_args)[source]¶

Shorthand for get_detached_signatures() on our singleton instance.

New in version 1.8.0.

class stem.descriptor.remote.Query(resource, descriptor_type=None, endpoints=None, compression=('gzip', ), retries=2, fall_back_to_authority=False, timeout=None, start=True, block=False, validate=False, document_handler='ENTRIES', **kwargs)[source]¶

Bases: object

Asynchronous request for descriptor content from a directory authority or mirror. These can either be made through the DescriptorDownloader or directly for more advanced usage.

To block on the response and get results either call run() or iterate over the Query. The run() method pass along any errors that arise...

from stem.descriptor.remote import Query

query = Query(
  '/tor/server/all',
  timeout = 30,
)

print('Current relays:')

try:
  for desc in Query('/tor/server/all', 'server-descriptor 1.0').run():
    print(desc.fingerprint)
except Exception as exc:
  print('Unable to retrieve the server descriptors: %s' % exc)

... while iterating fails silently...

print('Current relays:')

for desc in Query('/tor/server/all', 'server-descriptor 1.0'):
  print(desc.fingerprint)

In either case exceptions are available via our 'error' attribute.

Tor provides quite a few different descriptor resources via its directory protocol (see section 4.2 and later of the dir-spec). Commonly useful ones include...

Resource	Description
/tor/server/all	all present server descriptors
/tor/server/fp/<fp1>+<fp2>+<fp3>	server descriptors with the given fingerprints
/tor/extra/all	all present extrainfo descriptors
/tor/extra/fp/<fp1>+<fp2>+<fp3>	extrainfo descriptors with the given fingerprints
/tor/micro/d/<hash1>-<hash2>	microdescriptors with the given hashes
/tor/status-vote/current/consensus	present consensus
/tor/status-vote/current/consensus-microdesc	present microdescriptor consensus
/tor/status-vote/next/bandwidth	bandwidth authority heuristics for the next consenus
/tor/status-vote/next/consensus-signatures	detached signature, used for making the next consenus
/tor/keys/all	key certificates for the authorities
/tor/keys/fp/<v3ident1>+<v3ident2>	key certificates for specific authorities

ZSTD compression requires zstandard, and LZMA requires the lzma module.

For legacy reasons if our resource has a '.z' suffix then our compression argument is overwritten with Compression.GZIP.

Changed in version 1.7.0: Added support for downloading from ORPorts.

Changed in version 1.7.0: Added the compression argument.

Changed in version 1.7.0: Added the reply_headers attribute.

The class this provides changed between Python versions. In python2 this was called httplib.HTTPMessage, whereas in python3 the class was renamed to http.client.HTTPMessage.

Changed in version 1.7.0: Endpoints are now expected to be DirPort or ORPort instances. Usage of tuples for this argument is deprecated and will be removed in the future.

Changed in version 1.7.0: Avoid downloading from tor26. This directory authority throttles its DirPort to such an extent that requests either time out or take on the order of minutes.

Changed in version 1.7.0: Avoid downloading from Bifroest. This is the bridge authority so it doesn't vote in the consensus, and apparently times out frequently.

Changed in version 1.8.0: Serge has replaced Bifroest as our bridge authority. Avoiding descriptor downloads from it instead.

Changed in version 1.8.0: Defaulting to gzip compression rather than plaintext downloads.

Changed in version 1.8.0: Using Compression for our compression argument, usage of strings or this module's Compression enum is deprecated and will be removed in stem 2.x.

Variables:

resource (str) -- resource being fetched, such as '/tor/server/all' descriptor_type (str) -- type of descriptors being fetched (for options see parse_file()), this is guessed from the resource if None endpoints (list) -- DirPort or ORPort of the authority or mirror we're querying, this uses authorities if undefined compression (list) -- list of stem.descriptor.Compression we're willing to accept, when none are mutually supported downloads fall back to Compression.PLAINTEXT retries (int) -- number of times to attempt the request if downloading it fails fall_back_to_authority (bool) -- when retrying request issues the last request to a directory authority if True content (str) -- downloaded descriptor content error (Exception) -- exception if a problem occured is_done (bool) -- flag that indicates if our request has finished start_time (float) -- unix timestamp when we first started running reply_headers (http.client.HTTPMessage) -- headers provided in the response, None if we haven't yet made our request runtime (float) -- time our query took, this is None if it's not yet finished validate (bool) -- checks the validity of the descriptor's content if True, skips these checks otherwise document_handler (stem.descriptor.__init__.DocumentHandler) -- method in which to parse a NetworkStatusDocument kwargs (dict) -- additional arguments for the descriptor constructor

Following are only applicable when downloading from a DirPort...

Variables:	timeout (float) -- duration before we'll time out our request download_url (str) -- last url used to download the descriptor, this is unset until we've actually made a download attempt
Parameters:	start (bool) -- start making the request when constructed (default is True) block (bool) -- only return after the request has been completed, this is the same as running query.run(True) (default is False)

start()[source]¶

Starts downloading the scriptors if we haven't started already.

run(suppress=False)[source]¶

Blocks until our request is complete then provides the descriptors. If we haven't yet started our request then this does so.

Parameters:	suppress (bool) -- avoids raising exceptions if True
Returns:	list for the requested Descriptor instances
Raises :	Using the iterator can fail with the following if suppress is False... ValueError if the descriptor contents is malformed DownloadTimeout if our request timed out DownloadFailed if our request fails

class stem.descriptor.remote.DescriptorDownloader(use_mirrors=False, **default_args)[source]¶

Bases: object

Configurable class that issues Query instances on your behalf.

Parameters:	use_mirrors (bool) -- downloads the present consensus and uses the directory mirrors to fetch future requests, this fails silently if the consensus cannot be downloaded default_args -- default arguments for the Query constructor

use_directory_mirrors()[source]¶

Downloads the present consensus and configures ourselves to use directory mirrors, in addition to authorities.

Returns:	NetworkStatusDocumentV3 from which we got the directory mirrors
Raises :	Exception if unable to determine the directory mirrors

their_server_descriptor(**query_args)[source]¶

Provides the server descriptor of the relay we're downloading from.

New in version 1.7.0.

Parameters:	query_args -- additional arguments for the Query constructor
Returns:	Query for the server descriptors

get_server_descriptors(fingerprints=None, **query_args)[source]¶

Provides the server descriptors with the given fingerprints. If no fingerprints are provided then this returns all descriptors known by the relay.

Parameters:	fingerprints (str,list) -- fingerprint or list of fingerprints to be retrieved, gets all descriptors if None query_args -- additional arguments for the Query constructor
Returns:	Query for the server descriptors
Raises :	ValueError if we request more than 96 descriptors by their fingerprints (this is due to a limit on the url length by squid proxies).

get_extrainfo_descriptors(fingerprints=None, **query_args)[source]¶

Provides the extrainfo descriptors with the given fingerprints. If no fingerprints are provided then this returns all descriptors in the present consensus.

Parameters:	fingerprints (str,list) -- fingerprint or list of fingerprints to be retrieved, gets all descriptors if None query_args -- additional arguments for the Query constructor
Returns:	Query for the extrainfo descriptors
Raises :	ValueError if we request more than 96 descriptors by their fingerprints (this is due to a limit on the url length by squid proxies).

get_microdescriptors(hashes, **query_args)[source]¶

Provides the microdescriptors with the given hashes. To get these see the microdescriptor_digest attribute of RouterStatusEntryMicroV3. Note that these are only provided via the microdescriptor consensus. For exampe...

>>> import stem.descriptor.remote
>>> consensus = stem.descriptor.remote.get_consensus(microdescriptor = True).run()
>>> my_router_status_entry = list(filter(lambda desc: desc.nickname == 'caersidi', consensus))[0]
>>> print(my_router_status_entry.microdescriptor_digest)
IQI5X2A5p0WVN/MgwncqOaHF2f0HEGFEaxSON+uKRhU

>>> my_microdescriptor = stem.descriptor.remote.get_microdescriptors([my_router_status_entry.microdescriptor_digest]).run()[0]
>>> print(my_microdescriptor)
onion-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBAOJo9yyVgG8ksEHQibqPIEbLieI6rh1EACRPiDiV21YObb+9QEHaR3Cf
FNAzDbGhbvADLBB7EzuViL8w+eXQUOaIsJRdymh/wuUJ78bv5oEIJhthKq/Uqa4P
wKHXSZixwAHfy8NASTX3kxu9dAHWU3Owb+4W4lR2hYM0ZpoYYkThAgMBAAE=
-----END RSA PUBLIC KEY-----
ntor-onion-key kWOHNd+2uBlMpcIUbbpFLiq/rry66Ep6MlwmNpwzcBg=
id ed25519 xE/GeYImYAIB0RbzJXFL8kDLpDrj/ydCuCdvOgC4F/4

Parameters:	hashes (str,list) -- microdescriptor hash or list of hashes to be retrieved query_args -- additional arguments for the Query constructor
Returns:	Query for the microdescriptors
Raises :	ValueError if we request more than 92 microdescriptors by their hashes (this is due to a limit on the url length by squid proxies).

get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)[source]¶

Provides the present router status entries.

Changed in version 1.5.0: Added the microdescriptor argument.

Parameters:	authority_v3ident (str) -- fingerprint of the authority key for which to get the consensus, see 'v3ident' in tor's config.c for the values. microdescriptor (bool) -- provides the microdescriptor consensus if True, standard consensus otherwise query_args -- additional arguments for the Query constructor
Returns:	Query for the router status entries

get_vote(authority, **query_args)[source]¶

Provides the present vote for a given directory authority.

Parameters:	authority (stem.directory.Authority) -- authority for which to retrieve a vote for query_args -- additional arguments for the Query constructor
Returns:	Query for the router status entries

get_key_certificates(authority_v3idents=None, **query_args)[source]¶

Provides the key certificates for authorities with the given fingerprints. If no fingerprints are provided then this returns all present key certificates.

Parameters:	authority_v3idents (str) -- fingerprint or list of fingerprints of the authority keys, see 'v3ident' in tor's config.c for the values. query_args -- additional arguments for the Query constructor
Returns:	Query for the key certificates
Raises :	ValueError if we request more than 96 key certificates by their identity fingerprints (this is due to a limit on the url length by squid proxies).

get_bandwidth_file(**query_args)[source]¶

Provides the bandwidth authority heuristics used to make the next consensus.

New in version 1.8.0.

Parameters:	query_args -- additional arguments for the Query constructor
Returns:	Query for the bandwidth authority heuristics

get_detached_signatures(**query_args)[source]¶

Provides the detached signatures that will be used to make the next consensus. Please note that these are only available during minutes 55-60 each hour. If requested during minutes 0-55 tor will not service these requests, and this will fail with a 404.

For example...

import stem.descriptor.remote

detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]

for i, sig in enumerate(detached_sigs.signatures):
  print('Signature %i is from %s' % (i + 1, sig.identity))

When available (minutes 55-60 of the hour)

% python demo.py
Signature 1 is from 0232AF901C31A04EE9848595AF9BB7620D4C5B2E
Signature 2 is from 14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4
Signature 3 is from 23D15D965BC35114467363C165C4F724B64B4F66
...

When unavailable (minutes 0-55 of the hour)

% python demo.py
Traceback (most recent call last):
  File "demo.py", line 3, in
    detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
  File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 533, in run
    return list(self._run(suppress))
  File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 544, in _run
    raise self.error
stem.DownloadFailed: Failed to download from http://154.35.175.225:80/tor/status-vote/next/consensus-signatures (HTTPError): Not found

New in version 1.8.0.

Parameters:	query_args -- additional arguments for the Query constructor
Returns:	Query for the detached signatures

query(resource, **query_args)[source]¶

Issues a request for the given resource.

Changed in version 1.7.0: The fall_back_to_authority default when using this method is now False, like the Query class.

Parameters:	resource (str) -- resource being fetched, such as '/tor/server/all' query_args -- additional arguments for the Query constructor
Returns:	Query for the descriptors
Raises :	ValueError if resource is clearly invalid or the descriptor type can't be determined when 'descriptor_type' is None

stem.descriptor.remote.get_authorities()[source]¶

Provides cached Tor directory authority information. The directory information hardcoded into Tor and occasionally changes, so the information this provides might not necessarily match your version of tor.

Deprecated since version 1.7.0: Use stem.directory.Authority.from_cache() instead.

Returns:	dict of str nicknames to Authority instances