API Documentation

QueueHelper

class weasel_pipeline.QueueHelper

Helper class for exchanging messages between elements of a weasel-pipeline toolchain

async close() → None

Close the connection to RabbitMQ.

async static create(amqp_uri: str, prefetch_count: int = 5000)

Create a QueueHelper instance and establish connection to the RabbitMQ server.

Parameters

• amqp_uri – The connection URI for the RabbitMQ server

• prefetch_count – Controls the number of messages which are pre-fetched from each queue. Keeping this number high enough will significantly improve message throughput

Returns

The created QueueHelper instance

async declare_queue(name: str, queue_max_length: int = 1000) → None

Declare a queue with the given name.

Parameters

• name – The identifying name for the created queue

• queue_max_length – Defines the maximum length of this queue. This prevents the queue from growing indefinitely. When an element calls put_value to a queue which is currently full, this call will be suspended until there is free space in the queue. This way all elements in the pipeline get restricted to the speed of the bottleneck elements.

async initialize_pika(amqp_uri: str, prefetch_count: int) → None

Initialize the connection to RabbitMQ. This is only required if the instance is not created via the create method.

Parameters

• amqp_uri – The connection URI for the RabbitMQ server

• prefetch_count – Controls the number of messages which are pre-fetched from each queue. Keeping this number high enough will significantly improve message throughput

pull_values(name: str)

Pull values from the queue of the given name. This will return an iterator that is supposed to be used in an async for loop.

Parameters

name – The identifying name of the queue from which the values are read

async put_value(name: str, value: object) → None

Output a value to the queue of the given name. If the queue is currently full, this call will be suspended until there is free space again.

Parameters

• name – The identifying name of the queue to which the value is written

• value – The output value. Can be any data type supported by pickle

ConfigurationHelper

class weasel_pipeline.ConfigurationHelper

Helper class for parsing pipeline element configuration from multiple sources. Provides a thin wrapper around the configargparse library.

add_argument(keyword: str, argument_type: type = <class 'str'>, required: bool = True, help_text: Optional[str] = None) → None

Add an argument to the configuration parser.

Parameters

• keyword – The keyword for this argument, i.e. –<keyword> on the command line

• argument_type – The type of the argument value (e.g. str, int)

• required – Whether or not this argument is required

• help_text – The help text which is shown for this argument

parse() → None

Parse the passed arguments and store the result in the config attribute.

AsyncContext

class weasel_pipeline.AsyncContext

Helper class which installs uvloop to the asyncio event loop

classmethod get_loop() → asyncio.events.AbstractEventLoop

Get the event loop. This class method always returns the same event loop, realizing a singleton pattern, to avoid conflicting event loop objects. It also installs uvloop to the event loop instance to maximize performance.

classmethod run(async_fn) → None

Run async code in the uvloop-patched event loop.

Parameters

async_fn – The awaitable function/method to run