Tutorial Sheet Data Scientist

In order to understand the communication for the module, one must first understand the communication between modules.

Modules have a variadic number of inputs and a variadic number of outputs. The inputs are lettered A-Z, then AA, AB…, while outputs are numbered 1,2,3..,10,11. The implementation is hidden away within the module, significantly reducing the complexity of the platform to module developers.

Pulling

Each module input is available to the module developer with the simple pull function, which takes one argument specifying the number of data packets to take, defaulting to 1. Note that the packets are sorted by their Date. The pull function can also be extended with converter, which will take the raw input provided by the other module and immediately deserialise it.

For example, let’s assume that we have one data input source, and our messages are simple packets with the keys title-data. Then the underlying implementation could look like this:

# Module A received: {"k": "hi", "v": "test"}, {"k": "hi2", "v": "test2"}, ...

def convert(raw):
  return json.loads(raw)

three_messages = module.A.pull(3, converter=convert) 

Note that the operations are not idempotent as the messages once they are pulled will be deleted from the module. If there are two inputs, they can be accessed independently by A andB, e.g.:

messages_from_a = module.A.pull()
messages_from_b = module.B.pull()

Alternatively, on can be used to specify the input channel: module.on('A').pull(), A is syntactic sugar for on('A')

Putting

The module outputs are numbered and available to the module developer with a put function. However, due to the language limits of python, on must be used to specify the channel. The put function takes both, elements or lists. The put function expects the elements to be serialised, alternatively, a converter can be specified. In case a list of elements is provided, the converter will be run on each input independently treating the list as a list of inputs. If the list should be treated as a singular input, then the flag ignore_list should be set to true.

Let’s assume the same example as before, now with putting

def serialize(elem):
  return json.dumps(elem)

# V1
data = [{'k': 'a', 'v': 'b'}, {'k': 'a2', 'v': 'b2'}]
module.on('1').put(data, converter=serialize)

# V2
module.on('1').put({'k': 'a', 'v': 'b'}, converter=serialize)

# V3
module.on('1').put('{"k": "a", "v": "b"}')