preparables - python co-routines

If you are familiar with python and its yield statement, you know that it allows you to run a function and pause it in the middle of execution. But you may not know that you are able to pass information from outside the function back into it.

The following code is valid python!

def my_func():
    foo = yield "bar"
    print foo
    yield "baz"


g = my_func()
print next(g)
g.send("foo")
print next(g)

It prints out:

foo
bar
baz

Notice that we have to start the generator using next() before we can pass any data into it.

What would be the point of passing data into a generator, though? Preparables are why! Preparables are a way of structuring your code so that you can run multiple functions side by side and fetch / send data back into them. What’s nice about this is that you can write a function that doesn’t fall into callback hell. Instead of callbacks, you have an external mediator that fetches data and passes the data back into the function. This mediator can run multiple preparable functions at once, passing data as it is received into them. This lets you coordinate and run many functions in parallel.

The following code shows how preparables can be used to coordinate multiple functions at once.

from preparable import Preparer
from preparable import debug

def f(x):
    return x*x

def x(y):
    return y*y;

def top_level():
  debug("top level")
  return "foo"

def some_work(foo):
  debug("d DOING SOME WORK", foo)

def multi_step_one(first_arg):
  debug("a step 0", first_arg)
  second_arg = yield {
    "func" : f,
    "args" : [first_arg],
    "kwargs" : {},
    "cache_key" : "one"
  }

  debug("a step 1 received", second_arg)

  third_arg = yield {
    "func" : x,
    "args" : [second_arg]
  }
  debug("a step 2 received", third_arg)

def multi_step_two():
  debug("b step 0")
  bar = yield top_level
  baz = yield {
    "func" : x,
    "args" : [10]
  }

  debug("b step 0.5")
  bax = yield (f,[5],)
  debug("b step 1")

def multi_step_three():
  debug("c step 0")

  first_arg = yield {
    "func" : some_work,
    "args" : ["foo"],
    "cache_key" : "one"
  }
  debug("c step 1")

if __name__ == "__main__":
  prep = Preparer()
  prep.add(multi_step_one, [3])
  prep.add(multi_step_two)
  prep.add(multi_step_three)

  prep.run()
  prep.print_summary()

To use preparables, we create a Preparer instance and then load it with the functions we want to run in parallel. When we call prep.run(), the preparer automatically runs all the functions that were enqueued. When a function yields, the preparer instance automatically calls the returned function and then passes the results back into the preparable when they are ready.

The preparable style of execution allows for advanced delivery methods like BigPipe.

okay