Marcos Dione: callable-choices-for-django-rest-framework

At work I'm writing an API using Django/DRF. Suddenly I had to write an application (just a few pages for calling a few endpoints), so I (ab)used DRF's Serializers to build them. One of the problems I faced while doing this was that DRF's ChoiceField accepts only a sequence with the values for the dropdown, unlike Django's, who also accepts callables. This means that once you gave it a set of values, it never ever changes, at least until you restart the application.

Unless, of course, you cheat. Or hack. Aren't those synonyms?

class UpdatedSequence:
    def __init__(self, update_func):
        self.update_func = update_func
        self.restart = True

        self.data = None
        self.index = 0


    def __iter__(self):
        # we're our own iterator
        return self


    def __next__(self):
        # if we're iterating from the beginning, call the function
        # and cache the result
        if self.restart:
            self.data = self.update_func()
            self.index = 0

        try:
            datum = self.data[self.index]
        except IndexError:
            # we reached the limit, start all over
            self.restart = True
            raise StopIteration
        else:
            self.index += 1
            self.restart = False

        return datum

This simple class tracks when you start iterating over it and calls the function you pass to obtain the data. Then it iterates over the result. When you reach the end, it marks it to start all over, so the next time you iterate over it, it will call the function again. The function you pass can be the all() method of a QuerySet or anything else that goes fetch data and returns an iterable.

In my case in particular, I also added a TimedCache so I don't read twice the db to fill two dropdown with the same info in the same form:

class TimedCache:
    '''A function wrapper that caches the result for a while.'''
    def __init__(self, f, timeout):
        self.f = f
        self.timeout = timeout
        self.last_executed = None
        self.cache = None
        self.__name__ = f.__name__ + ' (Cached %ds)' % timeout


    def __call__(self):
        now = time.monotonic()

        if self.cache is None or (now - self.last_executed) > self.timeout:
            self.cache = self.f()
            self.last_executed = now

        return self.cache

I hope this helps someone.


python django drf

Mariano Guerra: Playing with Lasp in a 3 Node Cluster

After playing with Lasp in a single node in the previous post: Playing with Lasp and CRDTs

I created a rebar3 template to easily setup a 3 node cluster to play with Lasp where it shines, in a distributed environment.

Install rebar3 if you haven't already.

then install this template:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_lasp.git ~/.config/rebar3/templates/rebar3_template_lasp

Use

rebar3 new rebar3_template_lasp name=laspy
cd laspy

    # build the 3 node cluster
make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

On one of the nodes' shell run:

Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

On another one run:

{ok, AwMapRes} = lasp:query({<<"awmap">>,{state_awmap,[state_mvregister]}}).

AwMapRes.

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).

You should get:

[#{what => i_am_an_awmap_value}]

All the examples from the previous post should run.

A termcast (?) of the process:

Happy Lasping!

Mariano Guerra: Playing with Lasp and CRDTs

For a long time I was looking for some time to play with Lasp:

Lasp is a suite of libraries aimed at providing a comprehensive programming
system for planetary scale Elixir and Erlang applications.

Yesterday I had some time and played a little bit with it, and thanks to the help from @cmeik and @vitorenesduarte I got some code running and understood some things.

Here is what I achieved until now, note that I'm learning this stuff so I may say many things that are wrong, I will try to get people to review it and update it with the corrections.

Some initial snippets of code we will reuse:

% clear all variable bindings
f().
Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

A map with any type as value

If we want to have data which can be represented as a dict/map/key-value mapping, there are multiple alternatives we can use in lasp, here we will explore some.

First we are going to use an AwMap CRDT as a map which is defined as:

A dictionary where keys can be anything and the values are causal-CRDTs.

Since values must be casual-CRDTs we can't use a bare erlang type as value nor a non causal-CRDT type like lwwregister, which we will use later with another map type.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

We can also use a GMap CRDT (Grow Only Map, keys can't be removed):

A dictionary where keys can be anything and the values are join-semilattices.

In this case we can use non causal-CRDTs as values, to make it a little different we are going to use another register type, a lwwregister

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal = #{what => i_am_a_gmap_value}.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).
{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                            {set, Timestamp(), GMapVal}},
                                            self()).

{ok, GMapRes} = lasp:query(GMap1).

GMapRes.
% [{<<"key1">>,#{thing => 42,what => i_am_a_gmap_value}}]

[{_, GMapResVal}] = GMapRes.

GMapResVal.
% #{what => i_am_a_gmap_value}

We can use a gmap with an ivar as value type, an ivar is a register that can only be set once.

GMapIVarType = {state_gmap, [state_ivar]}.
GMapIVarVarName = <<"gmapivar">>.
GMapIVarVal1 = #{what => i_am_a_gmap_ivar_value}.
GMapIVarVal2 = #{what => i_am_a_gmap_ivar_update}.

{ok, {GMapIVar, _, _, _}} = lasp:declare({GMapIVarVarName, GMapIVarType}, GMapIVarType).
{ok, {GMapIVar1, _, _, _}} = lasp:update(GMapIVar, {apply, Key1,
                                            {set, GMapIVarVal1}},
                                            self()).
% try updating it, will throw an error (the value of GMapIVar1 will be lost)
{ok, {GMapIVar2, _, _, _}} = lasp:update(GMapIVar1, {apply, Key1,
                                            {set, GMapIVarVal2}},
                                            self()).

{ok, GMapIVarRes} = lasp:query(GMapIVar1).

GMapIVarRes.

[{_, GMapIVarResVal}] = GMapIVarRes.

GMapIVarResVal.
% #{what => i_am_a_gmap_ivar_value}

Types

My examples are all maps with some value type because that's the use case I'm most interested, here's a list of types and their operations:

"Scalars"

boolean

Operations:

true
Can be set to true once

Example:

BoolType = state_boolean.
BoolVarName = <<"boolvar">>.

{ok, {Bool, _, _, _}} = lasp:declare({BoolVarName, BoolType}, BoolType).
{ok, {Bool1, _, _, _}} = lasp:update(Bool, true, self()).

{ok, BoolRes} = lasp:query(Bool1).

BoolRes.
% true

ewflag:

Enable-Wins Flag CRDT

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

EWType = state_ewflag.
EWVarName = <<"ewvar">>.

{ok, {EW, _, _, _}} = lasp:declare({EWVarName, EWType}, EWType).
{ok, {EW1, _, _, _}} = lasp:update(EW, enable, self()).
{ok, EWRes1} = lasp:query(EW1).
{ok, {EW2, _, _, _}} = lasp:update(EW1, disable, self()).
{ok, EWRes2} = lasp:query(EW2).
{ok, {EW3, _, _, _}} = lasp:update(EW2, enable, self()).
{ok, EWRes3} = lasp:query(EW3).

EWRes1.
% true
EWRes2.
% false
EWRes3.
% true

dwflag:

Disable-Wins Flag CRDT

Follows the same strategy used in Enable-Wins Flag but,
instead of creating a new dot when enabling the flag,
we create a new dot when disabling it.

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

DWType = state_dwflag.
DWVarName = <<"dwvar">>.

{ok, {DW, _, _, _}} = lasp:declare({DWVarName, DWType}, DWType).
{ok, {DW1, _, _, _}} = lasp:update(DW, enable, self()).
{ok, DWRes1} = lasp:query(DW1).
{ok, {DW2, _, _, _}} = lasp:update(DW1, disable, self()).
{ok, DWRes2} = lasp:query(DW2).
{ok, {DW3, _, _, _}} = lasp:update(DW2, enable, self()).
{ok, DWRes3} = lasp:query(DW3).

DWRes1.
% true
DWRes2.
% false
DWRes3.
% true

GCounter CRDT: grow only counter

Operations:

increment
Increment the counter by 1

Example:

GCountType = state_gcounter.
GCountVarName = <<"gcountvar">>.

{ok, {GCount, _, _, _}} = lasp:declare({GCountVarName, GCountType}, GCountType).
{ok, {GCount1, _, _, _}} = lasp:update(GCount, increment, self()).
{ok, GCountRes1} = lasp:query(GCount1).
{ok, {GCount2, _, _, _}} = lasp:update(GCount1, increment, self()).
{ok, GCountRes2} = lasp:query(GCount2).
{ok, {GCount3, _, _, _}} = lasp:update(GCount2, increment, self()).
{ok, GCountRes3} = lasp:query(GCount3).

GCountRes1.
% 1
GCountRes2.
% 2
GCountRes3.
% 3

Some more I don't get completely what they could be used for:

Bounded Counter CRDT:

Modeled as a pair where the first component is a PNCounter and the second
component is a GMap.

This counter has sub counter for different ids (actors), each of which can't go below 0, with this you can model things like seats or some resource where you allocate counts to different parties (actors) and each can decrement their own count but not others, also each counter can't go below 0, if a given actor needs to decrement it has to move counts from other actor.

Operations:

{move, term()}
Moves permissions to decrement to another replica (if it has enough permissions)
increment
Increment counter, can always happen
decrement
Decrement counter, can happen when the replica has enough local increments, or has permissions received from other replicas

Example:

BCountType = state_bcounter.
BCountVarName = <<"bcountvar">>.
Actor1 = self().
Actor2 = <<"actor2-id">>.

{ok, {BCount, _, _, _}} = lasp:declare({BCountVarName, BCountType}, BCountType).
{ok, {BCount1, _, _, _}} = lasp:update(BCount, increment, Actor1).
{ok, BCountRes1} = lasp:query(BCount1).
{ok, {BCount2, _, _, _}} = lasp:update(BCount1, increment, Actor2).
{ok, BCountRes2} = lasp:query(BCount2).
{ok, {BCount3, _, _, _}} = lasp:update(BCount2, decrement, Actor1).
{ok, BCountRes3} = lasp:query(BCount3).

% here Actor1 has counter set to 0, can't go below 0, if it want's to
% decrement it has to move a 1 from another actor, Actor2 has 1 in its
% counter so we will move it and then decrement

BCountMoveFrom = Actor2.
BCountMoveTo = Actor1.
{ok, {BCount4, _, _, _}} = lasp:update(BCount3, {move, 1, BCountMoveTo}, BCountMoveFrom).

% now we can decrement from Actor1
{ok, {BCount5, _, _, _}} = lasp:update(BCount4, decrement, Actor1).
{ok, BCountRes5} = lasp:query(BCount5).

BCountRes1.
% 1
BCountRes2.
% 2
BCountRes3.
% 1

BCountRes5.
% 0

Max Int CRDT

Operations:

  • increment

Example:

MaxIntType = state_max_int.
MaxIntVarName = <<"maxintvar">>.

{ok, {MaxInt, _, _, _}} = lasp:declare({MaxIntVarName, MaxIntType}, MaxIntType).
{ok, {MaxInt1, _, _, _}} = lasp:update(MaxInt, increment, self()).
{ok, MaxIntRes1} = lasp:query(MaxInt1).
{ok, {MaxInt2, _, _, _}} = lasp:update(MaxInt1, increment, self()).
{ok, MaxIntRes2} = lasp:query(MaxInt2).
{ok, {MaxInt3, _, _, _}} = lasp:update(MaxInt2, increment, self()).
{ok, MaxIntRes3} = lasp:query(MaxInt3).

MaxIntRes1.
% 1
MaxIntRes2.
% 2
MaxIntRes3.
% 3

Lexicographic Counter

Operations:

  • increment
  • decrement

Example:

LCountType = state_lexcounter.
LCountVarName = <<"lexcountvar">>.

{ok, {LCount, _, _, _}} = lasp:declare({LCountVarName, LCountType}, LCountType).
{ok, {LCount1, _, _, _}} = lasp:update(LCount, increment, self()).
{ok, LCountRes1} = lasp:query(LCount1).
{ok, {LCount2, _, _, _}} = lasp:update(LCount1, increment, self()).
{ok, LCountRes2} = lasp:query(LCount2).
{ok, {LCount3, _, _, _}} = lasp:update(LCount2, decrement, self()).
{ok, LCountRes3} = lasp:query(LCount3).

LCountRes1.
% 1
LCountRes2.
% 2
LCountRes3.
% 1

PNCounter CRDT:

Counter that allows both increments and decrements.

Modeled as a dictionary where keys are replicas ids and values are pairs
where the first component is the number of increments and the second component
is the number of decrements.

An actor may only update its own entry in the dictionary.

The value of the counter is the sum of all first components minus the sum
of all second components.

Operations:

  • increment
  • decrement

Example:

PNCountType = state_pncounter.
PNCountVarName = <<"pncountvar">>.

{ok, {PNCount, _, _, _}} = lasp:declare({PNCountVarName, PNCountType}, PNCountType).
{ok, {PNCount1, _, _, _}} = lasp:update(PNCount, increment, self()).
{ok, PNCountRes1} = lasp:query(PNCount1).
{ok, {PNCount2, _, _, _}} = lasp:update(PNCount1, increment, self()).
{ok, PNCountRes2} = lasp:query(PNCount2).
{ok, {PNCount3, _, _, _}} = lasp:update(PNCount2, decrement, self()).
{ok, PNCountRes3} = lasp:query(PNCount3).

PNCountRes1.
% 1
PNCountRes2.
% 2
PNCountRes3.
% 1

"Registers"

LWWRegister:

We assume timestamp are unique, totally ordered and consistent
with causal order. We use integers as timestamps.
When using this, make sure you provide globally unique timestamps.

Operations:

{set, Timestamp, Value}
Set register to Value

Example:

LWWRegType = state_lwwregister.
LWWRegVarName = <<"lwwregister">>.

{ok, {LWWReg, _, _, _}} = lasp:declare({LWWRegVarName, LWWRegType}, LWWRegType).
{ok, {LWWReg1, _, _, _}} = lasp:update(LWWReg, {set, Timestamp(), foo}, self()).
{ok, LWWRegRes1} = lasp:query(LWWReg1).
{ok, {LWWReg2, _, _, _}} = lasp:update(LWWReg1, {set, Timestamp(), bar}, self()).
{ok, LWWRegRes2} = lasp:query(LWWReg2).
{ok, {LWWReg3, _, _, _}} = lasp:update(LWWReg2, {set, Timestamp(), baz}, self()).
{ok, LWWRegRes3} = lasp:query(LWWReg3).

LWWRegRes1.
% foo
LWWRegRes2.
% bar
LWWRegRes3.
% baz

IVar:

Single-assignment variable.
Write once register.

Operations:

{set, Value}
Set register to Value (only possible once)
IVarRegType = state_ivar.
IVarRegVarName = <<"ivar">>.

{ok, {IVarReg, _, _, _}} = lasp:declare({IVarRegVarName, IVarRegType}, IVarRegType).
{ok, {IVarReg1, _, _, _}} = lasp:update(IVarReg, {set, foo}, self()).
{ok, IVarRegRes1} = lasp:query(IVarReg1).

IVarRegRes1.
% foo

MVRegister:

Multi-Value Register CRDT.

Operations:

{set, Timestamp, Value}
the Timestamp will not be used. in order to have an unified API for all registers (since LWWRegister needs to receive a timestamp), the timestamp is also supplied here

Example:

MVRegType = state_mvregister.
MVRegVarName = <<"mvregister">>.

{ok, {MVReg, _, _, _}} = lasp:declare({MVRegVarName, MVRegType}, MVRegType).
{ok, {MVReg1, _, _, _}} = lasp:update(MVReg, {set, Timestamp(), foo}, self()).
{ok, MVRegRes1} = lasp:query(MVReg1).
{ok, {MVReg2, _, _, _}} = lasp:update(MVReg1, {set, nil, bar}, self()).
{ok, MVRegRes2} = lasp:query(MVReg2).
{ok, {MVReg3, _, _, _}} = lasp:update(MVReg2, {set, Timestamp(), baz}, self()).
{ok, MVRegRes3} = lasp:query(MVReg3).

sets:to_list(MVRegRes1).
% [foo]
sets:to_list(MVRegRes2).
% [bar]
sets:to_list(MVRegRes3).
% [baz]

Collections

AWMap CRDT:

Modeled as a dictionary where keys can be anything and the values are
causal-CRDTs.

Operations:

{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).
{rmv, Key}
Remove Key

Example:

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal1 = foo.
AwMapVal2 = bar.

{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, {AwMap2, _, _, _}} = lasp:update(AwMap1, {apply, Key1,
                                              {set, Timestamp(), AwMapVal2}},
                                      self()).

{ok, {AwMap3, _, _, _}} = lasp:update(AwMap2, {apply, Key2,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, AwMapRes3} = lasp:query(AwMap3).

{ok, {AwMap4, _, _, _}} = lasp:update(AwMap3, {rmv, Key2}, self()).

{ok, AwMapRes4} = lasp:query(AwMap4).

% before removing Key2
[{K, sets:to_list(V)} || {K, V} <- AwMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

[{K, sets:to_list(V)} || {K, V} <- AwMapRes4].
% [{<<"key1">>,[bar]}]

GMap CRDT:

Grow only map.
Modeled as a dictionary where keys can be anything and the
values are join-semilattices.
{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).

Example:

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal1 = foo.
GMapVal2 = bar.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).

{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, {GMap2, _, _, _}} = lasp:update(GMap1, {apply, Key1,
                                              {set, Timestamp(), GMapVal2}},
                                      self()).

{ok, {GMap3, _, _, _}} = lasp:update(GMap2, {apply, Key2,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, GMapRes3} = lasp:query(GMap3).
% {ok,[{<<"key1">>,bar},{<<"key2">>,foo}]}

MVMap:

Multi-Value Map CRDT.
MVMap = AWMap<MVRegister<V>>

Operations:

{set, Key, Value}
Set Key to Value

Example:

MVMapType = state_mvmap.
MVMapVarName = <<"mvmap">>.
MVMapVal1 = foo.
MVMapVal2 = bar.

{ok, {MVMap, _, _, _}} = lasp:declare({MVMapVarName, MVMapType}, MVMapType).

{ok, {MVMap1, _, _, _}} = lasp:update(MVMap, {set, Key1, MVMapVal1}, self()).
{ok, {MVMap2, _, _, _}} = lasp:update(MVMap1, {set, Key1, MVMapVal2}, self()).
{ok, {MVMap3, _, _, _}} = lasp:update(MVMap2, {set, Key2, MVMapVal1}, self()).

{ok, MVMapRes3} = lasp:query(MVMap3).

[{K, sets:to_list(V)} || {K, V} <- MVMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

Pair

Operations:

{fst, Value}
Mutates the first item in the pair
{snd, Value}
Mutates the second item in the pair

Example:

PairLeftType = state_lwwregister.
PairRightType = state_gcounter.
PairType = {state_pair, [PairLeftType, PairRightType]}.
PairVarName = <<"pair">>.
PairVal1 = foo.
PairVal2 = bar.

{ok, {Pair, _, _, _}} = lasp:declare({PairVarName, PairType}, PairType).

{ok, {Pair1, _, _, _}} = lasp:update(Pair, {fst, {set, Timestamp(), PairVal1}}, self()).
{ok, {Pair2, _, _, _}} = lasp:update(Pair1, {snd, increment}, self()).
{ok, {Pair3, _, _, _}} = lasp:update(Pair2, {fst, {set, Timestamp(), PairVal2}}, self()).
{ok, {Pair4, _, _, _}} = lasp:update(Pair3, {snd, increment}, self()).

{ok, PairRes4} = lasp:query(Pair4).

PairRes4.
% {bar,2}

ORSet CRDT:

Observed-Remove Set with tombstones
{add, Element}
Add Element to the Set
{add_by_token, token(), element()}
Add Element to the Set by Token (orset internally generates a unique token for the element, therefore, it's not deterministic, this allows it to be explicit, and therefore deterministic)
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

ORSetType = state_orset.
ORSetVarName = <<"orset">>.
ORSetVal1 = foo.
ORSetVal2 = bar.
ORSetVal3 = baz.
ORSetAllVals = [ORSetVal1, ORSetVal2, ORSetVal3].

{ok, {ORSet, _, _, _}} = lasp:declare({ORSetVarName, ORSetType}, ORSetType).

{ok, {ORSet1, _, _, _}} = lasp:update(ORSet, {add, ORSetVal1}, self()).
{ok, {ORSet2, _, _, _}} = lasp:update(ORSet1, {add, ORSetVal2}, self()).
% repeat value
{ok, {ORSet3, _, _, _}} = lasp:update(ORSet2, {add, ORSetVal1}, self()).
{ok, {ORSet4, _, _, _}} = lasp:update(ORSet3, {add, ORSetVal3}, self()).

{ok, ORSetRes4} = lasp:query(ORSet4).
sets:to_list(ORSetRes4).
% [bar,baz,foo]

{ok, {ORSet5, _, _, _}} = lasp:update(ORSet4, {rmv, ORSetVal3}, self()).
{ok, ORSetRes5} = lasp:query(ORSet5).
sets:to_list(ORSetRes5).
% [bar,foo]

{ok, {ORSet6, _, _, _}} = lasp:update(ORSet5, {rmv_all, ORSetAllVals}, self()).
{ok, ORSetRes6} = lasp:query(ORSet6).
sets:to_list(ORSetRes6).
% []

{ok, {ORSet7, _, _, _}} = lasp:update(ORSet6, {add_all, ORSetAllVals}, self()).
{ok, ORSetRes7} = lasp:query(ORSet7).
sets:to_list(ORSetRes7).
% [bar,baz,foo]

Add-Wins ORSet CRDT:

Observed-Remove Set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set
{filter, Function}
Filter elements in the set

Example:

AWSetType = state_awset.
AWSetVarName = <<"awset">>.
AWSetVal1 = foo.
AWSetVal2 = bar.
AWSetVal3 = baz.
AWSetAllVals = [AWSetVal1, AWSetVal2, AWSetVal3].

{ok, {AWSet, _, _, _}} = lasp:declare({AWSetVarName, AWSetType}, AWSetType).

{ok, {AWSet1, _, _, _}} = lasp:update(AWSet, {add, AWSetVal1}, self()).
{ok, {AWSet2, _, _, _}} = lasp:update(AWSet1, {add, AWSetVal2}, self()).
% repeat value
{ok, {AWSet3, _, _, _}} = lasp:update(AWSet2, {add, AWSetVal1}, self()).
{ok, {AWSet4, _, _, _}} = lasp:update(AWSet3, {add, AWSetVal3}, self()).

{ok, AWSetRes4} = lasp:query(AWSet4).
sets:to_list(AWSetRes4).
% [bar,baz,foo]

{ok, {AWSet5, _, _, _}} = lasp:update(AWSet4, {rmv, AWSetVal3}, self()).
{ok, AWSetRes5} = lasp:query(AWSet5).
sets:to_list(AWSetRes5).
% [bar,foo]

{ok, {AWSet6, _, _, _}} = lasp:update(AWSet5, {rmv_all, AWSetAllVals}, self()).
{ok, AWSetRes6} = lasp:query(AWSet6).
sets:to_list(AWSetRes6).
% []

{ok, {AWSet7, _, _, _}} = lasp:update(AWSet6, {add_all, AWSetAllVals}, self()).
{ok, AWSetRes7} = lasp:query(AWSet7).
sets:to_list(AWSetRes7).
% [bar,baz,foo]

Add-Wins Set CRDT with the provenance semiring:

add-wins set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

AWPSSetType = state_awset_ps.
AWPSSetVarName = <<"awset_ps">>.
AWPSSetVal1 = foo.
AWPSSetVal2 = bar.
AWPSSetVal3 = baz.
AWPSSetAllVals = [AWPSSetVal1, AWPSSetVal2, AWPSSetVal3].

{ok, {AWPSSet, _, _, _}} = lasp:declare({AWPSSetVarName, AWPSSetType}, AWPSSetType).

{ok, {AWPSSet1, _, _, _}} = lasp:update(AWPSSet, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet2, _, _, _}} = lasp:update(AWPSSet1, {add, AWPSSetVal2}, self()).
% repeat value
{ok, {AWPSSet3, _, _, _}} = lasp:update(AWPSSet2, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet4, _, _, _}} = lasp:update(AWPSSet3, {add, AWPSSetVal3}, self()).

{ok, AWPSSetRes4} = lasp:query(AWPSSet4).
sets:to_list(AWPSSetRes4).
% [bar,baz,foo]

{ok, {AWPSSet5, _, _, _}} = lasp:update(AWPSSet4, {rmv, AWPSSetVal3}, self()).
{ok, AWPSSetRes5} = lasp:query(AWPSSet5).
sets:to_list(AWPSSetRes5).
% [bar,foo]

{ok, {AWPSSet6, _, _, _}} = lasp:update(AWPSSet5, {rmv_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes6} = lasp:query(AWPSSet6).
sets:to_list(AWPSSetRes6).
% []

{ok, {AWPSSet7, _, _, _}} = lasp:update(AWPSSet6, {add_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes7} = lasp:query(AWPSSet7).
sets:to_list(AWPSSetRes7).
% [bar,baz,foo]

GSet CRDT:

Grow only set

Operations:

{add, Element}
Add Element to the Set

Example:

GSetType = state_gset.
GSetVarName = <<"gset">>.
GSetVal1 = foo.
GSetVal2 = bar.
GSetVal3 = baz.
GSetAllVals = [GSetVal1, GSetVal2, GSetVal3].

{ok, {GSet, _, _, _}} = lasp:declare({GSetVarName, GSetType}, GSetType).

{ok, {GSet1, _, _, _}} = lasp:update(GSet, {add, GSetVal1}, self()).
{ok, {GSet2, _, _, _}} = lasp:update(GSet1, {add, GSetVal2}, self()).
% repeat value
{ok, {GSet3, _, _, _}} = lasp:update(GSet2, {add, GSetVal1}, self()).
{ok, {GSet4, _, _, _}} = lasp:update(GSet3, {add, GSetVal3}, self()).

{ok, GSetRes4} = lasp:query(GSet4).
sets:to_list(GSetRes4).
% [bar,baz,foo]

2PSet CRDT:

Two-Phased Set

Once removed, elements cannot be added again.

Also, this is not an observed removed variant.

This means elements can be removed before being in the set.

Operations:

{add, Element}
Add Element to the Set
{rmv, Element}
Remove Element from the set

Example:

TPSetType = state_awset_ps.
TPSetVarName = <<"awset_ps">>.
TPSetVal1 = foo.
TPSetVal2 = bar.
TPSetVal3 = baz.
TPSetAllVals = [TPSetVal1, TPSetVal2, TPSetVal3].

{ok, {TPSet, _, _, _}} = lasp:declare({TPSetVarName, TPSetType}, TPSetType).

{ok, {TPSet1, _, _, _}} = lasp:update(TPSet, {add, TPSetVal1}, self()).
{ok, {TPSet2, _, _, _}} = lasp:update(TPSet1, {add, TPSetVal2}, self()).
% repeat value
{ok, {TPSet3, _, _, _}} = lasp:update(TPSet2, {add, TPSetVal1}, self()).
{ok, {TPSet4, _, _, _}} = lasp:update(TPSet3, {add, TPSetVal3}, self()).

{ok, TPSetRes4} = lasp:query(TPSet4).
sets:to_list(TPSetRes4).
% [bar,baz,foo]

{ok, {TPSet5, _, _, _}} = lasp:update(TPSet4, {rmv, TPSetVal3}, self()).
{ok, TPSetRes5} = lasp:query(TPSet5).
sets:to_list(TPSetRes5).
% [bar,foo]

Facundo Batista: A todo tren con la Asociación Civil


Durante el año terminó de salir y empezamos a hacer girar la Asociación Civil Python Argentina.

Hay que hacer TANTAS cosas!! Mucho papel, mucha firma, mucho trámite por todos lados...

Lo bueno es que ya la podemos disfrutar, ya la podemos aprovechar. La PyCon del mes pasado se hizo "legalmente" dentro de la Asociación. O sea, aunque los organizadores fueron realmente Tutuca y Gaucho, a nivel legal/contabilidad la responsable es la asociación... Leandro y yo tuvimos que laburar un montón también, esperemos que cuando estén los mecanismos más afilados, sea menos carga.

Es que muchas cosas que hicimos (para la PyCon o para la Asociación en general) eran "por primera vez", y eso nos tomó bastante trabajo a varios de la Comisión Directiva... definir cómo íbamos a manejar una caja chica con el Contador, estructurar los niveles de socios beneficiarios (para empresas o instituciones), ver cómo se iban a manejar las becas, analizar la mejor manera de reunir los datos legales de cada persona siendo lo más laxos posibles, hasta armar un circuito de registración que le permita a la gente ser socios de PyAr con el menos laburo posible.

Pero se fue haciendo. Noviembre estuvo cargadísimo por la PyCon en sí, y luego de la conferencia estuvimos a full procesando los formularios de socios que se llenaron esos días, más todo el laburo a nivel pagos y tesorería para terminar de cerrar todo. Pero tambien hicimos la asamblea de renovación de autoridades entre los que éramos socios en ese momento, y empezamos a comunicarnos por las redes (contando que Onapsis era la primer socia benefactora, agradeciendo a GCoop por sacarnos y cuidarnos los dominios .org.ar para la comunidad, etc.)

El broche de oro de diciembre fue el jueves pasado cuando nos juntamos los que pudimos de la Comisión Directiva en Devecoop, la cooperativa de desarrollo de software que gentilmente nos presta el lugar donde están ellos para tener la sede legal de la Asociación... nos caimos en su lugar de trabajo y les agradecimos de la mejor manera: con comida y bebida :p

La mejor manera de decir gracias :)

Los próximos pasos son definir cómo vamos a manejar el dinero del próximo PyCamp, empezar a procesar los registros de la gente que llenó el formulario online para hacerse socia/os, y hacer una reunión de Comisión Directiva para incorporara la/os nueva/os socia/os que ya tenemos!

Damián Avila: Binder + Nikola + Jupyter + Github = Blogging resourceless

You are in vacation time but you want to blog something nice to share with your friends.

And your blog is powered by Nikola.

And you are a Jupyter Notebook user.

But you don't have your laptop with you, because you are on vacation time, remember? ;-)

But you still have your phone and some connectivity.

What do you think if I say that you have a complete workflow to write your blogpost, build the site and deploy it just using your phone? But without using computational resources from it (that would be also interesting, btw).

Don't you believe me? Just read this post and I will show you how you can make this possible... and fun!

Read more… (16 min remaining to read)

Facundo Batista: Películas (y series) de fin de año


No todo son películas en la vida, también sigo con series... Estoy terminando la quinta temporada de Person of Interest (bastante repetitiva, pero sigue interesante) y me quedan la terceras temporadas de Halt and Catch Fire (pensé que me iba a aburrir un toque pero está muy bien) y Bron/Broen (muy muy buena, con unos personajes bárbaros).

Pregunta, ¿por qué Wikipedia en inglés no mantiene el nombre original de esta última serie, sino que usa la traducción yanqui de la misma? Guarda, che, no les vayan a tocar la dominación cultural de jolivud... Por otro lado, para Wikipedia en danés y en sueco parecen llamarse sólo "el puente", perdiendo también la dualidad que se ve en el título de la serie (y que es tan importante para la misma).

Con los chicos vemos (mezclado con cosas de Encuentro y Paka Paka) al Superagente 86 y Doctor Who (la de este siglo). Con Moni vamos por la segunda de Merlí, Y yo voy mechando por ahí la novena de The Big Bang Theory y la séptima (y última) de Star Trek TNG.

En fin, las películas:

  • 10 Cloverfield Lane: +0. Parece de esas películas previsibles, pero no. Sorprende.
  • 400 Days: -0. Lleva bien la tensión, pero las actuaciones no son buenas... y el desenlace menos.
  • Al final del túnel: +1. Muy buena! Te mantiene agarrado de la silla hasta el final, está muy bien hecha.
  • Amnesiac: -0. Lenta, y aunque da un par de giros interesantes, es más de lo mismo y aburre.
  • Arrival: +1. Hermosa. No, no es una de extraterrestres y militares. Es tanto, tanto más que eso...
  • Blade Runner 2049: +1. La historia no es demasiado densa y no termina de aportar demasiado, pero la fotografía, la música y todo lo conceptual está genial
  • Creative Control: -1. Ni la terminé de ver. La dinámica de los lentes de realidad aumentada y su interfaz no me interesó lo suficiente como para contrarrestar lo soso del resto de la película.
  • Criminal: +0. Tiene momentos interesantes, bastantes, pero le resta que atrasa 20 años con lo de que el malo es tan malo que al final es más bueno que los buenos...
  • Hail, Caesar!: -0. Bizarra, pero no tanto como para ser lo suficientemente entretenida.
  • Hush: +0. La típica de "un loco malo que mata un montón" pero con una vuelta de tuerca que la vuelve interesante
  • Kill Command: -0. iene sus momentos con respecto a la inteligencia artificial y como los robots podrían tomar control, pero le resta mucho que sea tanto de guerra, que tenga demasiados momentos "terminator 1", y encima un final abierto, como si la quisieran continuar...
  • La belle saison: +1. Una hermosa historia de amor y feminismo.
  • Precious Cargo: -1. Tan llena de clichés la primer media hora que la saqué, no había *nada* nuevo.
  • Rock the Kasbah: +0. Una historia divertida y simpática. Para pasar el rato.
  • Snowden: +1. Me gustó mucho, incluso habiendo visto (o quizás eso actuó en favor?) Citizenfour
  • The Face of an Angel: -0. Tiene buenos movimientos, pero no va a ningún lado.
  • The Girl on the Train: +0. La historia está buena, enroscada pero bien... eso sí, la peli es demasiado lenta...
  • The Hunger Games: Catching Fire, The Hunger Games: Mockingjay - Part 1, y The Hunger Games: Mockingjay - Part 2: +0. Lo más interesante de estas tres (dos) continuaciones es todo el relato de lo que sucede a nivel social con la gente, y el rol de la chica con eso y con sus propios temores y lo que la mueve... pero si querés sólo una prelícula de acción también paga, pero para eso solamente es un poco larga.
  • The Huntsman: Winter's War: +0. Es linda, mantiene ritmo, mezcla todo muy bien, pero no deja de ser una de fantasía para adolescentes.
  • The Man Who Knew Infinity: +1. Una película maravillosa, no sólo para los que nos gusta matemática, sino apta para el resto también (y de paso se llevan un poquito sobre lo que es la matemática...).
  • Valerian and the City of a Thousand Planets: +0. Es divertida y tiene muchos conceptos para pensar. A nivel gráfico y/o diseño de aliens y "mundos", es impecable. Un poco infantil, algo sexista también. Demasiado romántica.
  • Youth: -1. Tan lenta y sin dirección clara o interesante que me aburrió y la corté a la mitad.

Un buen paquete de peliculas anotadas para ver:

  • A Scanner Darkly (2006; Animation, Crime, Drama, Mystery, Sci-Fi, Thriller) In a totalitarian society in a near future, the undercover detective Bob Archor is working with a small time group of drug users trying to reach the big distributors of a brain-damaging drug called Substance D. His assignment is promoted by the recovery center New Path Corporation, and when Bob begins to lose his own identity and have schizophrenic behavior, he is submitted to tests to check his mental conditions.::Claudio Carvalho, Rio de Janeiro, Brazil [D: Richard Linklater; A: Rory Cochrane, Robert Downey Jr., Mitch Baker]
  • Counterpart (2018; Drama, Sci-Fi, Thriller) A UN employee discovers the agency he works for is hiding a gateway to a parallel dimension.::KalanKeis [D: Alik Sakharov, Morten Tyldum; A: Harry Lloyd, Nazanin Boniadi, J.K. Simmons]
  • Dunkirk (2017; Action, Drama, History, Thriller, War) Evacuation of Allied soldiers from Belgium, the British Empire, and France, who were cut off and surrounded by the German army from the beaches and harbor of Dunkirk, France, between May 26- June 04, 1940, during Battle of France in World War II.::Harvey [D: Christopher Nolan; A: Fionn Whitehead, Damien Bonnard, Aneurin Barnard]
  • Gerald's Game (2017; Drama, Horror, Thriller) When a harmless game between a married couple in a remote retreat suddenly becomes a harrowing fight for survival, wife Jessie must confront long-buried demons within her own mind - and possibly lurking in the shadows of her seemingly empty house.::Intrepid Pictures [D: Mike Flanagan; A: Carla Gugino, Bruce Greenwood, Chiara Aurelia]
  • It (2017; Horror, Thriller) In the Town of Derry, the local kids are disappearing one by one, leaving behind bloody remains. In a place known as 'The Barrens', a group of seven kids are united by their horrifying and strange encounters with an evil clown and their determination to kill It.::Emma Chapman [D: Andy Muschietti; A: Jaeden Lieberher, Jeremy Ray Taylor, Sophia Lillis]
  • La Cordillera (2017; Drama, Mystery, Thriller) The president of Argentina, Hernán Blanco, is facing a very important decision. He is participating in a meeting between different state leaders, which takes place in La Cordillera. From there, in the middle of the Summit of Latin American presidents, he will have to be able to solve a very complicated personal matter that can affect both his private and public life.::Binquin_Black [D: Santiago Mitre; A: Walter Andrade, Ricardo Darín, Dolores Fonzi]
  • Los decentes (2016; Drama) A housemaid, working in an exclusive gated community in the outskirts of Buenos Aires, embarks on a journey of sexual and mental liberation in a nudist swinger-club boarding the high security walls.::morroviolet [D: Lukas Valenta Rinner; A: Iride Mockert, Ivanna Colona Olsen, Mariano Sayavedra]
  • Bright (2017; Action, Crime, Fantasy, Sci-Fi, Thriller) Set in a world where mystical creatures live side by side with humans. A human cop is forced to work with an Orc to find a weapon everyone is prepared to kill for. [D: David Ayer; A: Will Smith, Noomi Rapace, Joel Edgerton]
  • El Aprendiz (2016; Crime, Drama) A young chef's apprentice finds himself at a crucial crossroads: love, family, friends, or career. Only one path can be followed and the choice will change his life forever. [D: Tomás De Leone; A: Germán de Silva, Nahuel Viale, Malena Sánchez]
  • Future Man (2017; Comedy, Sci-Fi) Josh Futturman, a janitor by day and a gamer by night, is recruited by mysterious visitors to travel through time to prevent the extinction of humanity.::Anonymous [D: Nisha Ganatra, Evan Goldberg, Seth Rogen, Brandon Trost; A: Josh Hutcherson, Derek Wilson, Eliza Coupe]
  • Gun Shy (2017; Action, Adventure, Comedy, Crime, Thriller) The story follows Turk Henry (Antonio Banderas); a mega platinum rock star who's married to a supermodel (Olga Kurylenko) and rich beyond his wildest dreams. Whilst on holiday, his wife is mysteriously abducted by a group of renegade, ship-less pirates. With little assistance from local authorities Turk is forced to embark on a mission to rescue his wife. With life skills better suited to playing bass, playing the field, and partying he is forced to navigate through deadly jungles and take on ruthless bandits in this truly hilarious, action-packed romp.::Teaser-Trailer.com [D: Simon West; A: Antonio Banderas, Olga Kurylenko, Ben Cura]
  • Incredibles 2 (2018; Animation, Action, Adventure, Family) Bob Parr (Mr. Incredible) is left to care for Jack-Jack while Helen (Elastigirl) is out saving the world. [D: Brad Bird; A: Samuel L. Jackson, Holly Hunter, Catherine Keener]
  • Infancia clandestina (2011; Drama) Juan lives in clandestinity. Just like his mum, his dad and his adored uncle Beto, outside his home he has another name. At school, Juan is known as Ernesto. And he meets María, who only has one name. Based on true facts, set in the Argentina of 1979, this film is "one about love".::Historias Cinematográficas [D: Benjamín Ávila; A: Ernesto Alterio, Natalia Oreiro, César Troncoso]
  • Marjorie Prime (2017; Comedy, Drama, Mystery, Sci-Fi) In the near future, a time of artificial intelligence: 86-year-old Marjorie has a handsome new companion who looks like her deceased husband and is programmed to feed the story of her life back to her. What would we remember, and what would we forget, if given the chance? [D: Michael Almereyda; A: Hannah Gross, Jon Hamm, Geena Davis]
  • Mother! (2017; Drama, Horror, Mystery) Amidst a wild flat meadow encircled by an Edenic lush forest, a couple has cocooned itself in a secluded grand mansion that was not so long ago burned to the ground, devotedly restored by the supportive wife. Within this safe environment, the once famous middle-aged poet husband is desirous of creating his magnum opus, however, he seems unable to break out of the persistent creative rut that haunts him. And then, unexpectedly, a knock at the door and the sudden arrival of a cryptic late-night visitor and his intrusive wife will stimulate the writer's stagnant imagination, and much to the perplexed wife's surprise, the more chaos he lets in their haven, the better for his punctured male ego. In the end, will this incremental mess blemish irreparably the couple's inviolable sanctuary?::Nick Riganas [D: Darren Aronofsky; A: Jennifer Lawrence, Javier Bardem, Ed Harris]
  • Ready Player One (2018; Action, Adventure, Sci-Fi, Thriller) Film centers on a young outcast named Wade Watts. In the near future, Watts escapes from his daily drudgery by logging onto an MMO game called 'The Oasis'. When the game's billionaire founder dies, he offers players his fortune as the prize in an easter egg hunt within The Oasis. Watts gets in on the action then after five years finds himself facing off against corporate foes who will go to any lengths to get the money -- in both the real world and in The Oasis.::Anonymous [D: Steven Spielberg; A: Olivia Cooke, Hannah John-Kamen, Ben Mendelsohn]
  • Replicas (2018; Crime, Mystery, Sci-Fi, Thriller) A daring synthetic biologist who, after a car accident kills his family, will stop at nothing to bring them back, even if it means pitting himself against a government-controlled laboratory, a police task force and the physical laws of science. [D: Jeffrey Nachmanoff; A: Keanu Reeves, Alice Eve, Emily Alyn Lind]
  • Suburbicon (2017; Crime, Drama, Mystery, Thriller) In the bosom of Suburbicon, a family-centred, all-white utopia of manicured lawns and friendly locals, a simmering tension is brewing, as the first African-American family moves in the idyllic community, in the hot summer of 1959. However, as the patriarch Gardner Lodge and his family start catching a few disturbing glimpses of the once welcoming neighbourhood's dark underbelly, acts of unprecedented violence paired with a gruesome death will inevitably blemish Suburbicon's picture-perfect facade. Who would have thought that darkness resides even in Paradise?::Nick Riganas [D: George Clooney; A: Steve Monroe, Gavin Wilde, Landon Gordon]
  • The Current War (2017; Biography, Drama, History) Starring Benedict Cumberbatch as Thomas Edison and Michael Shannon as George Westinghouse, THE CURRENT WAR is the epic story of the cutthroat competition between the greatest inventors of the industrial age over whose electrical system would power the new century. Backed by J.P. Morgan, Edison dazzles the world by lighting Manhattan. But Westinghouse, aided by Nikola Tesla, has seen fatal flaws in Edison's direct current design. Igniting a war of currents, Westinghouse and Tesla bet everything on risky and dangerous alternating current. Directed by Alfonso Gomez-Rejon (Me and Earl and the Dying Girl) and written by playwright Michael Mitnick (Sex Lives of our Parents), THE CURRENT WAR also stars Katherine Waterston, Nicholas Hoult, Tom Holland, Matthew Macfadyen, and Tuppence Middleton. [D: Alfonso Gomez-Rejon; A: Tom Holland, Katherine Waterston, Benedict Cumberbatch]

Finalmente, el conteo de pendientes por fecha:

(Ene-2012)    3
(Jul-2012)   11
(Nov-2012)   11   6
(Feb-2013)   14   8   2
(Jun-2013)   15  15  11   2
(Sep-2013)   18  17  16   8
(Dic-2013)   12  12  12  12   4
(Abr-2014)    9   8   8   8   3
(Jul-2014)   10  10  10  10  10   5   1
(Nov-2014)       24  22  22  22  22   7
(Feb-2015)           13  13  13  13  10
(Jun-2015)               16  16  15  13  11   1
(Dic-2015)                   21  19  19  18   6   1
(May-2016)                       26  25  23  21   9
(Sep-2016)                           19  19  18  14
(Feb-2017)                               26  25  23
(Jun-2017)                                   23  23
(Dic-2017)                                       19
Total:      103 100  94  91  89 100  94  97  94  89

Mariano Guerra: rebar3 partisan template

I created a rebar3 template to start a partisan project, the template is here:

rebar3 partisan template

What is partisan? from the docs:

Partisan is the technology that provides Lasp's scalable cluster
membership. Partisan bypasses the use of Distributed Erlang for manual
connection management via TCP, and has several pluggable backends for
different deployment scenarios.

To use:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_partisan.git ~/.config/rebar3/templates/rebar3_template_partisan
rebar3 new rebar3_template_partisan name=party
cd party
make release
make console

Clustering:

make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

# join node1 to node2 manually:
./_build/dev1/rel/party/bin/party-admin cluster join party2@127.0.0.1

# check node1 members
./_build/dev1/rel/party/bin/party-admin cluster members

# check node1 connections
./_build/dev1/rel/party/bin/party-admin cluster connections

A video to show how to use:

Facundo Batista: PyCon Argentina 2017


Este fin de semana se realizó una nueva PyCon en Argentina, esta vez (y es la primera vez que repetimos ciudad) en Córdoba.

Yo di dos charlas "formales"... bah, una charla ("Emulando paralelismo de forma asincrónica") y un taller ("Introducción a Python"). También di una mini-plenaria de 20 minutos donde hablé sobre la Asociación Civil Python Argentina.

Grupal

Hubieron un montón de charlas interesantes! Estas son las que más me gustaron:

  • "Distribuyendo código de py a PyPI", donde Matías Bordese nos contó todo el proceso de empaquetamiento y publicación de un programa o biblioteca en Python.
  • "Robótica educativa con software y hardware libre... y Python!", de Valentín Basel, que mostró cómo armaron el proyecto Icaro de forma que los niños puedan hacer cosas con componentes baratos o reciclado y aprender a programar en el camino.
  • "Magicicada: el fork open-source de Ubuntu One Filesync" donde Naty Bidart contó un poco la historia del servicio de sincronización de archivos en el que trabajamos en Canonical, su evolución luego de que fue liberado a Magicicada, y mostró la arquitectura del sistema y sus complejidades.
  • "Django Channeled", de Jonatas Baldin, que mostró como el concepto de "conexión permanente del cliente al servidor usado para que el servidor le pueda mandar mensajes al cliente en cualquier momento" se implementa en Django de manera elegante, integrándose correctamente con el resto del framework.
  • "Me están espiando! Cómo saber con Python si el imperialismo te persigue o te pasaste de Focusyn", de Nicolás Demarchi, donde aprendimos cómo calcular la posición de todos los satélites alrededor de la tierra, y darnos cuenta que si en un momento en particular podemos ver a determinado satélite (lo cual significa que ese satélite también nos puede ver).
  • "De la Rabbit, Pascal y Stored Procedures a la Beaglebone Black, Flask y PyZMQ", de Leandro Colombo Viña, la historia de la evolución de un hardware determinado, donde con conceptos modernos de comunicación y Python se lograron revolucionar las especificaciones del producto.
  • "Python en el browser, mil intentos y un invento", de Roberto Alsina, que nos contó un poco de historia y un poco de realidad por si queremos programar "en el browser, del lado del cliente" en Python, y no en javascript.

Las plenarias también estuvieron buenas. Aleksandra Sendecka nos contó en "Anatomy of a Code Review" el por qué y para qué de las revisiones de código por parte del equipo de trabajo, y especialmente el cómo, con un montón de consejos interesantes. Por otro lado Lucio Torre en "No hay tal cosa como un almuerzo gratis en temas de software" nos habló sobre cómo en desarrollo siempre las acciones que nos dan algo de beneficio por un lado, nos traen algo de perjuicio por otro, y que muchas veces entender dónde uno está parado (y saber leer eso, especialmente cuando uno no tiene toda la información y se basa en prejuicios) es fundamental para la evolución del desarrollo del sistema.

A nivel social, la comunidad de Python Argentina sigue mostrándose ejemplar. No sólo en lo personal (donde es una excusa para reencontrarse con amiga/os y charlar, charlar, charlar) sino también en la inclusión de nuevas personas en el grupo. El ejemplo que más me resuena a este respecto es esta serie de tuits:

Inclusión

Hasta donde sé se grabaron todas las charlas, así que avisaré cuando estén subidas. Y acá tienen fotos mías y las que sacó Yami (incluyendo la grupal que reproduzco arriba).

Mariano Guerra: Let's build a key value store in erlang

The code for this post is at https://github.com/marianoguerra/akvs

We first need to have erlang installed, I will show you how to setup any version you want to use, and a way to have the version I will use for this without affecting any other installation you may have.

Setting up kerl

For this we will use kerl, from it's github README:

Easy building and installing of Erlang/OTP instances.

Kerl aims to be shell agnostic and its only dependencies, excluding what's
required to actually build Erlang/OTP, are curl and git.

So, first we need to fetch kerl:

# create bin folder in our home directory if it's not already there
mkdir -p ~/bin

# cd to it
cd ~/bin

# download kerl script
curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl

# set execution permitions for our user
chmod u+x kerl

You will need to add ~/bin to your PATH variable so your shell can find the kerl script, you can do it like this in your shell:

# set the PATH environment variable to the value it had before plus a colon
# (path separator) and a new path which points to the bin folder we just
# created
PATH=$PATH:$HOME/bin

If you want to make this work every time you start a shell you need to put it it the rc file of your shell of choice, for bash it's ~/.bashrc, for zsh it's .zshrc, check your shell's docs for other shells, you will have to add a line like this:

export PATH=$PATH:$HOME/bin

After this, start a new shell or source your rc file so that it picks up your new PATH variable, you can check that it's set correctly by running:

echo $PATH

Building an Erlang release with kerl

We have kerl installed and available in our shell, now we need to build an Erlang release of our choice, for this we will need a compiler and other tools and libraries needed to compile it:

This are instructions on ubuntu 17.10, check the names for those packages on your distribution.

# required: basic tools and libraries needed
# (compiler, curses for the shell, ssl for crypto)
sudo apt-get -y install build-essential m4 libncurses5-dev libssl-dev

# optonal: if you want odbc support (database connectivity)
sudo apt-get install unixodbc-dev

# optonal: if you want pdf docs you need apache fop and xslt tools and java (fop is a java project)
sudo apt-get install -y fop xsltproc default-jdk

# optional: if you want to build jinterface you need a JDK
sudo apt-get install -y default-jdk

# optional: if you want wx (desktop GUI modules)
sudo apt-get install -y libwxgtk3.0-dev

Now that we have everything we need we can finally build our erlang release.

First we fetch an updated list of releases:

kerl update releases

The output in my case:

The available releases are:

R10B-0 R10B-10 R10B-1a R10B-2 R10B-3 R10B-4 R10B-5 R10B-6 R10B-7 R10B-8
R10B-9 R11B-0 R11B-1 R11B-2 R11B-3 R11B-4 R11B-5 R12B-0 R12B-1 R12B-2 R12B-3
R12B-4 R12B-5 R13A R13B01 R13B02-1 R13B02 R13B03 R13B04 R13B R14A R14B01
R14B02 R14B03 R14B04 R14B_erts-5.8.1.1 R14B R15B01 R15B02
R15B02_with_MSVCR100_installer_fix R15B03-1 R15B03 R15B
R16A_RELEASE_CANDIDATE R16B01 R16B02 R16B03-1 R16B03 R16B 17.0-rc1 17.0-rc2
17.0 17.1 17.3 17.4 17.5 18.0 18.1 18.2.1 18.2 18.3 19.0 19.1 19.2 19.3
20.0 20.1

Let's build the 20.1 version:

# this will take a while
kerl build 20.1 20.1

And install it:

kerl install 20.1 ~/bin/erl-20.1

Now everytime we want to use this version of erlang we need to run:

. $HOME/bin/erl-20.1/activate

Setting up rebar3

Now we have erlang, we need a build tool, we are going to use rebar3:

# download rebar3 to our bin directory
wget https://s3.amazonaws.com/rebar3/rebar3 -O $HOME/bin/rebar3

# set execution permissions for our user
chmod u+x rebar3

Just in case you have problems running the rebar3 commands with a different version, here's the version I'm using:

rebar3 version

Output:

rebar 3.4.7 on Erlang/OTP 20 Erts 9.1

Setting up our project

We are ready to start our project, go to a folder where you keep your code and if you haven't done it yet, add $HOME/bin to your path and activate erlang 20.1 as shown above, then run:

rebar3 new release name=akvs

The output should be something like this:

===> Writing akvs/apps/akvs/src/akvs_app.erl
===> Writing akvs/apps/akvs/src/akvs_sup.erl
===> Writing akvs/apps/akvs/src/akvs.app.src
===> Writing akvs/rebar.config
===> Writing akvs/config/sys.config
===> Writing akvs/config/vm.args
===> Writing akvs/.gitignore
===> Writing akvs/LICENSE
===> Writing akvs/README.md

Let's see what each file does:

First of all, we created a release, which is a kind of project that can have more than one application (a common way to structure a project is into applications and libraries)

Under the apps folder are all the applications we mantain for this release, in our case we only have one application, named akvc.

Under the akvs application folder we have a src folder where all the source code for that application will live, we can add other folders there, for tests, header files, private files etc.

apps/akvs/src/akvs_app.erl

The $APPNAME_app module is called when starting and stopping the app to do the setup and tear down of the application.

Check Erlang's manual for application or the user's guide entry for application for more information.

apps/akvs/src/akvs_sup.erl

The $APPNAME_sup module defines the root supervisor for the application, it implements the supervisor behavior and will be "hooked" into the supervisor hierarchy of this release when initialized.

Check Erlang's manual for supervisor or the user's guide entry for supervisor for more information.

apps/akvs/src/akvs.app.src

The $APPNANE.app.src is a file that contains metadata about this app.

Check ERlang's manual for application resource file for more information.

rebar.config

Contains information about the project, dependencies, how to build it, test it, and how to build a release.

Check rebar3 docs for details.

config/sys.config

Configuration parameters for the application.

Check sys.config's manual page for more information.

config/vm.args

Configuration parameters for the Erlang VM.

.gitignore

Git specific, files to ignore.

LICENSE

The license for this project, you should change it if the Apache License 2.0 isn't the one you want.

README.md

Project's readme.

Starting it for the first time

First we need to build a release:

cd akvs

# build a release, the result will be at _build/default/rel/akvs
rebar3 release

# start the release and attach to the console
./_build/default/rel/akvs/bin/akvs console

The output in my case is (redacted for clarity):

Exec: bin/erl-20.1/erts-9.1/bin/erlexec
         -boot src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/akvs
         -mode embedded -boot_var ERTS_LIB_DIR bin/erl-20.1/lib
         -config src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/sys.config
         -args_file src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/vm.args
         -pa -- console

Root: src/erl/akvs/_build/default/rel/akvs
src/erl/akvs/_build/default/rel/akvs

Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:4:4] [ds:4:4:10]
                          [async-threads:30] [kernel-poll:true]


=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_safe_sup}
                         started: [{pid,<0.225.0>},
                                           {id,alarm_handler},
                                           {mfargs,{alarm_handler,start_link,[]}},
                                           {restart_type,permanent},
                                           {shutdown,2000},
                                           {child_type,worker}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_sup}
                         started: [{pid,<0.224.0>},
                                           {id,sasl_safe_sup},
                                           {mfargs,
                                                   {supervisor,start_link,
                                                           [{local,sasl_safe_sup},sasl,safe]}},
                                           {restart_type,permanent},
                                           {shutdown,infinity},
                                           {child_type,supervisor}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_sup}
                         started: [{pid,<0.226.0>},
                                           {id,release_handler},
                                           {mfargs,{release_handler,start_link,[]}},
                                           {restart_type,permanent},
                                           {shutdown,2000},
                                           {child_type,worker}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                 application: sasl
                  started_at: akvs@ganesha
Eshell V9.1  (abort with ^G)
(akvs@ganesha)1>

There's not much we can do with our project at this stage, so we will just stop it and exit by running the q(). function in the shell:

(akvs@ganesha)1> q().
ok

Coding (and testing) the Key Value store modules

The way I usually code in erlang is to first build a stateless module that has an init function that returns some state, all other functions expect that state as first parameter, then those functions do something and return the state and the result.

This modules are really easy to use in the shell and test.

This will be our first module, we will call it akvs_kv and it will have the following API:

%% types:

-type error() :: {error, {atom(), iolist(), map()}}.
-type key()   :: binary().
-type value() :: any().

% we don't want other modules to know/care about the internal structure of
% the state type
-opaque state() :: map().

%% functions:

%% @doc create a new instance of a key value store
-spec new(map()) -> {ok, state()} | error().

%% @doc dispose resources associated with a previously created kv store
-spec dispose(state()) -> ok | error().

%% @doc set a value for a key in a kv store
-spec set(state(), key(), value()) -> {ok, state()} | error().

%% @doc get a value for a key or an error if not found
-spec get(state(), key()) -> {ok, value()} | error().

%% @doc get a value for a key or a default value if not found
-spec get(state(), key(), value()) -> {ok, value()} | error().

%% @doc remove a value for a key, if not found do nothing
-spec del(state(), key()) -> {ok, state()} | error().

Notice that to specify the API I used a specification of the types and functions, this is called spec, read more about it at the Types and Function Specifications sectio in the erlang reference manual.

Also for documentation comments I'm using the edoc format, read more about it at the edoc user's guide section.

You can see the full code of this module here: akvs_kv

But how do we know if it works?

At this point there are two ways: testing it in the shell, or writing tests for it, let's do the right thing and write some tests.

We are going to use Common Test for our tests.

First we need to create the test folder for our tests:

mkdir apps/akvs/test

Inside it we will create a module called akvs_kv_SUITE that will contain the tests for the akvs_kv module.

You can see the full code of this module here: akvs_kv_SUITE

To run the tests:

rebar3 ct

We can also use the type specs we defined to check our code using dialyzer:

rebar3 dialyzer

Everything seems to be right, let's move on to the next step.

But before that, in case you want to generate API docs for our code taking advantage of the edoc annotations, you can do so by running:

rebar3 edoc

And opening apps/akvs/doc/index.html with a browser.

Wrapping the state

Stateless modules are a good start and are really easy to test and use, but we don't want to pass the burden of threading the state to the users of our code, also we want to centralize the state management so that more than one process can call our module and see the state changes of other callers.

In this case we are using ETS to make it simpler but if our kv was backed by a map, or if we had some kind of cache, then state management would become really important to get right, otherwise the results seen by each caller would diverge.

To manage the state of our module we are going to wrap it in a process, a gen_server in this case.

The module will be called akvs_kv_s (_s for server, don't know if there's a convention for it).

The module is a basic gen_server that exposes a couple functions to call the kv API from the akvs_kv module, you can read the code here: akvs_kv_s.

We write tests for this module too, you can read the test's code here: akvs_kv_s_SUITE.

Run the tests:

rebar3 ct

An API for our key value stores

Now we can spawn a key value store in a gen_server and apply operations to it, but like with the stateless module, someone has to keep a reference to the process and provide a nicer way to find and operate on our key value stores, if it was only one it's easy to just start it as a registered process with a name and send messages to it by it's name, but in our case, we want to provide namespaces where each namespace holds a key value store of its own.

The abstract API or this module should be like this:

-type ns() :: binary().
-type key() :: akvs_kv:key().
-type value() :: akvs_kv:value().
-type error() :: akvs_kv:value().

%% @doc set Key to Value in namespace Ns
-spec set(ns(), key(), value()) -> ok | error().

%% @doc get Key from namespace Ns
-spec get(ns(), key()) -> {ok, value()} | error().

%% @doc get Key from namespace Ns or DefaultValue if Key not found
-spec get(ns(), key(), value()) -> {ok, value()} | error().

%% @doc delete  Key in namespace Ns
-spec del(ns(), key()) -> ok | error().

Right now we are going to solve the problem of who keeps the namespace to process mapping really simple so we can continue, we are going to setup a public ETS table at application startup and lookup the processes by namespace there, if not found we are going to start the process and register it under that namespace.

This solution is not recommendable at all but it will allow us to continue and since the API doesn't know a thing about the way we register/lookup namespaces we can explore different alternatives later.

You can view the source code for akvs module here: akvs and the tests here akvs_SUITE.

An HTTP API for our key value stores

We are at the point where we can expose our APIs to the world, we are going to do it by exposing a really basic HTTP API for it.

The API will look like this:

# set key in namespace to the binary value sent in body
# return status: 201
POST /kv/<namespace>/<key>
<body>

# get key in namespace
# return status:
#  200: if found
#  404: if not found
GET /kv/<namespace>/<key>

# delete key from namespace
# return status: 200
DELETE /kv/<namespace>/<key>

To create an HTTP API we need an HTTP server, in this case we will use Cowboy 2.

First we need to add it as a dependency in our rebar.config file in the deps section and in the release dependencies section.

Then we need to setup the routes in our application initialization code.

We are going to have only one route and handler, we are going to use a basic HTTP to keep it simple, you can read the handler's code here: akvs_h_kv.

Now we can test it by building a release, starting it and playing with the API using curl:

rebar3 release
_build/default/rel/akvs/bin/akvs console

In another shell:

curl http://localhost:8080/kv/foo/bar
Not Found

curl -X POST http://localhost:8080/kv/foo/bar -d "hello world"
Created

curl http://localhost:8080/kv/foo/bar
hello world

curl -X DELETE http://localhost:8080/kv/foo/bar
OK

curl http://localhost:8080/kv/foo/bar
Not Found

curl -X PUT http://localhost:8080/kv/foo/bar -d "hello world"
Method Not Allowed

Seems to work fine.

Now we can build a production release and try it:

rebar3 as prod release
cd _build/prod/rel
tar -czf akvs.tar.gz akvs
cd -
mv _build/prod/rel/akvs.tar.gz /tmp
cd /tmp
tar -xzf akvs.tar.gz
cd akvs
./bin/akvs start

The application is started, you can check it's running by pinging it:

./bin/akvs ping

In case you need, you can attach to it (you should exit with Ctrl+D, using q() won't only detach your console but also stop the system!):

./bin/akvs attach

You can try it again:

curl http://localhost:8080/kv/foo/bar
curl -X POST http://localhost:8080/kv/foo/bar -d "hello world"
curl http://localhost:8080/kv/foo/bar
curl -X DELETE http://localhost:8080/kv/foo/bar
curl http://localhost:8080/kv/foo/bar
curl -X PUT http://localhost:8080/kv/foo/bar -d "hello world"

When you are finished, you can stop it:

./bin/akvs stop

Now you can upload akvs.tar.gz to any bare server and start akvs there, as long as the operating system is similar (better if the same) as the one where you built the release, this is because when building the release we bundle the erlang runtime for simplicity, this assumes specific versions of libraries like libssl which may not be available on the target system if it's too different.

Another way is to build the release without bundling the erlang runtime and having it available on the target system, just make sure that the erlang runtime in the target system has the same version you used to build it, otherwise you may experience errors due to modules/functions not being available or bytecode incompatibility if the target runtime is older than the one used for the release.