Marcos Dione: callable-choices-for-django-rest-framework

At work I'm writing an API using Django/DRF. Suddenly I had to write an application (just a few pages for calling a few endpoints), so I (ab)used DRF's Serializers to build them. One of the problems I faced while doing this was that DRF's ChoiceField accepts only a sequence with the values for the dropdown, unlike Django's, who also accepts callables. This means that once you gave it a set of values, it never ever changes, at least until you restart the application.

Unless, of course, you cheat. Or hack. Aren't those synonyms?

class UpdatedSequence:
    def __init__(self, update_func):
        self.update_func = update_func
        self.restart = True

        self.data = None
        self.index = 0


    def __iter__(self):
        # we're our own iterator
        return self


    def __next__(self):
        # if we're iterating from the beginning, call the function
        # and cache the result
        if self.restart:
            self.data = self.update_func()
            self.index = 0

        try:
            datum = self.data[self.index]
        except IndexError:
            # we reached the limit, start all over
            self.restart = True
            raise StopIteration
        else:
            self.index += 1
            self.restart = False

        return datum

This simple class tracks when you start iterating over it and calls the function you pass to obtain the data. Then it iterates over the result. When you reach the end, it marks it to start all over, so the next time you iterate over it, it will call the function again. The function you pass can be the all() method of a QuerySet or anything else that goes fetch data and returns an iterable.

In my case in particular, I also added a TimedCache so I don't read twice the db to fill two dropdown with the same info in the same form:

class TimedCache:
    '''A function wrapper that caches the result for a while.'''
    def __init__(self, f, timeout):
        self.f = f
        self.timeout = timeout
        self.last_executed = None
        self.cache = None
        self.__name__ = f.__name__ + ' (Cached %ds)' % timeout


    def __call__(self):
        now = time.monotonic()

        if self.cache is None or (now - self.last_executed) > self.timeout:
            self.cache = self.f()
            self.last_executed = now

        return self.cache

I hope this helps someone.


python django drf

Mariano Guerra: ameo: Redis compatible GET,SET,DEL and PUBLISH/SUBSCRIBE on riak_core with WebSocket API

Last week I was invited by Erlang Solutions to give a talk at the London's Erlang Meetup and to help at a riak_core themed hackathon at their offices the next day.

The talk slides: Building distributed applications: riak_core vs partisan in case you are interested.

After giving the introduction to riak_core the teams started to work on their projects and since they were pretty busy and didn't need much help I decided to "participate" too by implementing an idea I had in mind for a while.

The result is called ameo and can be described as:

a Redis compatible, distributed in-memory key-value store and pubsub server
implemented using riak_core that exposes the topics via WebSockets.

Why?

Many languages make it hard to provide websocket connections and live connections with clients, they work on a request/response basis and/or make it really hard/expensive to handle multiple persistent connections.

I've seen solutions that involve starting a redis server and putting usually nodejs in the front to expose redis topics via websockets, this involves two moving parts and for many projects, managing nodejs which they may not have experience with.

The solution is to just start one instance of ameo or a cluster of ameos and expose the WebSocket API to clients and the Redis API to the servers.

Servers can use their preferred Redis client library and as long as they only use GET, PUT, DEL, PUBLISH, SUBSCRIBE and UNSUBSCRIBE it will look like they are talking to a Redis server.

On my way back I had some extra time at the airport so I implemented a basic web UI to play with the websocket client and to provide a reference implementation others can use.

You can see the result in this screencast:

Implementation details:

As said earlier I use riak_core for clustering, for the Redis part I took some modules from an Erlang implementation of Redis called edis and I created a library called edis_proto that allows any project to expose a Redis compatible API to their servers with a couple lines of code.

The WebSocket part is implemented using the Cowboy Web Server.

Mariano Guerra: ameo: Redis compatible GET,SET,DEL and PUBLISH/SUBSCRIBE on riak_core with WebSocket API

Last week I was invited by Erlang Solutions to give a talk at the London's Erlang Meetup and to help at a riak_core themed hackathon at their offices the next day.

The talk slides: Building distributed applications: riak_core vs partisan in case you are interested.

After giving the introduction to riak_core the teams started to work on their projects and since they were pretty busy and didn't need much help I decided to "participate" too by implementing an idea I had in mind for a while.

The result is called ameo and can be described as:

a Redis compatible, distributed in-memory key-value store and pubsub server
implemented using riak_core that exposes the topics via WebSockets.

Why?

Many languages make it hard to provide websocket connections and live connections with clients, they work on a request/response basis and/or make it really hard/expensive to handle multiple persistent connections.

I've seen solutions that involve starting a redis server and putting usually nodejs in the front to expose redis topics via websockets, this involves two moving parts and for many projects, managing nodejs which they may not have experience with.

The solution is to just start one instance of ameo or a cluster of ameos and expose the WebSocket API to clients and the Redis API to the servers.

Servers can use their preferred Redis client library and as long as they only use GET, PUT, DEL, PUBLISH, SUBSCRIBE and UNSUBSCRIBE it will look like they are talking to a Redis server.

On my way back I had some extra time at the airport so I implemented a basic web UI to play with the websocket client and to provide a reference implementation others can use.

You can see the result in this screencast:

Implementation details:

As said earlier I use riak_core for clustering, for the Redis part I took some modules from an Erlang implementation of Redis called edis and I created a library called edis_proto that allows any project to expose a Redis compatible API to their servers with a couple lines of code.

The WebSocket part is implemented using the Cowboy Web Server.

Mariano Guerra: Playing with Lasp in a 3 Node Cluster

After playing with Lasp in a single node in the previous post: Playing with Lasp and CRDTs

I created a rebar3 template to easily setup a 3 node cluster to play with Lasp where it shines, in a distributed environment.

Install rebar3 if you haven't already.

then install this template:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_lasp.git ~/.config/rebar3/templates/rebar3_template_lasp

Use

rebar3 new rebar3_template_lasp name=laspy
cd laspy

    # build the 3 node cluster
make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

On one of the nodes' shell run:

Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

On another one run:

{ok, AwMapRes} = lasp:query({<<"awmap">>,{state_awmap,[state_mvregister]}}).

AwMapRes.

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).

You should get:

[#{what => i_am_an_awmap_value}]

All the examples from the previous post should run.

A termcast (?) of the process:

Happy Lasping!

Mariano Guerra: Playing with Lasp in a 3 Node Cluster

After playing with Lasp in a single node in the previous post: Playing with Lasp and CRDTs

I created a rebar3 template to easily setup a 3 node cluster to play with Lasp where it shines, in a distributed environment.

Install rebar3 if you haven't already.

then install this template:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_lasp.git ~/.config/rebar3/templates/rebar3_template_lasp

Use

rebar3 new rebar3_template_lasp name=laspy
cd laspy

    # build the 3 node cluster
make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

On one of the nodes' shell run:

Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

On another one run:

{ok, AwMapRes} = lasp:query({<<"awmap">>,{state_awmap,[state_mvregister]}}).

AwMapRes.

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).

You should get:

[#{what => i_am_an_awmap_value}]

All the examples from the previous post should run.

A termcast (?) of the process:

Happy Lasping!

Mariano Guerra: Playing with Lasp and CRDTs

For a long time I was looking for some time to play with Lasp:

Lasp is a suite of libraries aimed at providing a comprehensive programming
system for planetary scale Elixir and Erlang applications.

Yesterday I had some time and played a little bit with it, and thanks to the help from @cmeik and @vitorenesduarte I got some code running and understood some things.

Here is what I achieved until now, note that I'm learning this stuff so I may say many things that are wrong, I will try to get people to review it and update it with the corrections.

Some initial snippets of code we will reuse:

% clear all variable bindings
f().
Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

A map with any type as value

If we want to have data which can be represented as a dict/map/key-value mapping, there are multiple alternatives we can use in lasp, here we will explore some.

First we are going to use an AwMap CRDT as a map which is defined as:

A dictionary where keys can be anything and the values are causal-CRDTs.

Since values must be casual-CRDTs we can't use a bare erlang type as value nor a non causal-CRDT type like lwwregister, which we will use later with another map type.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

We can also use a GMap CRDT (Grow Only Map, keys can't be removed):

A dictionary where keys can be anything and the values are join-semilattices.

In this case we can use non causal-CRDTs as values, to make it a little different we are going to use another register type, a lwwregister

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal = #{what => i_am_a_gmap_value}.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).
{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                            {set, Timestamp(), GMapVal}},
                                            self()).

{ok, GMapRes} = lasp:query(GMap1).

GMapRes.
% [{<<"key1">>,#{thing => 42,what => i_am_a_gmap_value}}]

[{_, GMapResVal}] = GMapRes.

GMapResVal.
% #{what => i_am_a_gmap_value}

We can use a gmap with an ivar as value type, an ivar is a register that can only be set once.

GMapIVarType = {state_gmap, [state_ivar]}.
GMapIVarVarName = <<"gmapivar">>.
GMapIVarVal1 = #{what => i_am_a_gmap_ivar_value}.
GMapIVarVal2 = #{what => i_am_a_gmap_ivar_update}.

{ok, {GMapIVar, _, _, _}} = lasp:declare({GMapIVarVarName, GMapIVarType}, GMapIVarType).
{ok, {GMapIVar1, _, _, _}} = lasp:update(GMapIVar, {apply, Key1,
                                            {set, GMapIVarVal1}},
                                            self()).
% try updating it, will throw an error (the value of GMapIVar1 will be lost)
{ok, {GMapIVar2, _, _, _}} = lasp:update(GMapIVar1, {apply, Key1,
                                            {set, GMapIVarVal2}},
                                            self()).

{ok, GMapIVarRes} = lasp:query(GMapIVar1).

GMapIVarRes.

[{_, GMapIVarResVal}] = GMapIVarRes.

GMapIVarResVal.
% #{what => i_am_a_gmap_ivar_value}

Types

My examples are all maps with some value type because that's the use case I'm most interested, here's a list of types and their operations:

"Scalars"

boolean

Operations:

true
Can be set to true once

Example:

BoolType = state_boolean.
BoolVarName = <<"boolvar">>.

{ok, {Bool, _, _, _}} = lasp:declare({BoolVarName, BoolType}, BoolType).
{ok, {Bool1, _, _, _}} = lasp:update(Bool, true, self()).

{ok, BoolRes} = lasp:query(Bool1).

BoolRes.
% true

ewflag:

Enable-Wins Flag CRDT

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

EWType = state_ewflag.
EWVarName = <<"ewvar">>.

{ok, {EW, _, _, _}} = lasp:declare({EWVarName, EWType}, EWType).
{ok, {EW1, _, _, _}} = lasp:update(EW, enable, self()).
{ok, EWRes1} = lasp:query(EW1).
{ok, {EW2, _, _, _}} = lasp:update(EW1, disable, self()).
{ok, EWRes2} = lasp:query(EW2).
{ok, {EW3, _, _, _}} = lasp:update(EW2, enable, self()).
{ok, EWRes3} = lasp:query(EW3).

EWRes1.
% true
EWRes2.
% false
EWRes3.
% true

dwflag:

Disable-Wins Flag CRDT

Follows the same strategy used in Enable-Wins Flag but,
instead of creating a new dot when enabling the flag,
we create a new dot when disabling it.

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

DWType = state_dwflag.
DWVarName = <<"dwvar">>.

{ok, {DW, _, _, _}} = lasp:declare({DWVarName, DWType}, DWType).
{ok, {DW1, _, _, _}} = lasp:update(DW, enable, self()).
{ok, DWRes1} = lasp:query(DW1).
{ok, {DW2, _, _, _}} = lasp:update(DW1, disable, self()).
{ok, DWRes2} = lasp:query(DW2).
{ok, {DW3, _, _, _}} = lasp:update(DW2, enable, self()).
{ok, DWRes3} = lasp:query(DW3).

DWRes1.
% true
DWRes2.
% false
DWRes3.
% true

GCounter CRDT: grow only counter

Operations:

increment
Increment the counter by 1

Example:

GCountType = state_gcounter.
GCountVarName = <<"gcountvar">>.

{ok, {GCount, _, _, _}} = lasp:declare({GCountVarName, GCountType}, GCountType).
{ok, {GCount1, _, _, _}} = lasp:update(GCount, increment, self()).
{ok, GCountRes1} = lasp:query(GCount1).
{ok, {GCount2, _, _, _}} = lasp:update(GCount1, increment, self()).
{ok, GCountRes2} = lasp:query(GCount2).
{ok, {GCount3, _, _, _}} = lasp:update(GCount2, increment, self()).
{ok, GCountRes3} = lasp:query(GCount3).

GCountRes1.
% 1
GCountRes2.
% 2
GCountRes3.
% 3

Some more I don't get completely what they could be used for:

Bounded Counter CRDT:

Modeled as a pair where the first component is a PNCounter and the second
component is a GMap.

This counter has sub counter for different ids (actors), each of which can't go below 0, with this you can model things like seats or some resource where you allocate counts to different parties (actors) and each can decrement their own count but not others, also each counter can't go below 0, if a given actor needs to decrement it has to move counts from other actor.

Operations:

{move, term()}
Moves permissions to decrement to another replica (if it has enough permissions)
increment
Increment counter, can always happen
decrement
Decrement counter, can happen when the replica has enough local increments, or has permissions received from other replicas

Example:

BCountType = state_bcounter.
BCountVarName = <<"bcountvar">>.
Actor1 = self().
Actor2 = <<"actor2-id">>.

{ok, {BCount, _, _, _}} = lasp:declare({BCountVarName, BCountType}, BCountType).
{ok, {BCount1, _, _, _}} = lasp:update(BCount, increment, Actor1).
{ok, BCountRes1} = lasp:query(BCount1).
{ok, {BCount2, _, _, _}} = lasp:update(BCount1, increment, Actor2).
{ok, BCountRes2} = lasp:query(BCount2).
{ok, {BCount3, _, _, _}} = lasp:update(BCount2, decrement, Actor1).
{ok, BCountRes3} = lasp:query(BCount3).

% here Actor1 has counter set to 0, can't go below 0, if it want's to
% decrement it has to move a 1 from another actor, Actor2 has 1 in its
% counter so we will move it and then decrement

BCountMoveFrom = Actor2.
BCountMoveTo = Actor1.
{ok, {BCount4, _, _, _}} = lasp:update(BCount3, {move, 1, BCountMoveTo}, BCountMoveFrom).

% now we can decrement from Actor1
{ok, {BCount5, _, _, _}} = lasp:update(BCount4, decrement, Actor1).
{ok, BCountRes5} = lasp:query(BCount5).

BCountRes1.
% 1
BCountRes2.
% 2
BCountRes3.
% 1

BCountRes5.
% 0

Max Int CRDT

Operations:

  • increment

Example:

MaxIntType = state_max_int.
MaxIntVarName = <<"maxintvar">>.

{ok, {MaxInt, _, _, _}} = lasp:declare({MaxIntVarName, MaxIntType}, MaxIntType).
{ok, {MaxInt1, _, _, _}} = lasp:update(MaxInt, increment, self()).
{ok, MaxIntRes1} = lasp:query(MaxInt1).
{ok, {MaxInt2, _, _, _}} = lasp:update(MaxInt1, increment, self()).
{ok, MaxIntRes2} = lasp:query(MaxInt2).
{ok, {MaxInt3, _, _, _}} = lasp:update(MaxInt2, increment, self()).
{ok, MaxIntRes3} = lasp:query(MaxInt3).

MaxIntRes1.
% 1
MaxIntRes2.
% 2
MaxIntRes3.
% 3

Lexicographic Counter

Operations:

  • increment
  • decrement

Example:

LCountType = state_lexcounter.
LCountVarName = <<"lexcountvar">>.

{ok, {LCount, _, _, _}} = lasp:declare({LCountVarName, LCountType}, LCountType).
{ok, {LCount1, _, _, _}} = lasp:update(LCount, increment, self()).
{ok, LCountRes1} = lasp:query(LCount1).
{ok, {LCount2, _, _, _}} = lasp:update(LCount1, increment, self()).
{ok, LCountRes2} = lasp:query(LCount2).
{ok, {LCount3, _, _, _}} = lasp:update(LCount2, decrement, self()).
{ok, LCountRes3} = lasp:query(LCount3).

LCountRes1.
% 1
LCountRes2.
% 2
LCountRes3.
% 1

PNCounter CRDT:

Counter that allows both increments and decrements.

Modeled as a dictionary where keys are replicas ids and values are pairs
where the first component is the number of increments and the second component
is the number of decrements.

An actor may only update its own entry in the dictionary.

The value of the counter is the sum of all first components minus the sum
of all second components.

Operations:

  • increment
  • decrement

Example:

PNCountType = state_pncounter.
PNCountVarName = <<"pncountvar">>.

{ok, {PNCount, _, _, _}} = lasp:declare({PNCountVarName, PNCountType}, PNCountType).
{ok, {PNCount1, _, _, _}} = lasp:update(PNCount, increment, self()).
{ok, PNCountRes1} = lasp:query(PNCount1).
{ok, {PNCount2, _, _, _}} = lasp:update(PNCount1, increment, self()).
{ok, PNCountRes2} = lasp:query(PNCount2).
{ok, {PNCount3, _, _, _}} = lasp:update(PNCount2, decrement, self()).
{ok, PNCountRes3} = lasp:query(PNCount3).

PNCountRes1.
% 1
PNCountRes2.
% 2
PNCountRes3.
% 1

"Registers"

LWWRegister:

We assume timestamp are unique, totally ordered and consistent
with causal order. We use integers as timestamps.
When using this, make sure you provide globally unique timestamps.

Operations:

{set, Timestamp, Value}
Set register to Value

Example:

LWWRegType = state_lwwregister.
LWWRegVarName = <<"lwwregister">>.

{ok, {LWWReg, _, _, _}} = lasp:declare({LWWRegVarName, LWWRegType}, LWWRegType).
{ok, {LWWReg1, _, _, _}} = lasp:update(LWWReg, {set, Timestamp(), foo}, self()).
{ok, LWWRegRes1} = lasp:query(LWWReg1).
{ok, {LWWReg2, _, _, _}} = lasp:update(LWWReg1, {set, Timestamp(), bar}, self()).
{ok, LWWRegRes2} = lasp:query(LWWReg2).
{ok, {LWWReg3, _, _, _}} = lasp:update(LWWReg2, {set, Timestamp(), baz}, self()).
{ok, LWWRegRes3} = lasp:query(LWWReg3).

LWWRegRes1.
% foo
LWWRegRes2.
% bar
LWWRegRes3.
% baz

IVar:

Single-assignment variable.
Write once register.

Operations:

{set, Value}
Set register to Value (only possible once)
IVarRegType = state_ivar.
IVarRegVarName = <<"ivar">>.

{ok, {IVarReg, _, _, _}} = lasp:declare({IVarRegVarName, IVarRegType}, IVarRegType).
{ok, {IVarReg1, _, _, _}} = lasp:update(IVarReg, {set, foo}, self()).
{ok, IVarRegRes1} = lasp:query(IVarReg1).

IVarRegRes1.
% foo

MVRegister:

Multi-Value Register CRDT.

Operations:

{set, Timestamp, Value}
the Timestamp will not be used. in order to have an unified API for all registers (since LWWRegister needs to receive a timestamp), the timestamp is also supplied here

Example:

MVRegType = state_mvregister.
MVRegVarName = <<"mvregister">>.

{ok, {MVReg, _, _, _}} = lasp:declare({MVRegVarName, MVRegType}, MVRegType).
{ok, {MVReg1, _, _, _}} = lasp:update(MVReg, {set, Timestamp(), foo}, self()).
{ok, MVRegRes1} = lasp:query(MVReg1).
{ok, {MVReg2, _, _, _}} = lasp:update(MVReg1, {set, nil, bar}, self()).
{ok, MVRegRes2} = lasp:query(MVReg2).
{ok, {MVReg3, _, _, _}} = lasp:update(MVReg2, {set, Timestamp(), baz}, self()).
{ok, MVRegRes3} = lasp:query(MVReg3).

sets:to_list(MVRegRes1).
% [foo]
sets:to_list(MVRegRes2).
% [bar]
sets:to_list(MVRegRes3).
% [baz]

Collections

AWMap CRDT:

Modeled as a dictionary where keys can be anything and the values are
causal-CRDTs.

Operations:

{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).
{rmv, Key}
Remove Key

Example:

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal1 = foo.
AwMapVal2 = bar.

{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, {AwMap2, _, _, _}} = lasp:update(AwMap1, {apply, Key1,
                                              {set, Timestamp(), AwMapVal2}},
                                      self()).

{ok, {AwMap3, _, _, _}} = lasp:update(AwMap2, {apply, Key2,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, AwMapRes3} = lasp:query(AwMap3).

{ok, {AwMap4, _, _, _}} = lasp:update(AwMap3, {rmv, Key2}, self()).

{ok, AwMapRes4} = lasp:query(AwMap4).

% before removing Key2
[{K, sets:to_list(V)} || {K, V} <- AwMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

[{K, sets:to_list(V)} || {K, V} <- AwMapRes4].
% [{<<"key1">>,[bar]}]

GMap CRDT:

Grow only map.
Modeled as a dictionary where keys can be anything and the
values are join-semilattices.
{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).

Example:

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal1 = foo.
GMapVal2 = bar.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).

{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, {GMap2, _, _, _}} = lasp:update(GMap1, {apply, Key1,
                                              {set, Timestamp(), GMapVal2}},
                                      self()).

{ok, {GMap3, _, _, _}} = lasp:update(GMap2, {apply, Key2,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, GMapRes3} = lasp:query(GMap3).
% {ok,[{<<"key1">>,bar},{<<"key2">>,foo}]}

MVMap:

Multi-Value Map CRDT.
MVMap = AWMap<MVRegister<V>>

Operations:

{set, Key, Value}
Set Key to Value

Example:

MVMapType = state_mvmap.
MVMapVarName = <<"mvmap">>.
MVMapVal1 = foo.
MVMapVal2 = bar.

{ok, {MVMap, _, _, _}} = lasp:declare({MVMapVarName, MVMapType}, MVMapType).

{ok, {MVMap1, _, _, _}} = lasp:update(MVMap, {set, Key1, MVMapVal1}, self()).
{ok, {MVMap2, _, _, _}} = lasp:update(MVMap1, {set, Key1, MVMapVal2}, self()).
{ok, {MVMap3, _, _, _}} = lasp:update(MVMap2, {set, Key2, MVMapVal1}, self()).

{ok, MVMapRes3} = lasp:query(MVMap3).

[{K, sets:to_list(V)} || {K, V} <- MVMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

Pair

Operations:

{fst, Value}
Mutates the first item in the pair
{snd, Value}
Mutates the second item in the pair

Example:

PairLeftType = state_lwwregister.
PairRightType = state_gcounter.
PairType = {state_pair, [PairLeftType, PairRightType]}.
PairVarName = <<"pair">>.
PairVal1 = foo.
PairVal2 = bar.

{ok, {Pair, _, _, _}} = lasp:declare({PairVarName, PairType}, PairType).

{ok, {Pair1, _, _, _}} = lasp:update(Pair, {fst, {set, Timestamp(), PairVal1}}, self()).
{ok, {Pair2, _, _, _}} = lasp:update(Pair1, {snd, increment}, self()).
{ok, {Pair3, _, _, _}} = lasp:update(Pair2, {fst, {set, Timestamp(), PairVal2}}, self()).
{ok, {Pair4, _, _, _}} = lasp:update(Pair3, {snd, increment}, self()).

{ok, PairRes4} = lasp:query(Pair4).

PairRes4.
% {bar,2}

ORSet CRDT:

Observed-Remove Set with tombstones
{add, Element}
Add Element to the Set
{add_by_token, token(), element()}
Add Element to the Set by Token (orset internally generates a unique token for the element, therefore, it's not deterministic, this allows it to be explicit, and therefore deterministic)
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

ORSetType = state_orset.
ORSetVarName = <<"orset">>.
ORSetVal1 = foo.
ORSetVal2 = bar.
ORSetVal3 = baz.
ORSetAllVals = [ORSetVal1, ORSetVal2, ORSetVal3].

{ok, {ORSet, _, _, _}} = lasp:declare({ORSetVarName, ORSetType}, ORSetType).

{ok, {ORSet1, _, _, _}} = lasp:update(ORSet, {add, ORSetVal1}, self()).
{ok, {ORSet2, _, _, _}} = lasp:update(ORSet1, {add, ORSetVal2}, self()).
% repeat value
{ok, {ORSet3, _, _, _}} = lasp:update(ORSet2, {add, ORSetVal1}, self()).
{ok, {ORSet4, _, _, _}} = lasp:update(ORSet3, {add, ORSetVal3}, self()).

{ok, ORSetRes4} = lasp:query(ORSet4).
sets:to_list(ORSetRes4).
% [bar,baz,foo]

{ok, {ORSet5, _, _, _}} = lasp:update(ORSet4, {rmv, ORSetVal3}, self()).
{ok, ORSetRes5} = lasp:query(ORSet5).
sets:to_list(ORSetRes5).
% [bar,foo]

{ok, {ORSet6, _, _, _}} = lasp:update(ORSet5, {rmv_all, ORSetAllVals}, self()).
{ok, ORSetRes6} = lasp:query(ORSet6).
sets:to_list(ORSetRes6).
% []

{ok, {ORSet7, _, _, _}} = lasp:update(ORSet6, {add_all, ORSetAllVals}, self()).
{ok, ORSetRes7} = lasp:query(ORSet7).
sets:to_list(ORSetRes7).
% [bar,baz,foo]

Add-Wins ORSet CRDT:

Observed-Remove Set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set
{filter, Function}
Filter elements in the set

Example:

AWSetType = state_awset.
AWSetVarName = <<"awset">>.
AWSetVal1 = foo.
AWSetVal2 = bar.
AWSetVal3 = baz.
AWSetAllVals = [AWSetVal1, AWSetVal2, AWSetVal3].

{ok, {AWSet, _, _, _}} = lasp:declare({AWSetVarName, AWSetType}, AWSetType).

{ok, {AWSet1, _, _, _}} = lasp:update(AWSet, {add, AWSetVal1}, self()).
{ok, {AWSet2, _, _, _}} = lasp:update(AWSet1, {add, AWSetVal2}, self()).
% repeat value
{ok, {AWSet3, _, _, _}} = lasp:update(AWSet2, {add, AWSetVal1}, self()).
{ok, {AWSet4, _, _, _}} = lasp:update(AWSet3, {add, AWSetVal3}, self()).

{ok, AWSetRes4} = lasp:query(AWSet4).
sets:to_list(AWSetRes4).
% [bar,baz,foo]

{ok, {AWSet5, _, _, _}} = lasp:update(AWSet4, {rmv, AWSetVal3}, self()).
{ok, AWSetRes5} = lasp:query(AWSet5).
sets:to_list(AWSetRes5).
% [bar,foo]

{ok, {AWSet6, _, _, _}} = lasp:update(AWSet5, {rmv_all, AWSetAllVals}, self()).
{ok, AWSetRes6} = lasp:query(AWSet6).
sets:to_list(AWSetRes6).
% []

{ok, {AWSet7, _, _, _}} = lasp:update(AWSet6, {add_all, AWSetAllVals}, self()).
{ok, AWSetRes7} = lasp:query(AWSet7).
sets:to_list(AWSetRes7).
% [bar,baz,foo]

Add-Wins Set CRDT with the provenance semiring:

add-wins set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

AWPSSetType = state_awset_ps.
AWPSSetVarName = <<"awset_ps">>.
AWPSSetVal1 = foo.
AWPSSetVal2 = bar.
AWPSSetVal3 = baz.
AWPSSetAllVals = [AWPSSetVal1, AWPSSetVal2, AWPSSetVal3].

{ok, {AWPSSet, _, _, _}} = lasp:declare({AWPSSetVarName, AWPSSetType}, AWPSSetType).

{ok, {AWPSSet1, _, _, _}} = lasp:update(AWPSSet, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet2, _, _, _}} = lasp:update(AWPSSet1, {add, AWPSSetVal2}, self()).
% repeat value
{ok, {AWPSSet3, _, _, _}} = lasp:update(AWPSSet2, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet4, _, _, _}} = lasp:update(AWPSSet3, {add, AWPSSetVal3}, self()).

{ok, AWPSSetRes4} = lasp:query(AWPSSet4).
sets:to_list(AWPSSetRes4).
% [bar,baz,foo]

{ok, {AWPSSet5, _, _, _}} = lasp:update(AWPSSet4, {rmv, AWPSSetVal3}, self()).
{ok, AWPSSetRes5} = lasp:query(AWPSSet5).
sets:to_list(AWPSSetRes5).
% [bar,foo]

{ok, {AWPSSet6, _, _, _}} = lasp:update(AWPSSet5, {rmv_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes6} = lasp:query(AWPSSet6).
sets:to_list(AWPSSetRes6).
% []

{ok, {AWPSSet7, _, _, _}} = lasp:update(AWPSSet6, {add_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes7} = lasp:query(AWPSSet7).
sets:to_list(AWPSSetRes7).
% [bar,baz,foo]

GSet CRDT:

Grow only set

Operations:

{add, Element}
Add Element to the Set

Example:

GSetType = state_gset.
GSetVarName = <<"gset">>.
GSetVal1 = foo.
GSetVal2 = bar.
GSetVal3 = baz.
GSetAllVals = [GSetVal1, GSetVal2, GSetVal3].

{ok, {GSet, _, _, _}} = lasp:declare({GSetVarName, GSetType}, GSetType).

{ok, {GSet1, _, _, _}} = lasp:update(GSet, {add, GSetVal1}, self()).
{ok, {GSet2, _, _, _}} = lasp:update(GSet1, {add, GSetVal2}, self()).
% repeat value
{ok, {GSet3, _, _, _}} = lasp:update(GSet2, {add, GSetVal1}, self()).
{ok, {GSet4, _, _, _}} = lasp:update(GSet3, {add, GSetVal3}, self()).

{ok, GSetRes4} = lasp:query(GSet4).
sets:to_list(GSetRes4).
% [bar,baz,foo]

2PSet CRDT:

Two-Phased Set

Once removed, elements cannot be added again.

Also, this is not an observed removed variant.

This means elements can be removed before being in the set.

Operations:

{add, Element}
Add Element to the Set
{rmv, Element}
Remove Element from the set

Example:

TPSetType = state_awset_ps.
TPSetVarName = <<"awset_ps">>.
TPSetVal1 = foo.
TPSetVal2 = bar.
TPSetVal3 = baz.
TPSetAllVals = [TPSetVal1, TPSetVal2, TPSetVal3].

{ok, {TPSet, _, _, _}} = lasp:declare({TPSetVarName, TPSetType}, TPSetType).

{ok, {TPSet1, _, _, _}} = lasp:update(TPSet, {add, TPSetVal1}, self()).
{ok, {TPSet2, _, _, _}} = lasp:update(TPSet1, {add, TPSetVal2}, self()).
% repeat value
{ok, {TPSet3, _, _, _}} = lasp:update(TPSet2, {add, TPSetVal1}, self()).
{ok, {TPSet4, _, _, _}} = lasp:update(TPSet3, {add, TPSetVal3}, self()).

{ok, TPSetRes4} = lasp:query(TPSet4).
sets:to_list(TPSetRes4).
% [bar,baz,foo]

{ok, {TPSet5, _, _, _}} = lasp:update(TPSet4, {rmv, TPSetVal3}, self()).
{ok, TPSetRes5} = lasp:query(TPSet5).
sets:to_list(TPSetRes5).
% [bar,foo]

Mariano Guerra: Playing with Lasp and CRDTs

For a long time I was looking for some time to play with Lasp:

Lasp is a suite of libraries aimed at providing a comprehensive programming
system for planetary scale Elixir and Erlang applications.

Yesterday I had some time and played a little bit with it, and thanks to the help from @cmeik and @vitorenesduarte I got some code running and understood some things.

Here is what I achieved until now, note that I'm learning this stuff so I may say many things that are wrong, I will try to get people to review it and update it with the corrections.

Some initial snippets of code we will reuse:

% clear all variable bindings
f().
Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

A map with any type as value

If we want to have data which can be represented as a dict/map/key-value mapping, there are multiple alternatives we can use in lasp, here we will explore some.

First we are going to use an AwMap CRDT as a map which is defined as:

A dictionary where keys can be anything and the values are causal-CRDTs.

Since values must be casual-CRDTs we can't use a bare erlang type as value nor a non causal-CRDT type like lwwregister, which we will use later with another map type.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

We can also use a GMap CRDT (Grow Only Map, keys can't be removed):

A dictionary where keys can be anything and the values are join-semilattices.

In this case we can use non causal-CRDTs as values, to make it a little different we are going to use another register type, a lwwregister

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal = #{what => i_am_a_gmap_value}.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).
{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                            {set, Timestamp(), GMapVal}},
                                            self()).

{ok, GMapRes} = lasp:query(GMap1).

GMapRes.
% [{<<"key1">>,#{thing => 42,what => i_am_a_gmap_value}}]

[{_, GMapResVal}] = GMapRes.

GMapResVal.
% #{what => i_am_a_gmap_value}

We can use a gmap with an ivar as value type, an ivar is a register that can only be set once.

GMapIVarType = {state_gmap, [state_ivar]}.
GMapIVarVarName = <<"gmapivar">>.
GMapIVarVal1 = #{what => i_am_a_gmap_ivar_value}.
GMapIVarVal2 = #{what => i_am_a_gmap_ivar_update}.

{ok, {GMapIVar, _, _, _}} = lasp:declare({GMapIVarVarName, GMapIVarType}, GMapIVarType).
{ok, {GMapIVar1, _, _, _}} = lasp:update(GMapIVar, {apply, Key1,
                                            {set, GMapIVarVal1}},
                                            self()).
% try updating it, will throw an error (the value of GMapIVar1 will be lost)
{ok, {GMapIVar2, _, _, _}} = lasp:update(GMapIVar1, {apply, Key1,
                                            {set, GMapIVarVal2}},
                                            self()).

{ok, GMapIVarRes} = lasp:query(GMapIVar1).

GMapIVarRes.

[{_, GMapIVarResVal}] = GMapIVarRes.

GMapIVarResVal.
% #{what => i_am_a_gmap_ivar_value}

Types

My examples are all maps with some value type because that's the use case I'm most interested, here's a list of types and their operations:

"Scalars"

boolean

Operations:

true
Can be set to true once

Example:

BoolType = state_boolean.
BoolVarName = <<"boolvar">>.

{ok, {Bool, _, _, _}} = lasp:declare({BoolVarName, BoolType}, BoolType).
{ok, {Bool1, _, _, _}} = lasp:update(Bool, true, self()).

{ok, BoolRes} = lasp:query(Bool1).

BoolRes.
% true

ewflag:

Enable-Wins Flag CRDT

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

EWType = state_ewflag.
EWVarName = <<"ewvar">>.

{ok, {EW, _, _, _}} = lasp:declare({EWVarName, EWType}, EWType).
{ok, {EW1, _, _, _}} = lasp:update(EW, enable, self()).
{ok, EWRes1} = lasp:query(EW1).
{ok, {EW2, _, _, _}} = lasp:update(EW1, disable, self()).
{ok, EWRes2} = lasp:query(EW2).
{ok, {EW3, _, _, _}} = lasp:update(EW2, enable, self()).
{ok, EWRes3} = lasp:query(EW3).

EWRes1.
% true
EWRes2.
% false
EWRes3.
% true

dwflag:

Disable-Wins Flag CRDT

Follows the same strategy used in Enable-Wins Flag but,
instead of creating a new dot when enabling the flag,
we create a new dot when disabling it.

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

DWType = state_dwflag.
DWVarName = <<"dwvar">>.

{ok, {DW, _, _, _}} = lasp:declare({DWVarName, DWType}, DWType).
{ok, {DW1, _, _, _}} = lasp:update(DW, enable, self()).
{ok, DWRes1} = lasp:query(DW1).
{ok, {DW2, _, _, _}} = lasp:update(DW1, disable, self()).
{ok, DWRes2} = lasp:query(DW2).
{ok, {DW3, _, _, _}} = lasp:update(DW2, enable, self()).
{ok, DWRes3} = lasp:query(DW3).

DWRes1.
% true
DWRes2.
% false
DWRes3.
% true

GCounter CRDT: grow only counter

Operations:

increment
Increment the counter by 1

Example:

GCountType = state_gcounter.
GCountVarName = <<"gcountvar">>.

{ok, {GCount, _, _, _}} = lasp:declare({GCountVarName, GCountType}, GCountType).
{ok, {GCount1, _, _, _}} = lasp:update(GCount, increment, self()).
{ok, GCountRes1} = lasp:query(GCount1).
{ok, {GCount2, _, _, _}} = lasp:update(GCount1, increment, self()).
{ok, GCountRes2} = lasp:query(GCount2).
{ok, {GCount3, _, _, _}} = lasp:update(GCount2, increment, self()).
{ok, GCountRes3} = lasp:query(GCount3).

GCountRes1.
% 1
GCountRes2.
% 2
GCountRes3.
% 3

Some more I don't get completely what they could be used for:

Bounded Counter CRDT:

Modeled as a pair where the first component is a PNCounter and the second
component is a GMap.

This counter has sub counter for different ids (actors), each of which can't go below 0, with this you can model things like seats or some resource where you allocate counts to different parties (actors) and each can decrement their own count but not others, also each counter can't go below 0, if a given actor needs to decrement it has to move counts from other actor.

Operations:

{move, term()}
Moves permissions to decrement to another replica (if it has enough permissions)
increment
Increment counter, can always happen
decrement
Decrement counter, can happen when the replica has enough local increments, or has permissions received from other replicas

Example:

BCountType = state_bcounter.
BCountVarName = <<"bcountvar">>.
Actor1 = self().
Actor2 = <<"actor2-id">>.

{ok, {BCount, _, _, _}} = lasp:declare({BCountVarName, BCountType}, BCountType).
{ok, {BCount1, _, _, _}} = lasp:update(BCount, increment, Actor1).
{ok, BCountRes1} = lasp:query(BCount1).
{ok, {BCount2, _, _, _}} = lasp:update(BCount1, increment, Actor2).
{ok, BCountRes2} = lasp:query(BCount2).
{ok, {BCount3, _, _, _}} = lasp:update(BCount2, decrement, Actor1).
{ok, BCountRes3} = lasp:query(BCount3).

% here Actor1 has counter set to 0, can't go below 0, if it want's to
% decrement it has to move a 1 from another actor, Actor2 has 1 in its
% counter so we will move it and then decrement

BCountMoveFrom = Actor2.
BCountMoveTo = Actor1.
{ok, {BCount4, _, _, _}} = lasp:update(BCount3, {move, 1, BCountMoveTo}, BCountMoveFrom).

% now we can decrement from Actor1
{ok, {BCount5, _, _, _}} = lasp:update(BCount4, decrement, Actor1).
{ok, BCountRes5} = lasp:query(BCount5).

BCountRes1.
% 1
BCountRes2.
% 2
BCountRes3.
% 1

BCountRes5.
% 0

Max Int CRDT

Operations:

  • increment

Example:

MaxIntType = state_max_int.
MaxIntVarName = <<"maxintvar">>.

{ok, {MaxInt, _, _, _}} = lasp:declare({MaxIntVarName, MaxIntType}, MaxIntType).
{ok, {MaxInt1, _, _, _}} = lasp:update(MaxInt, increment, self()).
{ok, MaxIntRes1} = lasp:query(MaxInt1).
{ok, {MaxInt2, _, _, _}} = lasp:update(MaxInt1, increment, self()).
{ok, MaxIntRes2} = lasp:query(MaxInt2).
{ok, {MaxInt3, _, _, _}} = lasp:update(MaxInt2, increment, self()).
{ok, MaxIntRes3} = lasp:query(MaxInt3).

MaxIntRes1.
% 1
MaxIntRes2.
% 2
MaxIntRes3.
% 3

Lexicographic Counter

Operations:

  • increment
  • decrement

Example:

LCountType = state_lexcounter.
LCountVarName = <<"lexcountvar">>.

{ok, {LCount, _, _, _}} = lasp:declare({LCountVarName, LCountType}, LCountType).
{ok, {LCount1, _, _, _}} = lasp:update(LCount, increment, self()).
{ok, LCountRes1} = lasp:query(LCount1).
{ok, {LCount2, _, _, _}} = lasp:update(LCount1, increment, self()).
{ok, LCountRes2} = lasp:query(LCount2).
{ok, {LCount3, _, _, _}} = lasp:update(LCount2, decrement, self()).
{ok, LCountRes3} = lasp:query(LCount3).

LCountRes1.
% 1
LCountRes2.
% 2
LCountRes3.
% 1

PNCounter CRDT:

Counter that allows both increments and decrements.

Modeled as a dictionary where keys are replicas ids and values are pairs
where the first component is the number of increments and the second component
is the number of decrements.

An actor may only update its own entry in the dictionary.

The value of the counter is the sum of all first components minus the sum
of all second components.

Operations:

  • increment
  • decrement

Example:

PNCountType = state_pncounter.
PNCountVarName = <<"pncountvar">>.

{ok, {PNCount, _, _, _}} = lasp:declare({PNCountVarName, PNCountType}, PNCountType).
{ok, {PNCount1, _, _, _}} = lasp:update(PNCount, increment, self()).
{ok, PNCountRes1} = lasp:query(PNCount1).
{ok, {PNCount2, _, _, _}} = lasp:update(PNCount1, increment, self()).
{ok, PNCountRes2} = lasp:query(PNCount2).
{ok, {PNCount3, _, _, _}} = lasp:update(PNCount2, decrement, self()).
{ok, PNCountRes3} = lasp:query(PNCount3).

PNCountRes1.
% 1
PNCountRes2.
% 2
PNCountRes3.
% 1

"Registers"

LWWRegister:

We assume timestamp are unique, totally ordered and consistent
with causal order. We use integers as timestamps.
When using this, make sure you provide globally unique timestamps.

Operations:

{set, Timestamp, Value}
Set register to Value

Example:

LWWRegType = state_lwwregister.
LWWRegVarName = <<"lwwregister">>.

{ok, {LWWReg, _, _, _}} = lasp:declare({LWWRegVarName, LWWRegType}, LWWRegType).
{ok, {LWWReg1, _, _, _}} = lasp:update(LWWReg, {set, Timestamp(), foo}, self()).
{ok, LWWRegRes1} = lasp:query(LWWReg1).
{ok, {LWWReg2, _, _, _}} = lasp:update(LWWReg1, {set, Timestamp(), bar}, self()).
{ok, LWWRegRes2} = lasp:query(LWWReg2).
{ok, {LWWReg3, _, _, _}} = lasp:update(LWWReg2, {set, Timestamp(), baz}, self()).
{ok, LWWRegRes3} = lasp:query(LWWReg3).

LWWRegRes1.
% foo
LWWRegRes2.
% bar
LWWRegRes3.
% baz

IVar:

Single-assignment variable.
Write once register.

Operations:

{set, Value}
Set register to Value (only possible once)
IVarRegType = state_ivar.
IVarRegVarName = <<"ivar">>.

{ok, {IVarReg, _, _, _}} = lasp:declare({IVarRegVarName, IVarRegType}, IVarRegType).
{ok, {IVarReg1, _, _, _}} = lasp:update(IVarReg, {set, foo}, self()).
{ok, IVarRegRes1} = lasp:query(IVarReg1).

IVarRegRes1.
% foo

MVRegister:

Multi-Value Register CRDT.

Operations:

{set, Timestamp, Value}
the Timestamp will not be used. in order to have an unified API for all registers (since LWWRegister needs to receive a timestamp), the timestamp is also supplied here

Example:

MVRegType = state_mvregister.
MVRegVarName = <<"mvregister">>.

{ok, {MVReg, _, _, _}} = lasp:declare({MVRegVarName, MVRegType}, MVRegType).
{ok, {MVReg1, _, _, _}} = lasp:update(MVReg, {set, Timestamp(), foo}, self()).
{ok, MVRegRes1} = lasp:query(MVReg1).
{ok, {MVReg2, _, _, _}} = lasp:update(MVReg1, {set, nil, bar}, self()).
{ok, MVRegRes2} = lasp:query(MVReg2).
{ok, {MVReg3, _, _, _}} = lasp:update(MVReg2, {set, Timestamp(), baz}, self()).
{ok, MVRegRes3} = lasp:query(MVReg3).

sets:to_list(MVRegRes1).
% [foo]
sets:to_list(MVRegRes2).
% [bar]
sets:to_list(MVRegRes3).
% [baz]

Collections

AWMap CRDT:

Modeled as a dictionary where keys can be anything and the values are
causal-CRDTs.

Operations:

{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).
{rmv, Key}
Remove Key

Example:

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal1 = foo.
AwMapVal2 = bar.

{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, {AwMap2, _, _, _}} = lasp:update(AwMap1, {apply, Key1,
                                              {set, Timestamp(), AwMapVal2}},
                                      self()).

{ok, {AwMap3, _, _, _}} = lasp:update(AwMap2, {apply, Key2,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, AwMapRes3} = lasp:query(AwMap3).

{ok, {AwMap4, _, _, _}} = lasp:update(AwMap3, {rmv, Key2}, self()).

{ok, AwMapRes4} = lasp:query(AwMap4).

% before removing Key2
[{K, sets:to_list(V)} || {K, V} <- AwMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

[{K, sets:to_list(V)} || {K, V} <- AwMapRes4].
% [{<<"key1">>,[bar]}]

GMap CRDT:

Grow only map.
Modeled as a dictionary where keys can be anything and the
values are join-semilattices.
{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).

Example:

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal1 = foo.
GMapVal2 = bar.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).

{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, {GMap2, _, _, _}} = lasp:update(GMap1, {apply, Key1,
                                              {set, Timestamp(), GMapVal2}},
                                      self()).

{ok, {GMap3, _, _, _}} = lasp:update(GMap2, {apply, Key2,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, GMapRes3} = lasp:query(GMap3).
% {ok,[{<<"key1">>,bar},{<<"key2">>,foo}]}

MVMap:

Multi-Value Map CRDT.
MVMap = AWMap<MVRegister<V>>

Operations:

{set, Key, Value}
Set Key to Value

Example:

MVMapType = state_mvmap.
MVMapVarName = <<"mvmap">>.
MVMapVal1 = foo.
MVMapVal2 = bar.

{ok, {MVMap, _, _, _}} = lasp:declare({MVMapVarName, MVMapType}, MVMapType).

{ok, {MVMap1, _, _, _}} = lasp:update(MVMap, {set, Key1, MVMapVal1}, self()).
{ok, {MVMap2, _, _, _}} = lasp:update(MVMap1, {set, Key1, MVMapVal2}, self()).
{ok, {MVMap3, _, _, _}} = lasp:update(MVMap2, {set, Key2, MVMapVal1}, self()).

{ok, MVMapRes3} = lasp:query(MVMap3).

[{K, sets:to_list(V)} || {K, V} <- MVMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

Pair

Operations:

{fst, Value}
Mutates the first item in the pair
{snd, Value}
Mutates the second item in the pair

Example:

PairLeftType = state_lwwregister.
PairRightType = state_gcounter.
PairType = {state_pair, [PairLeftType, PairRightType]}.
PairVarName = <<"pair">>.
PairVal1 = foo.
PairVal2 = bar.

{ok, {Pair, _, _, _}} = lasp:declare({PairVarName, PairType}, PairType).

{ok, {Pair1, _, _, _}} = lasp:update(Pair, {fst, {set, Timestamp(), PairVal1}}, self()).
{ok, {Pair2, _, _, _}} = lasp:update(Pair1, {snd, increment}, self()).
{ok, {Pair3, _, _, _}} = lasp:update(Pair2, {fst, {set, Timestamp(), PairVal2}}, self()).
{ok, {Pair4, _, _, _}} = lasp:update(Pair3, {snd, increment}, self()).

{ok, PairRes4} = lasp:query(Pair4).

PairRes4.
% {bar,2}

ORSet CRDT:

Observed-Remove Set with tombstones
{add, Element}
Add Element to the Set
{add_by_token, token(), element()}
Add Element to the Set by Token (orset internally generates a unique token for the element, therefore, it's not deterministic, this allows it to be explicit, and therefore deterministic)
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

ORSetType = state_orset.
ORSetVarName = <<"orset">>.
ORSetVal1 = foo.
ORSetVal2 = bar.
ORSetVal3 = baz.
ORSetAllVals = [ORSetVal1, ORSetVal2, ORSetVal3].

{ok, {ORSet, _, _, _}} = lasp:declare({ORSetVarName, ORSetType}, ORSetType).

{ok, {ORSet1, _, _, _}} = lasp:update(ORSet, {add, ORSetVal1}, self()).
{ok, {ORSet2, _, _, _}} = lasp:update(ORSet1, {add, ORSetVal2}, self()).
% repeat value
{ok, {ORSet3, _, _, _}} = lasp:update(ORSet2, {add, ORSetVal1}, self()).
{ok, {ORSet4, _, _, _}} = lasp:update(ORSet3, {add, ORSetVal3}, self()).

{ok, ORSetRes4} = lasp:query(ORSet4).
sets:to_list(ORSetRes4).
% [bar,baz,foo]

{ok, {ORSet5, _, _, _}} = lasp:update(ORSet4, {rmv, ORSetVal3}, self()).
{ok, ORSetRes5} = lasp:query(ORSet5).
sets:to_list(ORSetRes5).
% [bar,foo]

{ok, {ORSet6, _, _, _}} = lasp:update(ORSet5, {rmv_all, ORSetAllVals}, self()).
{ok, ORSetRes6} = lasp:query(ORSet6).
sets:to_list(ORSetRes6).
% []

{ok, {ORSet7, _, _, _}} = lasp:update(ORSet6, {add_all, ORSetAllVals}, self()).
{ok, ORSetRes7} = lasp:query(ORSet7).
sets:to_list(ORSetRes7).
% [bar,baz,foo]

Add-Wins ORSet CRDT:

Observed-Remove Set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set
{filter, Function}
Filter elements in the set

Example:

AWSetType = state_awset.
AWSetVarName = <<"awset">>.
AWSetVal1 = foo.
AWSetVal2 = bar.
AWSetVal3 = baz.
AWSetAllVals = [AWSetVal1, AWSetVal2, AWSetVal3].

{ok, {AWSet, _, _, _}} = lasp:declare({AWSetVarName, AWSetType}, AWSetType).

{ok, {AWSet1, _, _, _}} = lasp:update(AWSet, {add, AWSetVal1}, self()).
{ok, {AWSet2, _, _, _}} = lasp:update(AWSet1, {add, AWSetVal2}, self()).
% repeat value
{ok, {AWSet3, _, _, _}} = lasp:update(AWSet2, {add, AWSetVal1}, self()).
{ok, {AWSet4, _, _, _}} = lasp:update(AWSet3, {add, AWSetVal3}, self()).

{ok, AWSetRes4} = lasp:query(AWSet4).
sets:to_list(AWSetRes4).
% [bar,baz,foo]

{ok, {AWSet5, _, _, _}} = lasp:update(AWSet4, {rmv, AWSetVal3}, self()).
{ok, AWSetRes5} = lasp:query(AWSet5).
sets:to_list(AWSetRes5).
% [bar,foo]

{ok, {AWSet6, _, _, _}} = lasp:update(AWSet5, {rmv_all, AWSetAllVals}, self()).
{ok, AWSetRes6} = lasp:query(AWSet6).
sets:to_list(AWSetRes6).
% []

{ok, {AWSet7, _, _, _}} = lasp:update(AWSet6, {add_all, AWSetAllVals}, self()).
{ok, AWSetRes7} = lasp:query(AWSet7).
sets:to_list(AWSetRes7).
% [bar,baz,foo]

Add-Wins Set CRDT with the provenance semiring:

add-wins set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

AWPSSetType = state_awset_ps.
AWPSSetVarName = <<"awset_ps">>.
AWPSSetVal1 = foo.
AWPSSetVal2 = bar.
AWPSSetVal3 = baz.
AWPSSetAllVals = [AWPSSetVal1, AWPSSetVal2, AWPSSetVal3].

{ok, {AWPSSet, _, _, _}} = lasp:declare({AWPSSetVarName, AWPSSetType}, AWPSSetType).

{ok, {AWPSSet1, _, _, _}} = lasp:update(AWPSSet, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet2, _, _, _}} = lasp:update(AWPSSet1, {add, AWPSSetVal2}, self()).
% repeat value
{ok, {AWPSSet3, _, _, _}} = lasp:update(AWPSSet2, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet4, _, _, _}} = lasp:update(AWPSSet3, {add, AWPSSetVal3}, self()).

{ok, AWPSSetRes4} = lasp:query(AWPSSet4).
sets:to_list(AWPSSetRes4).
% [bar,baz,foo]

{ok, {AWPSSet5, _, _, _}} = lasp:update(AWPSSet4, {rmv, AWPSSetVal3}, self()).
{ok, AWPSSetRes5} = lasp:query(AWPSSet5).
sets:to_list(AWPSSetRes5).
% [bar,foo]

{ok, {AWPSSet6, _, _, _}} = lasp:update(AWPSSet5, {rmv_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes6} = lasp:query(AWPSSet6).
sets:to_list(AWPSSetRes6).
% []

{ok, {AWPSSet7, _, _, _}} = lasp:update(AWPSSet6, {add_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes7} = lasp:query(AWPSSet7).
sets:to_list(AWPSSetRes7).
% [bar,baz,foo]

GSet CRDT:

Grow only set

Operations:

{add, Element}
Add Element to the Set

Example:

GSetType = state_gset.
GSetVarName = <<"gset">>.
GSetVal1 = foo.
GSetVal2 = bar.
GSetVal3 = baz.
GSetAllVals = [GSetVal1, GSetVal2, GSetVal3].

{ok, {GSet, _, _, _}} = lasp:declare({GSetVarName, GSetType}, GSetType).

{ok, {GSet1, _, _, _}} = lasp:update(GSet, {add, GSetVal1}, self()).
{ok, {GSet2, _, _, _}} = lasp:update(GSet1, {add, GSetVal2}, self()).
% repeat value
{ok, {GSet3, _, _, _}} = lasp:update(GSet2, {add, GSetVal1}, self()).
{ok, {GSet4, _, _, _}} = lasp:update(GSet3, {add, GSetVal3}, self()).

{ok, GSetRes4} = lasp:query(GSet4).
sets:to_list(GSetRes4).
% [bar,baz,foo]

2PSet CRDT:

Two-Phased Set

Once removed, elements cannot be added again.

Also, this is not an observed removed variant.

This means elements can be removed before being in the set.

Operations:

{add, Element}
Add Element to the Set
{rmv, Element}
Remove Element from the set

Example:

TPSetType = state_awset_ps.
TPSetVarName = <<"awset_ps">>.
TPSetVal1 = foo.
TPSetVal2 = bar.
TPSetVal3 = baz.
TPSetAllVals = [TPSetVal1, TPSetVal2, TPSetVal3].

{ok, {TPSet, _, _, _}} = lasp:declare({TPSetVarName, TPSetType}, TPSetType).

{ok, {TPSet1, _, _, _}} = lasp:update(TPSet, {add, TPSetVal1}, self()).
{ok, {TPSet2, _, _, _}} = lasp:update(TPSet1, {add, TPSetVal2}, self()).
% repeat value
{ok, {TPSet3, _, _, _}} = lasp:update(TPSet2, {add, TPSetVal1}, self()).
{ok, {TPSet4, _, _, _}} = lasp:update(TPSet3, {add, TPSetVal3}, self()).

{ok, TPSetRes4} = lasp:query(TPSet4).
sets:to_list(TPSetRes4).
% [bar,baz,foo]

{ok, {TPSet5, _, _, _}} = lasp:update(TPSet4, {rmv, TPSetVal3}, self()).
{ok, TPSetRes5} = lasp:query(TPSet5).
sets:to_list(TPSetRes5).
% [bar,foo]

Facundo Batista: A todo tren con la Asociación Civil

Durante el año terminó de salir y empezamos a hacer girar la Asociación Civil Python Argentina.

Hay que hacer TANTAS cosas!! Mucho papel, mucha firma, mucho trámite por todos lados...

Lo bueno es que ya la podemos disfrutar, ya la podemos aprovechar. La PyCon del mes pasado se hizo "legalmente" dentro de la Asociación. O sea, aunque los organizadores fueron realmente Tutuca y Gaucho, a nivel legal/contabilidad la responsable es la asociación... Leandro y yo tuvimos que laburar un montón también, esperemos que cuando estén los mecanismos más afilados, sea menos carga.

Es que muchas cosas que hicimos (para la PyCon o para la Asociación en general) eran "por primera vez", y eso nos tomó bastante trabajo a varios de la Comisión Directiva... definir cómo íbamos a manejar una caja chica con el Contador, estructurar los niveles de socios beneficiarios (para empresas o instituciones), ver cómo se iban a manejar las becas, analizar la mejor manera de reunir los datos legales de cada persona siendo lo más laxos posibles, hasta armar un circuito de registración que le permita a la gente ser socios de PyAr con el menos laburo posible.

Pero se fue haciendo. Noviembre estuvo cargadísimo por la PyCon en sí, y luego de la conferencia estuvimos a full procesando los formularios de socios que se llenaron esos días, más todo el laburo a nivel pagos y tesorería para terminar de cerrar todo. Pero tambien hicimos la asamblea de renovación de autoridades entre los que éramos socios en ese momento, y empezamos a comunicarnos por las redes (contando que Onapsis era la primer socia benefactora, agradeciendo a GCoop por sacarnos y cuidarnos los dominios .org.ar para la comunidad, etc.)

El broche de oro de diciembre fue el jueves pasado cuando nos juntamos los que pudimos de la Comisión Directiva en Devecoop, la cooperativa de desarrollo de software que gentilmente nos presta el lugar donde están ellos para tener la sede legal de la Asociación... nos caimos en su lugar de trabajo y les agradecimos de la mejor manera: con comida y bebida :p

La mejor manera de decir gracias :)

Los próximos pasos son definir cómo vamos a manejar el dinero del próximo PyCamp, empezar a procesar los registros de la gente que llenó el formulario online para hacerse socia/os, y hacer una reunión de Comisión Directiva para incorporara la/os nueva/os socia/os que ya tenemos!

Facundo Batista: A todo tren con la Asociación Civil


Durante el año terminó de salir y empezamos a hacer girar la Asociación Civil Python Argentina.

Hay que hacer TANTAS cosas!! Mucho papel, mucha firma, mucho trámite por todos lados...

Lo bueno es que ya la podemos disfrutar, ya la podemos aprovechar. La PyCon del mes pasado se hizo "legalmente" dentro de la Asociación. O sea, aunque los organizadores fueron realmente Tutuca y Gaucho, a nivel legal/contabilidad la responsable es la asociación... Leandro y yo tuvimos que laburar un montón también, esperemos que cuando estén los mecanismos más afilados, sea menos carga.

Es que muchas cosas que hicimos (para la PyCon o para la Asociación en general) eran "por primera vez", y eso nos tomó bastante trabajo a varios de la Comisión Directiva... definir cómo íbamos a manejar una caja chica con el Contador, estructurar los niveles de socios beneficiarios (para empresas o instituciones), ver cómo se iban a manejar las becas, analizar la mejor manera de reunir los datos legales de cada persona siendo lo más laxos posibles, hasta armar un circuito de registración que le permita a la gente ser socios de PyAr con el menos laburo posible.

Pero se fue haciendo. Noviembre estuvo cargadísimo por la PyCon en sí, y luego de la conferencia estuvimos a full procesando los formularios de socios que se llenaron esos días, más todo el laburo a nivel pagos y tesorería para terminar de cerrar todo. Pero tambien hicimos la asamblea de renovación de autoridades entre los que éramos socios en ese momento, y empezamos a comunicarnos por las redes (contando que Onapsis era la primer socia benefactora, agradeciendo a GCoop por sacarnos y cuidarnos los dominios .org.ar para la comunidad, etc.)

El broche de oro de diciembre fue el jueves pasado cuando nos juntamos los que pudimos de la Comisión Directiva en Devecoop, la cooperativa de desarrollo de software que gentilmente nos presta el lugar donde están ellos para tener la sede legal de la Asociación... nos caimos en su lugar de trabajo y les agradecimos de la mejor manera: con comida y bebida :p

La mejor manera de decir gracias :)

Los próximos pasos son definir cómo vamos a manejar el dinero del próximo PyCamp, empezar a procesar los registros de la gente que llenó el formulario online para hacerse socia/os, y hacer una reunión de Comisión Directiva para incorporara la/os nueva/os socia/os que ya tenemos!

Damián Avila: Binder + Nikola + Jupyter + Github = Blogging resourceless

You are in vacation time but you want to blog something nice to share with your friends.

And your blog is powered by Nikola.

And you are a Jupyter Notebook user.

But you don't have your laptop with you, because you are on vacation time, remember? ;-)

But you still have your phone and some connectivity.

What do you think if I say that you have a complete workflow to write your blogpost, build the site and deploy it just using your phone? But without using computational resources from it (that would be also interesting, btw).

Don't you believe me? Just read this post and I will show you how you can make this possible... and fun!

Read more… (16 min remaining to read)