Facundo Batista: Vacaciones en familia, playa y sierras


Durante el veranito nos tomamos unos días con la familia para pasear por distintos lados.

Sí, hace un montón. Es que nos tardamos en procesar las fotos... no escala; para las próximas vacaciones largas, creo que voy a poner a la familia a revisar fotos durante las vacaciones mismas, sino llegamos y tenemos 1238941 fotos, y revisando un cachito durante la cena en los sucesivos días, no terminamos más... al punto incluso que todavía no están listas! Pero bueno, ya no espero más, saco este post, las fotos vendrán después...


Playa

Entre Navidad y Año Nuevo nos fuimos en carpa a Mar Azul. Apuntábamos al camping al que habíamos ido cuando Felipe era pequeño, pero ese ya no existe más y el único que quedó es el Camping de Los Ingenieros, que estuvo bastante bien (aunque el baño que teníamos cerca nunca tuvo agua caliente, y en general le faltan mesas).

La pasamos muy bien, aunque hizo mucho calor y en algunos momentos era agobiante (apenas sobrevivíamos bajo unas sombritas poco densas). Claro, está el mar para meterse, pero hay toda una franja horaria en la que no íbamos a llevar a los chicos a la playa para que se calcinen bajo el sol.

Preparándonos para ir a la playa

Almorzando en el camping

Obviamente fuimos en carpa, y nos dimos cuenta que ya es un poco chica para los cuatro. Eso, sumado a que se quebró una vara de la estructura y rasgó el techo, nos estaría forzando a comprar una carpa nueva más grande, tenemos que ver qué hay...

La playa de Mar Azul es muy linda, bastante ancha, y aunque en algunos momentos había bastante gente, nunca estuvimos amuchados (pero a veces se complicaba armar una cancha de tejo grande...).

Atardecer en la playa, el día que llegamos

Jugando al tejo con Felipe


Al Este de las Sierras

En Enero nos fuimos a Córdoba un par de semanas (bien en contra mano de "findes de semana" o "quincenas", para no desquiciarnos al viajar). Estuvimos unos días en El Espinillo, y después cruzamos las sierras y nos quedamos otros días en La Población.

Un minuto de tranquilidad :)

El primer lugar, en El Espinillo, era en un complejo de tres cabañas con una pileta compartida. La pileta estuvo muy bien, ya que esos días hizo mucho calor entonces los chicos se metieron varias veces y la disfrutaron un montón. Con Moni no nos metimos tanto pero aprovechamos para tomar sol. Incluso, como a veces la gente de las otras cabañas se iban temprano, tuve un par de días en los que pude tomar sol en bolas lo más tranquilo.

Además de la pileta, también hicimos "agua" en varios ríos cercanos (Del Espinillo, Del Medio, y Los Reartes)... algunos con más sombra, otros con más pastito alrededor, algunos con el piso de piedra, en otros teníamos playita de arena... muchas combinaciones, lo importante era que nos podíamos meter un rato al agua, tomar sol, disfrutar el verde, pasarla bien.

Los chicos disfrutando la pileta a full

En un arroyo de por ahíEn un arroyo de por ahí

Uno de los días fuimos a visitar la Villa General Belgrano. Estuvo bueno porque visitamos el Museo Politemático Castillo Romano y paseamos un rato, pero el restaurant al qué íbamos siempre ya no es lo que era (la cerveza artesanal como aguada, la comida meh, la atención dejó que desear) y en general es una ciudad "demasiado turística" a la que no llegamos a disfrutar.

Un paseo que sí nos encantó de esos días fue el del Parque Recreativo de La Serranita, un lugar bárbaro con mil juegos y actividades. Hicimos minigolf, arquería, futpool, jugamos al tejo, carrera de bolitas, nos tiramos por un tobogán gigante, recorrimos un laberinto, hicimos actividades de treparse y equilibrio, adivinanzas y un montón de actividades más... estuvimos como seis horas ahí adentro (sin contar el almuerzo), súper recomendado para ir con la familia!

Practicando el mini-golf

Felipe atacando un helado con ganas

Otra visita que disfrutamos fue a Nico y Jesi en La Quintana, donde pasamos una tarde bárbara charlando y tomando mate. Estuvo complicado llegar porque ya en viaje nos agarró una lluvia torrencial, y cuando llegamos a La Quintana los caminos estaban complicados... no sólo por el barro, sino también porque la lluvia había sido tanta que cada calle era un pequeño arroyo.

Encima tuve la suerte de presenciar (y casi participar!) en la instalación del primer prototipo funcional del LibreRouter en su antena, :D

Visitando La Quintana

En el monumento a Mercedes Sosa


Las Sierras, lado Oeste

A mitad de las vacaciones cambiamos de lugar. Hicimos un pequeño viaje cruzando las sierras y nos fuimos hasta el segundo lugar que teníamos alquilado.

Era una casita mediana, súper equipada, en el medio de un inmenso terreno, cerca de una casa bien grande que era donde vivía la dueña. El paisaje era muy lindo, la verdad, todo muy cuidado, verde por doquier, hermoso.

Los peques, contemplando el paisaje

Foto grupal en una de las construcciones de la Villa

Los días de la segunda fase estuvieron en general más frescos, lo que nos llevó a realizar menos actividades "acuáticas", pero no por eso aprovechamos menos las vacaciones: además de un par de veces en la pileta, y algunas en ríos/arroyos, "pancheamos" a full y paseamos algo.

A nivel de chapoteo, nos fuimos una vez a un Balneario en Paso de las Tropas, cerca de Nono, con la idea de almorzar, pasar la tarde, y luego apuntar para el Museo Rocsen, pero se nos fueron pasando las horas y después ya estábamos muy cansados, así que nos volvimos, dejamos el museo para la próxima. También encontramos una zona linda para aprovechar un arroyo cerca de donde nos hospedábamos (recorrimos un caminito con el auto hasta que no se pudo más, y luego un tramo a pie siguiendo el arroyo, hasta llegar a una linda zona para pasar la tarde). La verdad es que la primera vez nos quedamos en un punto con una pequeña cascadita, pero la segunda vez como cuando llegamos ya había gente, seguimos caminando un poco más y encontramos un lugar mejor :)

En un arroyito con unas cascadas hermosas

Preparando el almuerzo al costado del agua

Con respecto a los paseos, fuimos varias veces a la Plaza de San Javier, unos kilómetros al norte. No sólo para las compras de supermercado y eso, sino porque había feria artesanal todos los días, entonces visitamos varias veces, e incluso caimos de casualidad en una obra de títeres que estuvo muy buena. Encima era una zona donde había varios lugares lindos para comer, e incluso un pequeño parque cervecero/heladería, al cual fuimos más de una vez :)

También hicimos varios kilómetros para el sur y visitamos la fábrica de aceite de oliva Sierra Pura. Nos encantó. Te reciben, te muestran los árboles contándote las diferencias entre los tipos de olivo que hay, luego te muestran las máquinas con las que producen, y te explican todo el proceso, contestándote las preguntas que se te ocurran. Como no hay turnos ni horarios, llegás, y al rato ya te hacen el paseo, sean 3, 7, o 15 personas. Al final, hay una degustación de todos los aceites que hacen (tres blends, tres varietales, y muchos saborizados), y obvio uno puede comprar ahí con algún descuento :)

Malena, disfrutando del aire libre

En fin. Vacaciones, paseo, descanso. Ahora de nuevo en la jungla, desde hace rato...

Marcelo Fernández: CORE Network Emulator

Investigando herramientas de simulación/emulación de redes con fines educativos, un amigo pasó esta página, bastante nutrida por cierto, con un listado de simuladores/emuladores de red, recomendando probar CORE.

C.O.R.E., acrónimo de Common Open Research Environment, fue un proyecto inicialmente de Boeing (sí, la de los aviones) y que ahora es sponsoreado por el Laboratorio de Investigación Naval de los Estados Unidos. Le dediqué un rato a revisar qué tal funcionaba, y aquí está lo que pude probar.

A priori, resulta que la última versión (4.8) está en los repos de Ubuntu 16.04, por lo que fue fácil la instalación [1]:

marcelo@marcelo-notebook:~$ apt-cache search core-network
 core-network - intuitive network emulator that interacts with real nets (metapackage)
 core-network-daemon - intuitive network emulator that interacts with real nets (daemon)
 core-network-gui - intuitive network emulator that interacts with real nets (GUI)

Se instalan esos paquetes y listo el pollo, las dependencias son básicas (TCL/TK y quagga):

marcelo@marcelo-notebook:~$ sudo apt install core-network core-network-gui
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias
Leyendo la información de estado... Hecho
Se instalarán los siguientes paquetes adicionales:
 core-network-daemon libev4 libtcl8.5 libtk-img libtk8.5 quagga tcl8.5 tk8.5
Paquetes sugeridos:
 libtk-img-doc snmpd tcl-tclreadline
Se instalarán los siguientes paquetes NUEVOS:
 core-network core-network-daemon core-network-gui libev4 libtcl8.5 libtk-img libtk8.5 quagga tcl8.5 tk8.5
0 actualizados, 10 nuevos se instalarán, 0 para eliminar y 2 no actualizados.
Se necesita descargar 3.905 kB de archivos.
Se utilizarán 16,4 MB de espacio de disco adicional después de esta operación.
¿Desea continuar? [S/n]

Usarlo es tanto como ejecutar “core-gui” y empezar a armar el mapa de la red tal como se ve en el video de la página que citó Mauro:

Vista de Red - CORE

Vista de Red – CORE

El direccionamiento lo hace automáticamente (aunque es configurable, haciéndole doble click a cada nodo):

… y la configuración del ruteo se hace sola (ver más abajo cómo), por lo que luego de armar esa red sencilla, ambos hosts (“PC” y “Server”) se ven automáticamente.

Luego de armar la red, se le da al botón de “Play” y se ejecuta todo. Todos los nodos son Linux (nada de emulación routers Cisco ni nada como sucede con GNS3), y acá viene lo interesante:

  • Todos son containers (LXC) del mismo host de uno, pero sin estar en un chroot (?!?),
  • Es extremadamente rápido gracias a LXC, pero “raro” ya que todo corre en el mismo filesystem de la máquina de uno (puedo ir a mi home directamente desde cada nodo).
  • Dado que es un container, todos los kernels guest son los mismos del host.
  • ¡El software que se ejecuta es el mismo del host! Es decir, si digo que este nodo va a tener Apache (servicio “HTTP”), hay que instalar el paquete apache2 en el host, no en el guest. No hay imagen de máquina virtual ni nada por el estilo como pasa con otros emuladores.
  • El directorio que te abre al entrar a cada guest es/tmp/pycore.<PID>/<nombredelhost>.conf/
  • La configuración de los servicios/apps que va a usar en cada host la genera el entorno antes de darle “Play” en /tmp/pycore.<PID>/<nombredelhost>.conf/, por ejemplo:

Uno puede ir y editar los archivos en el directorio etc.apache2/ que generó el sistema una vez que se le dio ejecución, o puede hacerlo antes, tocando en el botón “Services” de la configuración del host:

Y luego en el ícono de llave inglesa:
Este es un ps desde el guest Server:

root@Server:/tmp/pycore.34211/Server.conf# ps faxu
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 9676 1708 ? S 12:20 0:00 /usr/sbin/vnoded -v -c /tmp/pycore.34211/Server -l /tmp/pycore.34211/Server.log -p /tmp/pycore.34211/Server.pid -C /tmp/pycore.34211/Server.conf
root 46 0.0 0.0 65508 3096 ? Ss 12:20 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config
root 52 0.0 0.0 56680 3772 ? Ss 12:20 0:00 /usr/sbin/apache2 -k start
www-data 53 0.0 0.0 411412 3504 ? Sl 12:20 0:00 \_ /usr/sbin/apache2 -k start
www-data 54 0.0 0.0 411412 3504 ? Sl 12:20 0:00 \_ /usr/sbin/apache2 -k start
root 110 0.0 0.0 23848 3968 pts/19 Ss 12:20 0:00 /bin/bash
root 236 0.0 0.0 39932 3316 pts/19 R+ 12:22 0:00 \_ ps faxu
root@Server:/tmp/pycore.34211/Server.conf#

Este es el running-config generado por core-network de un router (r1), entrando a Quagga con vtysh:Así que ahí vemos que define OSPF automáticamente:

Respecto a si se pueden guardar/cargar laboratorios completos, sí se puede, en el formato “imn” (ya que CORE es un fork de Imunes, y heredó su formato). Algo muy interesante es que se puede configurar el QoS de cada link, haciéndole doble click a cada línea (no lo probé):

Y que te grafica el ancho de banda en tiempo real de la red:

Por último, para hacer capturas de tráfico es bastante sencillo, probé dos opciones:

  1. Se corre en cada nodo que se desea un tcpdump guardando el tráfico, queda en el “home” de cada container guest (/tmp/pycore.<PID>/<nombredelhost>.conf/), y uno desde el host directamente ejecuta Wireshark y lo abre.
  2. Pude guardar todo el tráfico de la red en una única captura (esto, con netkit, no lo pudimos hacer de forma directa). Para ello, primero se arranca el entorno de simulación; esto hace que se creen dinámicamente los bridges a nivel de host y las interfaces virtuales de cada container, adjuntas a cada bridge (el virbr0 de la captura es de Virtualbox, ignórenlo):

Y bueno, para hacer la captura de toda la red al mismo tiempo, hay que decirle a Wireshark que capture en todos los bridges:

Filtré porque aparece tráfico OSPF todo el tiempo, lógicamente. Es cuestión de desactivarlo por defecto (configurando el servicio Quagga/Zebra). Un detalle de capturar así es que hay que ordenar por tiempo, porque el packet number queda desordenado. Pero ordenando por tiempo, lo seguí y aparentemente (hice un ping, nada más) el orden se mantiene (el “reloj” sería el host).

El manual está acá, parece que se pueden hacer muchísimas cosas más (entornos distribuidos, scripting automatizado en Python, etc.):
https://downloads.pf.itd.nrl.navy.mil/docs/core/core-html/index.html

Parece ser una muy buena herramienta para simular entornos de red, practicar y aprender sobre protocolos.

[1] Resulta que hace poco lo sacaron de Debian/Ubuntu, porque claro, el entorno gráfico se ejecuta como usuario normal, pero al abrir la consola de cualquier nodo que uno creó entra a la VM como root (y recuerden que no se está dentro de un chroot, con lo cual es root en el host con acceso al filesystem):

https://github.com/coreemu/core/issues/117

No lo solucionaron, entonces Debian los sacó, por ende no está en Debian Stretch. Igual maintainer del paquete tiene un repositorio personal para Debian/Ubuntu:

http://eriberto.pro.br/core/

Saludos

Marcos Dione: reprojecting-and-splitting-huge-datasets

Another aspect I've been looking into with respect to optimizing the rendering speed is data sources that are not in the target projection. Not being in the target projection forces Mapnik to reproject them on the fly, and this for each (meta)tile, whereas it would make more sense to have them already reprojected.

In my case I have 3 datasets, big ones, in EPSG:4258. The three datasets are elevation, hill and slope shade based on EEA's DEM files. The source tiles amount to almost 29GiB, and the elevation layer, being RGB, takes more than that. So I set off to try to reproject the things.

My first, more obvious approach was to reproject every 5x5°, 18000x18000px tile, then derive the data I need, but I started to get gaps between tiles. Notice that the original data is cleanly cut (5° x 3600"/° x 1px/" == 18000px), without any overlapping.

The next approach was to merge them all in a .vrt file, then reproject chunks of it with gdalwarp. What I wanted as output was the same 5x5° tiles, reprojected, but with an extra pixel, so they overlap. This last requirement was the problematic one. See, the final projection makes any square in the original projection a tall rectangle, stretching more and more towards the poles. The closest I could get was to use the -ts option, but that meant that I didn't get any control about how many extra pixels I got in the vertical/latitude direction. My OCD started thrashing :) In fact what happened was that I was not sure how GDAL would handle the possible partial pixel, whether rounding down (meaning excluding it), up (finishing it), or simply leaving the pixel with partial data and impacting the final rendering.

Even Rouault pointed to me that gdalwarp can do something fantastic: it can generate a .vrt file too with all the parameters needed for the reprojection, so reading from there was automatically reprojecting the original data. The resulting dataset is 288,000x325,220px (the original is 288,000x180,000px), so I'm definitely going to cut it down in small tiles. After consulting with a eight-ball, I decided to discard the idea of tiles with boundaries based on coordinates, which might not even make sense anymore, but settle for pixel based sizes, still with an extra pixel. The chosen size is 2**14+1 a.k.a. 16385. For this gdal_translate is perfect.

The final algorithm is like this:

gdalwarp -t_srs "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 \
    +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" \
    -r lanczos -tr 30.92208077590933 -30.92208077590933 \
    -of VRT EU-DEM.vrt EU-DEM-corrected.vrt

The values for the -tr option is the pixel size in meters, which is the unit declared in the SRS. Notice that as Mercator stretches towards the poles, this is the size at the origin of the projection; in this case, at 0° lat, 0° lon.

Then the reprojection (by reading from the reprojecting dataset) and cut, in a couple of loops:

tile_size=$((2**14)); \
for i in $(seq 0 17); do
    for j in $(seq 0 5); do
        for k in $(seq 0 3); do
            l=$((4*$j+$k));
            gdal_translate -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZMA \
                -co LZMA_PRESET=9 \
                -srcwin $(($tile_size*$i)) $(($tile_size*$l)) \
                    $(($tile_size+1)) $(($tile_size+1)) \
                -of GTiff EU-DEM-corrected.vrt \
                $(printf "%03dx%03d-corrected.tif" $i $l) &
        done;
        wait;
    done;
done

There's an extra loop to be able to launch 4 workers at the same time, because I have 4 cores. This doesn't occupy the 4 cores a 100% of the time (cores that already finished stay idle until the other finished), but it was getting akward to express in a Makefile, and this is run only once.

Before deriving the rest of the data there's an extra step: removing those generated tiles that actually have no data. I do a similar thing with empty sea tiles in the rendering process. Notice also that the original data is not tile complete for the covered region (79 tiles instead of the 160 they should be).


openstreetmap gis gdal

Marcos Dione: callable-choices-for-django-rest-framework

At work I'm writing an API using Django/DRF. Suddenly I had to write an application (just a few pages for calling a few endpoints), so I (ab)used DRF's Serializers to build them. One of the problems I faced while doing this was that DRF's ChoiceField accepts only a sequence with the values for the dropdown, unlike Django's, who also accepts callables. This means that once you gave it a set of values, it never ever changes, at least until you restart the application.

Unless, of course, you cheat. Or hack. Aren't those synonyms?

class UpdatedSequence:
    def __init__(self, update_func):
        self.update_func = update_func
        self.restart = True

        self.data = None
        self.index = 0


    def __iter__(self):
        # we're our own iterator
        return self


    def __next__(self):
        # if we're iterating from the beginning, call the function
        # and cache the result
        if self.restart:
            self.data = self.update_func()
            self.index = 0

        try:
            datum = self.data[self.index]
        except IndexError:
            # we reached the limit, start all over
            self.restart = True
            raise StopIteration
        else:
            self.index += 1
            self.restart = False

        return datum

This simple class tracks when you start iterating over it and calls the function you pass to obtain the data. Then it iterates over the result. When you reach the end, it marks it to start all over, so the next time you iterate over it, it will call the function again. The function you pass can be the all() method of a QuerySet or anything else that goes fetch data and returns an iterable.

In my case in particular, I also added a TimedCache so I don't read twice the db to fill two dropdown with the same info in the same form:

class TimedCache:
    '''A function wrapper that caches the result for a while.'''
    def __init__(self, f, timeout):
        self.f = f
        self.timeout = timeout
        self.last_executed = None
        self.cache = None
        self.__name__ = f.__name__ + ' (Cached %ds)' % timeout


    def __call__(self):
        now = time.monotonic()

        if self.cache is None or (now - self.last_executed) > self.timeout:
            self.cache = self.f()
            self.last_executed = now

        return self.cache

I hope this helps someone.


python django drf

Mariano Guerra: ameo: Redis compatible GET,SET,DEL and PUBLISH/SUBSCRIBE on riak_core with WebSocket API

Last week I was invited by Erlang Solutions to give a talk at the London's Erlang Meetup and to help at a riak_core themed hackathon at their offices the next day.

The talk slides: Building distributed applications: riak_core vs partisan in case you are interested.

After giving the introduction to riak_core the teams started to work on their projects and since they were pretty busy and didn't need much help I decided to "participate" too by implementing an idea I had in mind for a while.

The result is called ameo and can be described as:

a Redis compatible, distributed in-memory key-value store and pubsub server
implemented using riak_core that exposes the topics via WebSockets.

Why?

Many languages make it hard to provide websocket connections and live connections with clients, they work on a request/response basis and/or make it really hard/expensive to handle multiple persistent connections.

I've seen solutions that involve starting a redis server and putting usually nodejs in the front to expose redis topics via websockets, this involves two moving parts and for many projects, managing nodejs which they may not have experience with.

The solution is to just start one instance of ameo or a cluster of ameos and expose the WebSocket API to clients and the Redis API to the servers.

Servers can use their preferred Redis client library and as long as they only use GET, PUT, DEL, PUBLISH, SUBSCRIBE and UNSUBSCRIBE it will look like they are talking to a Redis server.

On my way back I had some extra time at the airport so I implemented a basic web UI to play with the websocket client and to provide a reference implementation others can use.

You can see the result in this screencast:

Implementation details:

As said earlier I use riak_core for clustering, for the Redis part I took some modules from an Erlang implementation of Redis called edis and I created a library called edis_proto that allows any project to expose a Redis compatible API to their servers with a couple lines of code.

The WebSocket part is implemented using the Cowboy Web Server.

Mariano Guerra: ameo: Redis compatible GET,SET,DEL and PUBLISH/SUBSCRIBE on riak_core with WebSocket API

Last week I was invited by Erlang Solutions to give a talk at the London's Erlang Meetup and to help at a riak_core themed hackathon at their offices the next day.

The talk slides: Building distributed applications: riak_core vs partisan in case you are interested.

After giving the introduction to riak_core the teams started to work on their projects and since they were pretty busy and didn't need much help I decided to "participate" too by implementing an idea I had in mind for a while.

The result is called ameo and can be described as:

a Redis compatible, distributed in-memory key-value store and pubsub server
implemented using riak_core that exposes the topics via WebSockets.

Why?

Many languages make it hard to provide websocket connections and live connections with clients, they work on a request/response basis and/or make it really hard/expensive to handle multiple persistent connections.

I've seen solutions that involve starting a redis server and putting usually nodejs in the front to expose redis topics via websockets, this involves two moving parts and for many projects, managing nodejs which they may not have experience with.

The solution is to just start one instance of ameo or a cluster of ameos and expose the WebSocket API to clients and the Redis API to the servers.

Servers can use their preferred Redis client library and as long as they only use GET, PUT, DEL, PUBLISH, SUBSCRIBE and UNSUBSCRIBE it will look like they are talking to a Redis server.

On my way back I had some extra time at the airport so I implemented a basic web UI to play with the websocket client and to provide a reference implementation others can use.

You can see the result in this screencast:

Implementation details:

As said earlier I use riak_core for clustering, for the Redis part I took some modules from an Erlang implementation of Redis called edis and I created a library called edis_proto that allows any project to expose a Redis compatible API to their servers with a couple lines of code.

The WebSocket part is implemented using the Cowboy Web Server.

Mariano Guerra: Playing with Lasp in a 3 Node Cluster

After playing with Lasp in a single node in the previous post: Playing with Lasp and CRDTs

I created a rebar3 template to easily setup a 3 node cluster to play with Lasp where it shines, in a distributed environment.

Install rebar3 if you haven't already.

then install this template:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_lasp.git ~/.config/rebar3/templates/rebar3_template_lasp

Use

rebar3 new rebar3_template_lasp name=laspy
cd laspy

    # build the 3 node cluster
make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

On one of the nodes' shell run:

Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

On another one run:

{ok, AwMapRes} = lasp:query({<<"awmap">>,{state_awmap,[state_mvregister]}}).

AwMapRes.

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).

You should get:

[#{what => i_am_an_awmap_value}]

All the examples from the previous post should run.

A termcast (?) of the process:

Happy Lasping!

Mariano Guerra: Playing with Lasp in a 3 Node Cluster

After playing with Lasp in a single node in the previous post: Playing with Lasp and CRDTs

I created a rebar3 template to easily setup a 3 node cluster to play with Lasp where it shines, in a distributed environment.

Install rebar3 if you haven't already.

then install this template:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_lasp.git ~/.config/rebar3/templates/rebar3_template_lasp

Use

rebar3 new rebar3_template_lasp name=laspy
cd laspy

    # build the 3 node cluster
make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

On one of the nodes' shell run:

Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

On another one run:

{ok, AwMapRes} = lasp:query({<<"awmap">>,{state_awmap,[state_mvregister]}}).

AwMapRes.

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).

You should get:

[#{what => i_am_an_awmap_value}]

All the examples from the previous post should run.

A termcast (?) of the process:

Happy Lasping!

Mariano Guerra: Playing with Lasp and CRDTs

For a long time I was looking for some time to play with Lasp:

Lasp is a suite of libraries aimed at providing a comprehensive programming
system for planetary scale Elixir and Erlang applications.

Yesterday I had some time and played a little bit with it, and thanks to the help from @cmeik and @vitorenesduarte I got some code running and understood some things.

Here is what I achieved until now, note that I'm learning this stuff so I may say many things that are wrong, I will try to get people to review it and update it with the corrections.

Some initial snippets of code we will reuse:

% clear all variable bindings
f().
Key1 = <<"key1">>.
Key2 = <<"key2">>.
Timestamp = fun () -> erlang:unique_integer([monotonic, positive]) end.

A map with any type as value

If we want to have data which can be represented as a dict/map/key-value mapping, there are multiple alternatives we can use in lasp, here we will explore some.

First we are going to use an AwMap CRDT as a map which is defined as:

A dictionary where keys can be anything and the values are causal-CRDTs.

Since values must be casual-CRDTs we can't use a bare erlang type as value nor a non causal-CRDT type like lwwregister, which we will use later with another map type.

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal = #{what => i_am_an_awmap_value}.

% declare the variable
{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

% update its content setting Key1 = AwMapVal
{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal}},
                                      self()).
% timestamp argument is not needed in mvregister, it's only for compatibility
% with lwwregister
{ok, _} = lasp:update(AwMap, {apply, Key1, {set, nil, AwMapVal}}, self()).

% get the value
{ok, AwMapRes} = lasp:query(AwMap1).

AwMapRes.
% {ok,[{<<"key1">>, {set, ...#{what => i_am_an_awmap_value} ... }}]}

[{_, AwMapSet}] = AwMapRes.
sets:to_list(AwMapSet).
% [#{what => i_am_an_awmap_value}]

We can also use a GMap CRDT (Grow Only Map, keys can't be removed):

A dictionary where keys can be anything and the values are join-semilattices.

In this case we can use non causal-CRDTs as values, to make it a little different we are going to use another register type, a lwwregister

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal = #{what => i_am_a_gmap_value}.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).
{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                            {set, Timestamp(), GMapVal}},
                                            self()).

{ok, GMapRes} = lasp:query(GMap1).

GMapRes.
% [{<<"key1">>,#{thing => 42,what => i_am_a_gmap_value}}]

[{_, GMapResVal}] = GMapRes.

GMapResVal.
% #{what => i_am_a_gmap_value}

We can use a gmap with an ivar as value type, an ivar is a register that can only be set once.

GMapIVarType = {state_gmap, [state_ivar]}.
GMapIVarVarName = <<"gmapivar">>.
GMapIVarVal1 = #{what => i_am_a_gmap_ivar_value}.
GMapIVarVal2 = #{what => i_am_a_gmap_ivar_update}.

{ok, {GMapIVar, _, _, _}} = lasp:declare({GMapIVarVarName, GMapIVarType}, GMapIVarType).
{ok, {GMapIVar1, _, _, _}} = lasp:update(GMapIVar, {apply, Key1,
                                            {set, GMapIVarVal1}},
                                            self()).
% try updating it, will throw an error (the value of GMapIVar1 will be lost)
{ok, {GMapIVar2, _, _, _}} = lasp:update(GMapIVar1, {apply, Key1,
                                            {set, GMapIVarVal2}},
                                            self()).

{ok, GMapIVarRes} = lasp:query(GMapIVar1).

GMapIVarRes.

[{_, GMapIVarResVal}] = GMapIVarRes.

GMapIVarResVal.
% #{what => i_am_a_gmap_ivar_value}

Types

My examples are all maps with some value type because that's the use case I'm most interested, here's a list of types and their operations:

"Scalars"

boolean

Operations:

true
Can be set to true once

Example:

BoolType = state_boolean.
BoolVarName = <<"boolvar">>.

{ok, {Bool, _, _, _}} = lasp:declare({BoolVarName, BoolType}, BoolType).
{ok, {Bool1, _, _, _}} = lasp:update(Bool, true, self()).

{ok, BoolRes} = lasp:query(Bool1).

BoolRes.
% true

ewflag:

Enable-Wins Flag CRDT

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

EWType = state_ewflag.
EWVarName = <<"ewvar">>.

{ok, {EW, _, _, _}} = lasp:declare({EWVarName, EWType}, EWType).
{ok, {EW1, _, _, _}} = lasp:update(EW, enable, self()).
{ok, EWRes1} = lasp:query(EW1).
{ok, {EW2, _, _, _}} = lasp:update(EW1, disable, self()).
{ok, EWRes2} = lasp:query(EW2).
{ok, {EW3, _, _, _}} = lasp:update(EW2, enable, self()).
{ok, EWRes3} = lasp:query(EW3).

EWRes1.
% true
EWRes2.
% false
EWRes3.
% true

dwflag:

Disable-Wins Flag CRDT

Follows the same strategy used in Enable-Wins Flag but,
instead of creating a new dot when enabling the flag,
we create a new dot when disabling it.

Operations:

enable
Enable the flag
disable
Disable the flag

Example:

DWType = state_dwflag.
DWVarName = <<"dwvar">>.

{ok, {DW, _, _, _}} = lasp:declare({DWVarName, DWType}, DWType).
{ok, {DW1, _, _, _}} = lasp:update(DW, enable, self()).
{ok, DWRes1} = lasp:query(DW1).
{ok, {DW2, _, _, _}} = lasp:update(DW1, disable, self()).
{ok, DWRes2} = lasp:query(DW2).
{ok, {DW3, _, _, _}} = lasp:update(DW2, enable, self()).
{ok, DWRes3} = lasp:query(DW3).

DWRes1.
% true
DWRes2.
% false
DWRes3.
% true

GCounter CRDT: grow only counter

Operations:

increment
Increment the counter by 1

Example:

GCountType = state_gcounter.
GCountVarName = <<"gcountvar">>.

{ok, {GCount, _, _, _}} = lasp:declare({GCountVarName, GCountType}, GCountType).
{ok, {GCount1, _, _, _}} = lasp:update(GCount, increment, self()).
{ok, GCountRes1} = lasp:query(GCount1).
{ok, {GCount2, _, _, _}} = lasp:update(GCount1, increment, self()).
{ok, GCountRes2} = lasp:query(GCount2).
{ok, {GCount3, _, _, _}} = lasp:update(GCount2, increment, self()).
{ok, GCountRes3} = lasp:query(GCount3).

GCountRes1.
% 1
GCountRes2.
% 2
GCountRes3.
% 3

Some more I don't get completely what they could be used for:

Bounded Counter CRDT:

Modeled as a pair where the first component is a PNCounter and the second
component is a GMap.

This counter has sub counter for different ids (actors), each of which can't go below 0, with this you can model things like seats or some resource where you allocate counts to different parties (actors) and each can decrement their own count but not others, also each counter can't go below 0, if a given actor needs to decrement it has to move counts from other actor.

Operations:

{move, term()}
Moves permissions to decrement to another replica (if it has enough permissions)
increment
Increment counter, can always happen
decrement
Decrement counter, can happen when the replica has enough local increments, or has permissions received from other replicas

Example:

BCountType = state_bcounter.
BCountVarName = <<"bcountvar">>.
Actor1 = self().
Actor2 = <<"actor2-id">>.

{ok, {BCount, _, _, _}} = lasp:declare({BCountVarName, BCountType}, BCountType).
{ok, {BCount1, _, _, _}} = lasp:update(BCount, increment, Actor1).
{ok, BCountRes1} = lasp:query(BCount1).
{ok, {BCount2, _, _, _}} = lasp:update(BCount1, increment, Actor2).
{ok, BCountRes2} = lasp:query(BCount2).
{ok, {BCount3, _, _, _}} = lasp:update(BCount2, decrement, Actor1).
{ok, BCountRes3} = lasp:query(BCount3).

% here Actor1 has counter set to 0, can't go below 0, if it want's to
% decrement it has to move a 1 from another actor, Actor2 has 1 in its
% counter so we will move it and then decrement

BCountMoveFrom = Actor2.
BCountMoveTo = Actor1.
{ok, {BCount4, _, _, _}} = lasp:update(BCount3, {move, 1, BCountMoveTo}, BCountMoveFrom).

% now we can decrement from Actor1
{ok, {BCount5, _, _, _}} = lasp:update(BCount4, decrement, Actor1).
{ok, BCountRes5} = lasp:query(BCount5).

BCountRes1.
% 1
BCountRes2.
% 2
BCountRes3.
% 1

BCountRes5.
% 0

Max Int CRDT

Operations:

  • increment

Example:

MaxIntType = state_max_int.
MaxIntVarName = <<"maxintvar">>.

{ok, {MaxInt, _, _, _}} = lasp:declare({MaxIntVarName, MaxIntType}, MaxIntType).
{ok, {MaxInt1, _, _, _}} = lasp:update(MaxInt, increment, self()).
{ok, MaxIntRes1} = lasp:query(MaxInt1).
{ok, {MaxInt2, _, _, _}} = lasp:update(MaxInt1, increment, self()).
{ok, MaxIntRes2} = lasp:query(MaxInt2).
{ok, {MaxInt3, _, _, _}} = lasp:update(MaxInt2, increment, self()).
{ok, MaxIntRes3} = lasp:query(MaxInt3).

MaxIntRes1.
% 1
MaxIntRes2.
% 2
MaxIntRes3.
% 3

Lexicographic Counter

Operations:

  • increment
  • decrement

Example:

LCountType = state_lexcounter.
LCountVarName = <<"lexcountvar">>.

{ok, {LCount, _, _, _}} = lasp:declare({LCountVarName, LCountType}, LCountType).
{ok, {LCount1, _, _, _}} = lasp:update(LCount, increment, self()).
{ok, LCountRes1} = lasp:query(LCount1).
{ok, {LCount2, _, _, _}} = lasp:update(LCount1, increment, self()).
{ok, LCountRes2} = lasp:query(LCount2).
{ok, {LCount3, _, _, _}} = lasp:update(LCount2, decrement, self()).
{ok, LCountRes3} = lasp:query(LCount3).

LCountRes1.
% 1
LCountRes2.
% 2
LCountRes3.
% 1

PNCounter CRDT:

Counter that allows both increments and decrements.

Modeled as a dictionary where keys are replicas ids and values are pairs
where the first component is the number of increments and the second component
is the number of decrements.

An actor may only update its own entry in the dictionary.

The value of the counter is the sum of all first components minus the sum
of all second components.

Operations:

  • increment
  • decrement

Example:

PNCountType = state_pncounter.
PNCountVarName = <<"pncountvar">>.

{ok, {PNCount, _, _, _}} = lasp:declare({PNCountVarName, PNCountType}, PNCountType).
{ok, {PNCount1, _, _, _}} = lasp:update(PNCount, increment, self()).
{ok, PNCountRes1} = lasp:query(PNCount1).
{ok, {PNCount2, _, _, _}} = lasp:update(PNCount1, increment, self()).
{ok, PNCountRes2} = lasp:query(PNCount2).
{ok, {PNCount3, _, _, _}} = lasp:update(PNCount2, decrement, self()).
{ok, PNCountRes3} = lasp:query(PNCount3).

PNCountRes1.
% 1
PNCountRes2.
% 2
PNCountRes3.
% 1

"Registers"

LWWRegister:

We assume timestamp are unique, totally ordered and consistent
with causal order. We use integers as timestamps.
When using this, make sure you provide globally unique timestamps.

Operations:

{set, Timestamp, Value}
Set register to Value

Example:

LWWRegType = state_lwwregister.
LWWRegVarName = <<"lwwregister">>.

{ok, {LWWReg, _, _, _}} = lasp:declare({LWWRegVarName, LWWRegType}, LWWRegType).
{ok, {LWWReg1, _, _, _}} = lasp:update(LWWReg, {set, Timestamp(), foo}, self()).
{ok, LWWRegRes1} = lasp:query(LWWReg1).
{ok, {LWWReg2, _, _, _}} = lasp:update(LWWReg1, {set, Timestamp(), bar}, self()).
{ok, LWWRegRes2} = lasp:query(LWWReg2).
{ok, {LWWReg3, _, _, _}} = lasp:update(LWWReg2, {set, Timestamp(), baz}, self()).
{ok, LWWRegRes3} = lasp:query(LWWReg3).

LWWRegRes1.
% foo
LWWRegRes2.
% bar
LWWRegRes3.
% baz

IVar:

Single-assignment variable.
Write once register.

Operations:

{set, Value}
Set register to Value (only possible once)
IVarRegType = state_ivar.
IVarRegVarName = <<"ivar">>.

{ok, {IVarReg, _, _, _}} = lasp:declare({IVarRegVarName, IVarRegType}, IVarRegType).
{ok, {IVarReg1, _, _, _}} = lasp:update(IVarReg, {set, foo}, self()).
{ok, IVarRegRes1} = lasp:query(IVarReg1).

IVarRegRes1.
% foo

MVRegister:

Multi-Value Register CRDT.

Operations:

{set, Timestamp, Value}
the Timestamp will not be used. in order to have an unified API for all registers (since LWWRegister needs to receive a timestamp), the timestamp is also supplied here

Example:

MVRegType = state_mvregister.
MVRegVarName = <<"mvregister">>.

{ok, {MVReg, _, _, _}} = lasp:declare({MVRegVarName, MVRegType}, MVRegType).
{ok, {MVReg1, _, _, _}} = lasp:update(MVReg, {set, Timestamp(), foo}, self()).
{ok, MVRegRes1} = lasp:query(MVReg1).
{ok, {MVReg2, _, _, _}} = lasp:update(MVReg1, {set, nil, bar}, self()).
{ok, MVRegRes2} = lasp:query(MVReg2).
{ok, {MVReg3, _, _, _}} = lasp:update(MVReg2, {set, Timestamp(), baz}, self()).
{ok, MVRegRes3} = lasp:query(MVReg3).

sets:to_list(MVRegRes1).
% [foo]
sets:to_list(MVRegRes2).
% [bar]
sets:to_list(MVRegRes3).
% [baz]

Collections

AWMap CRDT:

Modeled as a dictionary where keys can be anything and the values are
causal-CRDTs.

Operations:

{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).
{rmv, Key}
Remove Key

Example:

AwMapType = {state_awmap, [state_mvregister]}.
AwMapVarName = <<"awmap">>.
AwMapVal1 = foo.
AwMapVal2 = bar.

{ok, {AwMap, _, _, _}} = lasp:declare({AwMapVarName, AwMapType}, AwMapType).

{ok, {AwMap1, _, _, _}} = lasp:update(AwMap, {apply, Key1,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, {AwMap2, _, _, _}} = lasp:update(AwMap1, {apply, Key1,
                                              {set, Timestamp(), AwMapVal2}},
                                      self()).

{ok, {AwMap3, _, _, _}} = lasp:update(AwMap2, {apply, Key2,
                                              {set, Timestamp(), AwMapVal1}},
                                      self()).

{ok, AwMapRes3} = lasp:query(AwMap3).

{ok, {AwMap4, _, _, _}} = lasp:update(AwMap3, {rmv, Key2}, self()).

{ok, AwMapRes4} = lasp:query(AwMap4).

% before removing Key2
[{K, sets:to_list(V)} || {K, V} <- AwMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

[{K, sets:to_list(V)} || {K, V} <- AwMapRes4].
% [{<<"key1">>,[bar]}]

GMap CRDT:

Grow only map.
Modeled as a dictionary where keys can be anything and the
values are join-semilattices.
{apply, Key, Op}
Update Key with operation Op (since value is a type, you have to supply an operation, for example if it's a register it may be set, if it's a counter it could be increment).

Example:

GMapType = {state_gmap, [state_lwwregister]}.
GMapVarName = <<"gmap">>.
GMapVal1 = foo.
GMapVal2 = bar.

{ok, {GMap, _, _, _}} = lasp:declare({GMapVarName, GMapType}, GMapType).

{ok, {GMap1, _, _, _}} = lasp:update(GMap, {apply, Key1,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, {GMap2, _, _, _}} = lasp:update(GMap1, {apply, Key1,
                                              {set, Timestamp(), GMapVal2}},
                                      self()).

{ok, {GMap3, _, _, _}} = lasp:update(GMap2, {apply, Key2,
                                              {set, Timestamp(), GMapVal1}},
                                      self()).

{ok, GMapRes3} = lasp:query(GMap3).
% {ok,[{<<"key1">>,bar},{<<"key2">>,foo}]}

MVMap:

Multi-Value Map CRDT.
MVMap = AWMap<MVRegister<V>>

Operations:

{set, Key, Value}
Set Key to Value

Example:

MVMapType = state_mvmap.
MVMapVarName = <<"mvmap">>.
MVMapVal1 = foo.
MVMapVal2 = bar.

{ok, {MVMap, _, _, _}} = lasp:declare({MVMapVarName, MVMapType}, MVMapType).

{ok, {MVMap1, _, _, _}} = lasp:update(MVMap, {set, Key1, MVMapVal1}, self()).
{ok, {MVMap2, _, _, _}} = lasp:update(MVMap1, {set, Key1, MVMapVal2}, self()).
{ok, {MVMap3, _, _, _}} = lasp:update(MVMap2, {set, Key2, MVMapVal1}, self()).

{ok, MVMapRes3} = lasp:query(MVMap3).

[{K, sets:to_list(V)} || {K, V} <- MVMapRes3].
% [{<<"key1">>,[bar]},{<<"key2">>,[foo]}]

Pair

Operations:

{fst, Value}
Mutates the first item in the pair
{snd, Value}
Mutates the second item in the pair

Example:

PairLeftType = state_lwwregister.
PairRightType = state_gcounter.
PairType = {state_pair, [PairLeftType, PairRightType]}.
PairVarName = <<"pair">>.
PairVal1 = foo.
PairVal2 = bar.

{ok, {Pair, _, _, _}} = lasp:declare({PairVarName, PairType}, PairType).

{ok, {Pair1, _, _, _}} = lasp:update(Pair, {fst, {set, Timestamp(), PairVal1}}, self()).
{ok, {Pair2, _, _, _}} = lasp:update(Pair1, {snd, increment}, self()).
{ok, {Pair3, _, _, _}} = lasp:update(Pair2, {fst, {set, Timestamp(), PairVal2}}, self()).
{ok, {Pair4, _, _, _}} = lasp:update(Pair3, {snd, increment}, self()).

{ok, PairRes4} = lasp:query(Pair4).

PairRes4.
% {bar,2}

ORSet CRDT:

Observed-Remove Set with tombstones
{add, Element}
Add Element to the Set
{add_by_token, token(), element()}
Add Element to the Set by Token (orset internally generates a unique token for the element, therefore, it's not deterministic, this allows it to be explicit, and therefore deterministic)
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

ORSetType = state_orset.
ORSetVarName = <<"orset">>.
ORSetVal1 = foo.
ORSetVal2 = bar.
ORSetVal3 = baz.
ORSetAllVals = [ORSetVal1, ORSetVal2, ORSetVal3].

{ok, {ORSet, _, _, _}} = lasp:declare({ORSetVarName, ORSetType}, ORSetType).

{ok, {ORSet1, _, _, _}} = lasp:update(ORSet, {add, ORSetVal1}, self()).
{ok, {ORSet2, _, _, _}} = lasp:update(ORSet1, {add, ORSetVal2}, self()).
% repeat value
{ok, {ORSet3, _, _, _}} = lasp:update(ORSet2, {add, ORSetVal1}, self()).
{ok, {ORSet4, _, _, _}} = lasp:update(ORSet3, {add, ORSetVal3}, self()).

{ok, ORSetRes4} = lasp:query(ORSet4).
sets:to_list(ORSetRes4).
% [bar,baz,foo]

{ok, {ORSet5, _, _, _}} = lasp:update(ORSet4, {rmv, ORSetVal3}, self()).
{ok, ORSetRes5} = lasp:query(ORSet5).
sets:to_list(ORSetRes5).
% [bar,foo]

{ok, {ORSet6, _, _, _}} = lasp:update(ORSet5, {rmv_all, ORSetAllVals}, self()).
{ok, ORSetRes6} = lasp:query(ORSet6).
sets:to_list(ORSetRes6).
% []

{ok, {ORSet7, _, _, _}} = lasp:update(ORSet6, {add_all, ORSetAllVals}, self()).
{ok, ORSetRes7} = lasp:query(ORSet7).
sets:to_list(ORSetRes7).
% [bar,baz,foo]

Add-Wins ORSet CRDT:

Observed-Remove Set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set
{filter, Function}
Filter elements in the set

Example:

AWSetType = state_awset.
AWSetVarName = <<"awset">>.
AWSetVal1 = foo.
AWSetVal2 = bar.
AWSetVal3 = baz.
AWSetAllVals = [AWSetVal1, AWSetVal2, AWSetVal3].

{ok, {AWSet, _, _, _}} = lasp:declare({AWSetVarName, AWSetType}, AWSetType).

{ok, {AWSet1, _, _, _}} = lasp:update(AWSet, {add, AWSetVal1}, self()).
{ok, {AWSet2, _, _, _}} = lasp:update(AWSet1, {add, AWSetVal2}, self()).
% repeat value
{ok, {AWSet3, _, _, _}} = lasp:update(AWSet2, {add, AWSetVal1}, self()).
{ok, {AWSet4, _, _, _}} = lasp:update(AWSet3, {add, AWSetVal3}, self()).

{ok, AWSetRes4} = lasp:query(AWSet4).
sets:to_list(AWSetRes4).
% [bar,baz,foo]

{ok, {AWSet5, _, _, _}} = lasp:update(AWSet4, {rmv, AWSetVal3}, self()).
{ok, AWSetRes5} = lasp:query(AWSet5).
sets:to_list(AWSetRes5).
% [bar,foo]

{ok, {AWSet6, _, _, _}} = lasp:update(AWSet5, {rmv_all, AWSetAllVals}, self()).
{ok, AWSetRes6} = lasp:query(AWSet6).
sets:to_list(AWSetRes6).
% []

{ok, {AWSet7, _, _, _}} = lasp:update(AWSet6, {add_all, AWSetAllVals}, self()).
{ok, AWSetRes7} = lasp:query(AWSet7).
sets:to_list(AWSetRes7).
% [bar,baz,foo]

Add-Wins Set CRDT with the provenance semiring:

add-wins set without tombstones

Operations:

{add, Element}
Add Element to the Set
{add_all, ListOfElements}
Add a list of elements to the set
{rmv, Element}
Remove Element from the set
{rmv_all, ListOfElements}
Remove a list of elements from the set

Example:

AWPSSetType = state_awset_ps.
AWPSSetVarName = <<"awset_ps">>.
AWPSSetVal1 = foo.
AWPSSetVal2 = bar.
AWPSSetVal3 = baz.
AWPSSetAllVals = [AWPSSetVal1, AWPSSetVal2, AWPSSetVal3].

{ok, {AWPSSet, _, _, _}} = lasp:declare({AWPSSetVarName, AWPSSetType}, AWPSSetType).

{ok, {AWPSSet1, _, _, _}} = lasp:update(AWPSSet, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet2, _, _, _}} = lasp:update(AWPSSet1, {add, AWPSSetVal2}, self()).
% repeat value
{ok, {AWPSSet3, _, _, _}} = lasp:update(AWPSSet2, {add, AWPSSetVal1}, self()).
{ok, {AWPSSet4, _, _, _}} = lasp:update(AWPSSet3, {add, AWPSSetVal3}, self()).

{ok, AWPSSetRes4} = lasp:query(AWPSSet4).
sets:to_list(AWPSSetRes4).
% [bar,baz,foo]

{ok, {AWPSSet5, _, _, _}} = lasp:update(AWPSSet4, {rmv, AWPSSetVal3}, self()).
{ok, AWPSSetRes5} = lasp:query(AWPSSet5).
sets:to_list(AWPSSetRes5).
% [bar,foo]

{ok, {AWPSSet6, _, _, _}} = lasp:update(AWPSSet5, {rmv_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes6} = lasp:query(AWPSSet6).
sets:to_list(AWPSSetRes6).
% []

{ok, {AWPSSet7, _, _, _}} = lasp:update(AWPSSet6, {add_all, AWPSSetAllVals}, self()).
{ok, AWPSSetRes7} = lasp:query(AWPSSet7).
sets:to_list(AWPSSetRes7).
% [bar,baz,foo]

GSet CRDT:

Grow only set

Operations:

{add, Element}
Add Element to the Set

Example:

GSetType = state_gset.
GSetVarName = <<"gset">>.
GSetVal1 = foo.
GSetVal2 = bar.
GSetVal3 = baz.
GSetAllVals = [GSetVal1, GSetVal2, GSetVal3].

{ok, {GSet, _, _, _}} = lasp:declare({GSetVarName, GSetType}, GSetType).

{ok, {GSet1, _, _, _}} = lasp:update(GSet, {add, GSetVal1}, self()).
{ok, {GSet2, _, _, _}} = lasp:update(GSet1, {add, GSetVal2}, self()).
% repeat value
{ok, {GSet3, _, _, _}} = lasp:update(GSet2, {add, GSetVal1}, self()).
{ok, {GSet4, _, _, _}} = lasp:update(GSet3, {add, GSetVal3}, self()).

{ok, GSetRes4} = lasp:query(GSet4).
sets:to_list(GSetRes4).
% [bar,baz,foo]

2PSet CRDT:

Two-Phased Set

Once removed, elements cannot be added again.

Also, this is not an observed removed variant.

This means elements can be removed before being in the set.

Operations:

{add, Element}
Add Element to the Set
{rmv, Element}
Remove Element from the set

Example:

TPSetType = state_awset_ps.
TPSetVarName = <<"awset_ps">>.
TPSetVal1 = foo.
TPSetVal2 = bar.
TPSetVal3 = baz.
TPSetAllVals = [TPSetVal1, TPSetVal2, TPSetVal3].

{ok, {TPSet, _, _, _}} = lasp:declare({TPSetVarName, TPSetType}, TPSetType).

{ok, {TPSet1, _, _, _}} = lasp:update(TPSet, {add, TPSetVal1}, self()).
{ok, {TPSet2, _, _, _}} = lasp:update(TPSet1, {add, TPSetVal2}, self()).
% repeat value
{ok, {TPSet3, _, _, _}} = lasp:update(TPSet2, {add, TPSetVal1}, self()).
{ok, {TPSet4, _, _, _}} = lasp:update(TPSet3, {add, TPSetVal3}, self()).

{ok, TPSetRes4} = lasp:query(TPSet4).
sets:to_list(TPSetRes4).
% [bar,baz,foo]

{ok, {TPSet5, _, _, _}} = lasp:update(TPSet4, {rmv, TPSetVal3}, self()).
{ok, TPSetRes5} = lasp:query(TPSet5).
sets:to_list(TPSetRes5).
% [bar,foo]