Marcos Dione: breaking-bad

For the last two weeks I've been trying to slightly change Python's syntax. The goal was to allow a mix of keyword and normal parameters in a function, such as:

f (a=42, b)

I know that this sounds crazy, but (I think) I have a good reason for it:

  • in ayrton some functions act as frontends for executables;
  • keyword parameters for these functions are converted to options (a=42 gets converted to -a 42, foo=27 to --foo 27);
  • there is an arbitrary number of commands that consider their arguments as positional, like requiring that all the options come before the arguments; notably, rsync.

So, how to accomplish this? Well, there are good news and bad news... and plain news. The good news is that the lexer allows this kind of constructions. The plain news is that keywords-after-arguments is enforced in the syntax checker, which is implemented in Python/ast.c. This code ends up in the libpython. The bad news is that because all the functions defined in this file are declared as static, which means, among other things, that they're not exported by the library they're in. So, even if I only want to overwrite ast_for_call(), I still have to copy over all the rest of the functions. Finally, I already mentioned a slightly bad news, which is that I will have to make my own interpreter so this change applies, which implies also copying the Python interpreter's main function.

All that aside, how to make this syntax Python-compatible again? Well, the idea is simple: if a keyword is found before an argument, I convert it into a tuple with a string with the name of the parameter and its value. Thus, a=42 becomes ('a', 42) and then the frontend function resolves that to -a 42.

Between the plain and good news camps, the patch is rather simple:

diff -r d047928ae3f6 Python/ast.c
--- a/Python/ast.c      Sun May 12 19:50:34 2013 +0200
+++ b/Python/ast.c      Thu Oct 17 12:37:17 2013 +0200
@@ -2408,7 +2408,7 @@
       argument: [test '='] (test) [comp_for]        # Really [keyword '='] test

-    int i, nargs, nkeywords, ngens;
+    int i, nargs, nkeywords, ngens, convert_keywords;
     asdl_seq *args;
     asdl_seq *keywords;
     expr_ty vararg = NULL, kwarg = NULL;
@@ -2418,15 +2418,20 @@
     nargs = 0;
     nkeywords = 0;
     ngens = 0;
+    convert_keywords= 0;
     for (i = 0; i < NCH(n); i++) {
         node *ch = CHILD(n, i);
         if (TYPE(ch) == argument) {
-            if (NCH(ch) == 1)
+            if (NCH(ch) == 1) {
-            else if (TYPE(CHILD(ch, 1)) == comp_for)
+                if (nkeywords) {
+                    convert_keywords= 1;
+                }
+            } else if (TYPE(CHILD(ch, 1)) == comp_for)
-            else
+            else {
+            }
     if (ngens > 1 || (ngens && (nargs || nkeywords))) {
@@ -2440,6 +2445,11 @@
         return NULL;

+    if (convert_keywords) {
+        nargs+= nkeywords;
+        nkeywords= 0;
+    }
     args = asdl_seq_new(nargs + ngens, c->c_arena);
     if (!args)
         return NULL;
@@ -2451,13 +2461,15 @@
     for (i = 0; i < NCH(n); i++) {
         node *ch = CHILD(n, i);
         if (TYPE(ch) == argument) {
-            expr_ty e;
+            expr_ty e, e1;
             if (NCH(ch) == 1) {
+                /*
                 if (nkeywords) {
                     ast_error(c, CHILD(ch, 0),
                               "non-keyword arg after keyword arg");
                     return NULL;
+                */
                 if (vararg) {
                     ast_error(c, CHILD(ch, 0),
                               "only named arguments may follow *expression");
@@ -2478,26 +2489,27 @@
                 keyword_ty kw;
                 identifier key, tmp;
                 int k;
+                asdl_seq *t;

                 /* CHILD(ch, 0) is test, but must be an identifier? */
-                e = ast_for_expr(c, CHILD(ch, 0));
-                if (!e)
+                e1 = ast_for_expr(c, CHILD(ch, 0));
+                if (!e1)
                     return NULL;
                 /* f(lambda x: x[0] = 3) ends up getting parsed with
                  * LHS test = lambda x: x[0], and RHS test = 3.
                  * SF bug 132313 points out that complaining about a keyword
                  * then is very confusing.
-                if (e->kind == Lambda_kind) {
+                if (e1->kind == Lambda_kind) {
                     ast_error(c, CHILD(ch, 0), "lambda cannot contain assignment");
                     return NULL;
-                } else if (e->kind != Name_kind) {
+                } else if (e1->kind != Name_kind) {
                     ast_error(c, CHILD(ch, 0), "keyword can't be an expression");
                     return NULL;
-                } else if (forbidden_name(c, e->, ch, 1)) {
+                } else if (forbidden_name(c, e1->, ch, 1)) {
                     return NULL;
-                key = e->;
+                key = e1->;
                 for (k = 0; k < nkeywords; k++) {
                     tmp = ((keyword_ty)asdl_seq_GET(keywords, k))->arg;
                     if (!PyUnicode_Compare(tmp, key)) {
@@ -2508,10 +2520,21 @@
                 e = ast_for_expr(c, CHILD(ch, 2));
                 if (!e)
                     return NULL;
-                kw = keyword(key, e, c->c_arena);
-                if (!kw)
-                    return NULL;
-                asdl_seq_SET(keywords, nkeywords++, kw);
+                if (!convert_keywords) {
+                    kw = keyword(key, e, c->c_arena);
+                    if (!kw)
+                        return NULL;
+                    asdl_seq_SET(keywords, nkeywords++, kw);
+                } else {
+                    /* build a tuple ('name', expr) */
+                    t = asdl_seq_new(2, c->c_arena);
+                    if (!t)
+                        return NULL;
+                    asdl_seq_SET(t, 0, Str (key, e1->lineno, e1->col_offset, c->c_arena));
+                    asdl_seq_SET(t, 1, e);
+                    /* ... and out it as an argument */
+                    asdl_seq_SET(args, nargs++, Tuple (t, Load, e1->lineno, e1->col_offset, c->c_arena));
+                }
         else if (TYPE(ch) == STAR) {

Don't you hate when you have to modify a line just to add braces?

But of course, this break in the syntax should only be allowed for the frontend functions. This check is not implemented yet, so technically the new syntax is valid also for the Python code ayrton runs. This is not ideal, but hey, this is still a WIP :) For the moment this line of development will stay out of the main line. I will integrate it when I'm really convinced that it's a good idea.

python ayrton

Gustavo Campanelli: Ejemplo de diccionario en python con las regiones del Eve Online

No puedo ocultar lo mucho que me gusta Eve Online, y creo que no es secreto que me fascina  programar. Uniendo ambas cosas, ultimamente estoy desarrollando en python pequeñas herramientas de asistencia al juego. Pequeñas herramientas que solo tienen sentido para la forma en que yo juego el juego, entonces no puedo pretender que otro las programe. En medio de estas pequeñas cosas, a medida que

Joaquin Tita: controldeck.js on Mac Os

Disclaimer: this post might not be the better solution, probably there are other solutions over the net.

My intention was to give a try to controldeck.js. The reason is that this kind of presentations blow my mind.
Well is not all straightforward.
You need dependecies to be able to run controldeck.js.
Eaaaaasy i thought, there only two: node-static and and also node js...
I already had node js so i only needed node-static and
How can i can install node packages modules? with npm.
I tried npm in my console and surprisingly was already installed in my system!
The way that you install node pkg modules is running npm install package.
So I tried npm install node-static and an error appeared (same error also for
 TypeError Arguments to path.resolve must be strings
Reading and trying lot of things (most of them without knowing what they were really doing) i found a post that said to delete node js and reinstall it using nvm (node version manager).
Why not?
First, you must delete node js from the system, so i followed this post.
Yes, its tedious but in this way i also discovered that i had two differents versions of node js in my system. Probably that was the reason why i couldn't install node-static and After deleting all node's and node_modules, i installed nvm running in my console.
I installed nvm running:
curl | sh
Later, installed node js with nvm and worked! (checkout the usage section to do this)
To be sure if it was installed correctly type node -v in the console.
At this moment i was like the beginning, with node js installed (more than one version installed but working...).
I tried  in the console npm install static-node and worked!!
Then i also tried npm install worked too!!
Finally i run node app.js  from controldeck.js folder and opened the url http://localhost:8080 in a browser.
Controldeck.js was running...

Marcos Dione: ayrton-an-update

«So, by simply copying that one and, modifying the latter so it uses my modified syntax checker, and modifying the file, I should be done». Famous last words.

I was completely wrong. The _ast module imported by is not implemented in Python/ast.c but in Python/Python-ast.c. Understanding this, I reread this post I linked to last time and finally[1] understood that changing the syntax would not be so easy. Another eye-opener was the stack trace I got when ast.c's ast_for_call() (the function I'm planning to change) is called when a function call is found in the source. Here, have a look:

#0  ast_for_call (c=0x7fffffffa300, n=0x7ffff700d918, func=0x9ccf40) at Python/ast.c:2470
#1  0x00000000005bfe73 in ast_for_trailer (c=0x7fffffffa300, n=0x7ffff700db98, left_expr=0x9ccf40) at Python/ast.c:2105
#2  0x00000000005c03c0 in ast_for_power (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2217
#3  0x00000000005c0b01 in ast_for_expr (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2406
#4  0x00000000005c1108 in ast_for_testlist (c=0x7fffffffa300, n=0x7ffff6ca4030) at Python/ast.c:2562
#5  0x00000000005c118e in ast_for_expr_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:2584
#6  0x00000000005c3e84 in ast_for_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:3586
#7  0x00000000005bc94c in <span class="createlink">PyAST FromNodeObject</span> (n=0x7ffff6face18, flags=0x7fffffffa4e0, filename=0x7ffff6d56370, arena=0x8c8780) at Python/ast.c:709
#8  0x000000000050c9e8 in <span class="createlink">PyParser ASTFromStringObject</span> (s=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, arena=0x8c8780) at Python/pythonrun.c:2160
#9  0x000000000050c6be in <span class="createlink">Py CompileStringObject</span> (str=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, optimize=-1) at Python/pythonrun.c:2068
#10 0x00000000004c6e54 in builtin_compile (self=0x7ffff70f9c28, args=0x7ffff6ca04f8, kwds=0x0) at Python/bltinmodule.c:671

The compile() builtin (#10) passes through a couple of obscure functions till it reaches the AST build/syntax check code (all the ast_for_*() functions), and we quickly reach it from ast.parse():

def parse(source, filename='<unknown>', mode='exec'):
    return compile(source, filename, mode, PyCF_ONLY_AST)

In a desperate move, I though of hacking a modified version of just that function and do some LD_PRELOAD jedi moves, but the Python interpreter binary (/usr/bin/python3) is statically linked against libpython3. Future versions[2] of the Python interpreter main() function[3] will be really simple:

/* Minimal main program -- everything is loaded from the library */

#include "Python.h"

#ifdef __FreeBSD__
#include <floatingpoint.h>

main(int argc, char **argv)
        /* 754 requires that FP exceptions run in "no stop" mode by default,
         * and until C vendors implement C99's ways to control FP exceptions,
         * Python requires non-stop mode.  Alas, some platforms enable FP
         * exceptions by default.  Here we disable them.
#ifdef __FreeBSD__
        fp_except_t m;

        m = fpgetmask();
        fpsetmask(m & ~FP_X_OFL);
        return Py_Main(argc, argv);

So it's not so crazy to think of creating a new interpreter that uses my version of ast_for_call(), but I haven't found the linking command that makes that possible. So this part of the project is in the freezer.

Meanwhile, I spent a couple of hours with the code and managed to implement piping and redirection, even with some tests! You can have an idea of how it works reading the README[4]. Also, expect a release soon.

As for replacing sh, no efforts (besides burning some glucose and firing some neurons) have been done.

[1] Also famous last words.

[2] Python-3.3 has a more complex version which has to handle making two copies of argv while dealing with the system's encoding.

[3] It's weird, the source code of the executable resides in Modules/python.c. Go figure.

[4] Some day I'll convert that file to reST, make it the tutorial, and have it with the rest of the documentation.

python ayrton

Marcos Dione: near-future-of-ayrton

It might look like ayrton's development has stalled. First, yes, a little. Last week I barely put any time on it. Second, development will be less release centered, and only because I have three things in mind that are very interesing to me that I want to implement, and it seems that more or less I will be doing them on a «I want to hack on this right now» basis.

One is to replace sh. So far I have been tiptoeing around it to make output go to the terminal by default, which forces the intoroduction of Capture, and to not raise an exception if the command fails. But recently I tried to make an ayrton script to more or less automatize releases and I found out that sh really messes with the subprocess' view of the terminal. In short, editors are not usable if they're launched from sh. I could not say that this is the last straw; I still think that sh is very good at what it does, but this definetely is like a sequoia trunk after a few straws. This is called issue #15.

Another is to try to make several changes on the way output is captured, piped and redirected. I find this thing of messing with _out too annoying. Piping could be handled by |, but currently is not possible to do it. Redirections could be handled with >, but as long as I don't have |, it doesn't make much sense. I'm already playing with checking the AST produced by the source, because I'm trying to minimize the situations in which unknown names are resolved into executables, and these things also require AST handling, so it should be more or less easy to achieve.

There is another, more ambitious thing. So far I've been using Python3's syntax, but this imposes some restrictions. I like sh's keyword arguments, but the syntax doesn't allow you to put positional arguments after keyword arguments, like:

rsync (archive=True, src, dst)

So the goal is to use another parser. As this is the only modification in mind, it only makes sense to take Python3's parser, modify what I want, and ship with that. But the question is, where is the parser code?

The answer is not simple, but at least exists besides, the source code. I mean, I tried to untangle how this works, but as Eli Bendersky shows in this post, it is not a simple thing. Luckily, what I plan to change is not implemented in the parser itself, but in the syntax checker, which is implemented in the file ast.c. So, by simply copying that one and, modifying the latter so it uses my modified syntax checker, and modifying the file, I should be done. Let's see how it goes...

python ayrton

Joaquin Sorianello: Chau Carlitos

La semana pasada falleció mi querido abuelo Carlos, un inmenso filosofo y guia en el trajín de la existencia.

Sentí la necesidad de escribir unas palabras para leerle a la familia, antes de que su cuerpo sea cenizas. Lo comparto aquí:

Pensé que sabía como era la tristeza.
Pero no sabia.
En este ultimo tiempo, empece sin darme cuenta a vivir como vivías: mas despacio, mas en el presente; saboreando el tiempo sin apuro, disfrutando toda la belleza sutil que los hombres pudieron ver en un tiempo. Y es tan bello.
Que suerte tuvimos de tenerte cerca, paisano del medioevo. Nos dejaste las herramientas y la sabiduría para transformarte en leyenda. Nos dejaste el amor por la simpleza, por lo bello, por los culos de rebosantes gorditas porteñas.
Nos querias. Lo demostrabas con formas acordes a tu personalidad.Nunca regalaste un libro. Regalaste invitaciones a leerlos, y cumpliste tu objetivo de despertar miradas y enseñar a ver.
Ahora la digestora del tiempo nos hará recordarte. Nos la dejaste fácil. Basta con repetir tus frases, escuchar tus canciones favoritas. Mirar con ojos atentos, o sentarse en el banquito (del museo de Bellas Artes) a planear un robo romantico e imposible. Como vos.
Vas a ser el héroe de tataranietos curiosos por saber de ese pequeño cúmulo de sabiduría, amor y sensibilidad. Gracias. Por escuchar y por ser. Por dejarnos un  cacho tuyo aun por descubrir.
Ha muerto un gran hombre. Lloro por los irreversible.

Marcos Dione: ayrton-0.2

It's that time of the year already: a new ayrton version. This time I got more improvements than before, and for the first time[1] I introduce a backwards incompatible change, but at least is well Documented in the Debian-like NEWS.rst file, and it's for a good cause[2].

So, without further addo, the ChangeLog (mixed with the NEWS)[3]:

  • New function options() is similar to bash's set command. So far only the errexit and its short versions is accepted.
  • The ssh() context manager was renamed to remote() so the ssh executable is stills reachable from code. This was due to the fact that ssh is too complex to mimic.
  • New function shift() similar to bash's command of the same name. See the docs.
  • New function source() similar to bash's command[4] allows to "import" another script's local definitions into the "sourcing" script's.

As usual, you can grab the sources from pypi or GitHub.

[1] No, really, who can actually come up with that phrases when the product has only 3 previous releases... take it with a grain of salt.

[2] That cause being that I'm lazy and I don't plan to reimplement all the functionality available in the ssh executable.

[3] Actually, this changelog is proof-read nad more complete that any other that you will find elsewhere. Code burns in the disk and I'm not so used to this release thing yet. I hope to get better soon.

[4] Do you see the pattern here?

python ayrton

Gustavo Campanelli: Eve Online: Ayudas externas

Una de las particularidades del Eve Online es el grado de respeto que tiene CCP, la compañía, con sus usuarios. A la vez, el juego pose estructura y reglas, pero además de eso es un gran sandbox (arenenro) donde cada cual inveta su juego. Esto se traduce en una base de usuarios tremendamente fiel que además de generar eventos dentro del juego, construye cosas alrededor del juego, afuera del juego