OSCON: Tee Hee Wii

My co-worker who also attended OSCON this year was making fun of me for putting my business card in all and every contest receptacle. But today I found that it paid off. I won a drawing for a wii.

Who's laughing now monkey boy?!

emacs python mode from scratch: stage 1 - syntax coloring

First a note on my method. In order to keep my regular python mode functional while I'm playing around, my copy of the python mode uses different function names where possible (prepend "db-" to the normal name) and I'm targeting .pydb files (instead of .py).

So far this seems to work fine. I have the following in my .emacs file:

;; (load-library "db-python-1")
;; (add-to-list 'auto-mode-alist '("\\.pydb\\'" . db-python-mode))

After changes, I run the load-library command and it seems to pick up any changes that I've made recently. (In case it's not clear my stage 1 library is in a file called "db-python-1.el" and it provides db-python-mode). I plan to change the library file name as I progress and leave the db-python-mode name constant.

I am surprised on how much stuff is going on to get a mode working. But even more surprising is how understandable it is. I definitely don't understand all the working parts so far and not everything I've tried has worked but in general the pieces are falling pleasantly into place.

For stage 1, syntax coloring *mostly* works. For whatever reason I wasn't able to get comments to color correctly. My first guess was that one or more of the following:

(set (make-local-variable 'parse-sexp-lookup-properties) t)
(set (make-local-variable 'parse-sexp-ignore-comments) t)
(set (make-local-variable 'comment-start) "# ")

would do it, but so far no avail. But since I didn't get it working that means I don't really understand the lines above so by my rules I can't add them to the file. I'm hoping that at some point as I'm schlepping code from one place to another it *will* start working and hopefully then everything will fall into place.

I'm intrigued by the rx syntax for regular expressions. A little verbose but it might be better than the fairly non-standard emacs regex syntax. It's interesting to look at the actual regular expression that rx produces. For instance the keyword list that is ORed together dumps out as:

"\\(?:\\_<\\(?:False\\|None\\|True\\|a\\(?:nd\\|s\\(?:sert\\)?\\)\\|break\\|continue\\|del\\|e\\(?:l\\(?:if\\|se\\)\\|x\\(?:cept\\|ec\\)\\)\\|f\\(?:inally\\|or\\|rom\\)\\|global\\|i\\(?:mport\\|[fns]\\)\\|lambda\\|not\\|or\\|p\\(?:ass\\|rint\\)\\|r\\(?:aise\\|eturn\\)\\|self\\|try\\|w\\(?:hile\\|ith\\)\\|yield\\)\\_>\\)"

Which seems pretty bad. If you look carefully you can see that it has done some consolidating of terms. I'm not sure how useful it is to do it like that, but if nothing else the rx style is quite readable compared to the above final output.

I also removed db-python-font-lock-syntactic-keywords since adding it in didn't seem to do anything and I really want to understand what all the pieces do.

One thing I had never noticed before is that globals at the top level of the module get a special syntax coloring.

So with no further ado, here is what I have so far:: db-python-1.el

(defvar db-python-font-lock-keywords
  `(,(rx symbol-start
         ;; From v 2.4 reference.
         ;; def and class dealt with separately below
         (or "and" "assert" "break" "continue" "del" "elif" "else"
             "except" "exec" "finally" "for" "from" "global" "if"
             "import" "in" "is" "lambda" "not" "or" "pass" "print"
             "raise" "return" "try" "while" "yield"
             ;; Future keywords
             "as" "None" "with"
             ;; Not real keywords, but close enough to be fontified as such
             "self" "True" "False")
         symbol-end)
    ;; Definitions
    (,(rx symbol-start (group "class") (1+ space) (group (1+ (or word ?_))))
     (1 font-lock-keyword-face) (2 font-lock-type-face))
    (,(rx symbol-start (group "def") (1+ space) (group (1+ (or word ?_))))
     (1 font-lock-keyword-face) (2 font-lock-function-name-face))
    ;; Top-level assignments are worth highlighting.
    (,(rx line-start (group (1+ (or word ?_))) (0+ space) "=")
     (1 font-lock-variable-name-face))
    (,(rx "@" (1+ (or word ?_))) ; decorators
    (0 font-lock-preprocessor-face))))

(define-derived-mode db-python-mode fundamental-mode "(DB)Python-1"
  (set (make-local-variable 'font-lock-defaults)
       '(db-python-font-lock-keywords nil nil nil nil
                                   ;;(font-lock-syntactic-keywords
                                   ;; . db-python-font-lock-syntactic-keywords)
                                   ))
)

(provide 'db-python)

OSCON: Larry Wall is boring

I don't say this lightly. Though I currently work in python, perl was one of my first loves and I'll always have a soft spot in my heart (and head) for it. This is my 4th OSCON and the 4th time I've seen Larry give his state of the onion address. And until tonight they have always been a highlight. Don't get me wrong, they aren't as insanely clever as Damian or deliciously scholarly like r0ml, but they have a definite low key humor and fun that is contagious no matter your language.

But tonight he bored my socks off. Actually, that's not true. Looking down I do in fact seem to be wearing my socks. But he just gave a straight forward technical speech. No whimsy. No screensavers for slides. No magick (sp?) the gathering cards. Just straight technology.

It was really weird. I almost left early but I hung in until the end because I wanted to see if there would be some crazy grand finale.

But nothing.

Oh, well, maybe it's hard to be whimsical when your pet project is so far overdue.

Creating a python mode for emacs from scratch

One of many ideas that has been rattling around my brain for a while is to *really* learn emacslisp. At the minimum I'd like to just understand the tools that are available. And beyond that I'd like to contribute back a little. But that's just not going to happen until I have a deeper understanding of the monster that is emacs and emacslisp.

Rather than just goof around with toy projects I thought I'd dive right in and try to grok something that is very useful to me and something I'd like to understand even better. A good candidate for this is python.el. I use it everyday, I feel I could be using more of it and I'd love to help make it better.

My plan plan, then, is to recreate this python-mode from scratch. Of course I'll be cheating mightily by just cutting and pasting code, the game is that I can only cut and paste things that I understand what they do and how they fit into the big picture (and it has to work at each stage).

So, I start with a blank file and I'm not done until it's an exact match to the original. The original (which came with my emacs for windows install "This is GNU Emacs 22.0.990.1 (i386-mingw-nt6.0.6000) of 2007-05-23 on LENNART-69DE564 (patched)") is python.el (CVS 1.58.2.4) for those of you playing the home versoin of the game.

Here is a brief outline of what I'll need to address

126 matches for "^(def" in buffer: python.el
     75:(defgroup python nil
     90:(defvar python-font-lock-keywords
    114:(defconst python-font-lock-syntactic-keywords
    131:(defun python-quote-syntax (n)
    195:(defvar python-mode-map
    280:(defvar python-mode-syntax-table
    301:(defsubst python-in-string/comment ()
    306:(defconst python-space-backslash-table
    312:(defun python-skip-comments/blanks (&optional backward)
    330:(defun python-backslash-continuation-line-p ()
    335:(defun python-continuation-line-p ()
    350:(defun python-comment-line-p ()
    358:(defun python-blank-line-p ()
    364:(defun python-beginning-of-string ()
    371:(defun python-open-block-statement-p (&optional bos)
    380:(defun python-close-block-statement-p (&optional bos)
    391:(defun python-outdent-p ()
    406:(defcustom python-indent 4
    413:(defcustom python-guess-indent t
    418:(defcustom python-indent-string-contents t
    429:(defcustom python-honour-comment-indentation nil
    437:(defcustom python-continuation-offset 4
    444:(defun python-guess-indent ()
    475:(defvar python-indent-list nil
    478:(defvar python-indent-list-length nil
    481:(defvar python-indent-index nil
    484:(defun python-calculate-indentation ()
    594:(defun python-initial-text ()
    605:(defconst python-block-pairs
    614:(defun python-first-word ()
    620:(defun python-indentation-levels ()
    681:(defun python-indent-line-1 (&optional leave)
    700:(defun python-indent-line ()
    724:(defun python-indent-region (start end)
    739:(defun python-block-end-p ()
    754:(defun python-beginning-of-defun ()
    786:(defun python-end-of-defun ()
    832:(defun python-beginning-of-statement ()
    849:(defun python-skip-out (&optional forward syntax)
    871:(defun python-end-of-statement ()
    897:(defun python-previous-statement (&optional count)
    912:(defun python-next-statement (&optional count)
    928:(defun python-beginning-of-block (&optional arg)
    969:(defun python-end-of-block (&optional arg)
   1003:(defvar python-recursing)
   1004:(defun python-imenu-create-index ()
   1062:(defun python-electric-colon (arg)
   1076:(defun python-backspace (arg)
   1104:(defcustom python-check-command "pychecker --stdlib"
   1109:(defvar python-saved-check-command nil
   1113:(defun python-check (command)
   1136:(defcustom python-python-command "python"
   1144:(defcustom python-jython-command "jython"
   1150:(defvar python-command python-python-command
   1156:(defvar python-buffer nil
   1175:(defconst python-compilation-regexp-alist
   1188:(defvar inferior-python-mode-map
   1202:(defvar inferior-python-mode-syntax-table
   1214:(define-derived-mode inferior-python-mode comint-mode "Inferior Python"
   1245:(defcustom inferior-python-filter-regexp "\\`\\s-*\\S-?\\S-?\\s-*\\'"
   1251:(defun python-input-filter (str)
   1257:(defun python-args-to-list (string)
   1266:(defvar python-preoutput-result nil
   1269:(defvar python-preoutput-leftover nil)
   1270:(defvar python-preoutput-skip-next-prompt nil)
   1274:(defun python-preoutput-filter (s)
   1324:(defun run-python (&optional cmd noshow new)
   1385:(defun python-send-command (command)
   1403:(defun python-send-region (start end)
   1439:(defun python-send-string (string)
   1451:(defun python-send-buffer ()
   1458:(defun python-send-defun ()
   1464:(defun python-switch-to-python (eob-p)
   1473:(defun python-send-region-and-go (start end)
   1480:(defcustom python-source-modes '(python-mode jython-mode)
   1488:(defvar python-prev-dir/file nil
   1494:(defun python-load-file (file-name)
   1517:(defun python-proc ()
   1528:(defun python-set-proc ()
   1541:(defconst python-dotty-syntax-table
   1549:(defvar view-return-to-alist)
   1552:(defvar python-imports)                 ; forward declaration
   1557:(defun python-describe-symbol (symbol)
   1599:(defun python-send-receive (string)
   1614:(defun python-eldoc-function ()
   1643:(defun python-after-info-look ()
   1695:(defcustom python-jython-packages '("java" "javax" "org" "com")
   1704:(defun python-maybe-jython ()
   1731:(defun python-fill-paragraph (&optional justify)
   1774:(defun python-shift-left (start end &optional count)
   1798:(defun python-shift-right (start end &optional count)
   1811:(defun python-outline-level ()
   1818:(defun python-current-defun ()
   1835:(defun python-mark-block ()
   1851:(defvar python-imports nil
   1861:(defun python-find-imports ()
   1896:(defun python-symbol-completions (symbol)
   1912:(defun python-partial-symbol ()
   1923:(defun python-complete-symbol ()
   1957:(defun python-try-complete (old)
   1979:(defun python-module-path (module)
   1988:(defcustom python-use-skeletons nil
   1995:(defvar python-skeletons nil
   1999:(defvar python-mode-abbrev-table nil
   2002:(define-abbrev-table 'python-mode-abbrev-table ())
   2007:(defmacro def-python-skeleton (name &rest elements)
   2025:(def-python-skeleton if
   2035:(define-skeleton python-else
   2043:(def-python-skeleton while
   2049:(def-python-skeleton for
   2055:(def-python-skeleton try/except
   2066:(define-skeleton python-target
   2070:(def-python-skeleton try/finally
   2077:(def-python-skeleton def
   2084:(def-python-skeleton class
   2094:(defvar python-default-template "if"
   2098:(defun python-expand-template (name)
   2122:(defun python-setup-brm ()
   2166:(defvar outline-heading-end-regexp)
   2167:(defvar eldoc-documentation-function)
   2170:(defun python-abbrev-pc-hook ()
   2175:(defvar python-abbrev-syntax-table
   2179:(defun python-pea-hook ()
   2185:(defvar python-mode-running)            ;Dynamically scoped var.
   2188:(define-derived-mode python-mode fundamental-mode "Python"
   2297:(define-derived-mode jython-mode python-mode  "Jython"

Damn! That looks like a bit of work. OK, this mode isn't going to understand itself, let's get cracking. Tune in soon for the first of (potentially) many installments.

OSCON: Life Extension

As icing on the cake today there was a BOF on Life Extension. My wife mentioned that that seemed strange. It doesn't seem that strange to me. It actually fits right in.

I would say I'm on the shallow end of Life Extension. I'm basically interested in finding an optimal diet and exercise regimen to help me stay healthy and active and mentally acute as long as possible. I can't say how hardcore I'm am even at this shallow end because I've only had this as a clear objective in my life for the last 6 months or so, so I can't say how permanent my changes have been. But I've lost weight and I'll soon be getting a physical to see how my cholesterol (which had been a bit high when last checked) is responding.

The session was led by Christine Peterson who also was one of the keynote speakers from the morning session. While by many standards she would be seen as hardcore (lots of supplements, mandatory 2 week vacations per year, strict weight control, etc) she was definitely middle of the road compared to the true extremes of the life extension movement. The truly hardcore take on the order of 100 supplements a day (how long does that even take to swallow?), or do calorie restriction, weekly intravenous treatments, etc. And then of course there is the backup plan of having your head frozen if you do die.

I'm not particularly opposed to any of the hardcore stuff I'm just new to this field of thought and haven't explored it enough to have a strong opinion either way. The one thing I'm intrigued by is the fact that if I live long enough I might get to take advantage of the increase in lifespan provided by medical advances. If science extends human life by a year (or more) every year, then who ever is alive at that point (and has the cash) could live a very long time.

In any case I signed up to get her slides emailed to me, so I'm going to have to wait until I get those before I can say more about any changes I might make. Until then I need to look again at taking some supplements.

100 Pushups

When I came across this link for doing 100 pushups I just sorted of laughed and thought to myself, "yah right". But the idea got under my skin and so as of a couple days ago I finished week 4.

100 seems like a magical number and impossibly far away. Even at the 4 week mark I'm only up to 40 (ok, 39 and 1/2).

But there *is* something magical about this goal to me. Surely if I can use discipline and hard work to get this goal then I've learned something important about what I *thought* were my limits. If I can reach this far past what I thought was feasible then where else have I unnecessarily limited myself?

Of course if I fail then I've just confirmed that I suck. :)

I'm guessing I'll (at the very least) need to repeat the last couple of a weeks at least once. But, I think I can do this....

OSCON: Damian Conway

I thought tonight that I'd try taking some notes while Damian spoke to try to give a sense of how cool his talks are. Unfortunately it seems that the ability to take notes on his talks is indirectly proportional to how cool his talks are.

And it it was cool. (Quantum Mechanics, General Relativity, Nano-Machines and programs that run in negative time - with example code).

And I have no notes. So it must have been pretty good.

OSCON: Python 3.0 conversion tutorial

This was actually more interesting than I was expecting. Anthony Baxter gave a full afternoon tutorial on the things that will change as we go to python 3.0 and some helpful hints on how to get there.

- Looks like python 2.7 is a pretty likely eventuality (but definitely nothing past 2.9 :)
- You can import in several features "from the future" in 2.6 to get some of the 3.0 flavor (print as a function, non-relative imports, etc).
- optional type declaration in function defs could be cool
- foo, *stuff, bar = my_list # will be awesome
- {1, 2, 3} # as explicit set constructor is handy
- lambda isn't going away (you knew that, I just like to remind you). But so long my old friend reduce... :(

I'm very interested to try out the 2to3 converter. Actually the word I want is probably scared. I think alot of my personal code will be fine but the code at work will be a bear to get converted. Well, maybe it will force more wide spread unit testing (the real goal of py3k).

In any case I left the session more interested in 3.0 and less afraid of the conversion process. But I have a feeling it will be a couple years before this happens at work.

OSCON: seaside tutorial

Randall Schwartz gave an interesting seaside tutorial on Monday. Here are the slides.

I have been through a few squeak tutorials and seaside tutorials but was interested to see the big picture presented in one shot. I enjoyed the tutorial but I have to mention a couple of things that bugged me.

- tutorial spent way too much time on the basics (of smalltalk) and not enough on the cool stuff that makes seaside completely awesome (e.g. didn't get to the last 1/4 or so of slides). This is totally understandable because I think it's expected that most people haven't messed with smalltalk before and the syntax would probably be pretty distracting on a first view. Maybe it's time to have a seaside master class.

- Randall made a comment about how other web frameworks have continuations now. He cited django as an example. Hmmm... that doesn't seem right. My best guess is that he confused ruby (which has continuations) and python. But as far as I know neither rails or django use continuations.

But in general it was well done (despite the typos ;-) and I hope to see a continued/growing seaside presence at OSCON. This is the 3rd year in a row that has had at least one presentation on seaside. Maybe we need an in depth look at dabbledb next year...

Here is a seaside tutorial I recommend.

[added: 2008-07-23]
Reading the above again, the tone seemed more negative than I intended. Just want to leave the final thought that I really enjoyed the tutorial. It was well thought out and well presented. It's just easier to focus on the little nits. :)