One New Language a Year: (was) Smalltalk

OK, so I had this crazy plan of doing the "one new language a year" thing. I really like the idea of this. Get out of your comfort zone, prevent settling for a "blub" language, learn new techniques. So I choose smalltalk and thought this would fit the bill. I was really excited to work with turtles all the way down. "Real" object oriented programming. Etc.

And here it is May and I'm just not doing so well. Best laid schemes and all that. In fact I've decided to abandon it for, get this, emacslisp. Only time will tell if this was a good decision or just my normal fickleness.

Part of this decision is just pure pragmatism. My learning windows for this plan are of necessity going to be early in the morning before breakfast or late at night after the family's in bed. I've tried both slots for a month or more and both put me to sleep. You can't learn much while you are asleep so no matter the plan/desire I have to be doing something that engages my attention.

OK, I said it. Smalltalk puts me to sleep. And I feel bad for saying it. I really want to be the guy who likes smalltalk and wields it with authority. But there is just so much I don't enjoy about it and while I'm sure there are some cool things to learn from it, it just doesn't entice me. At least not at 6 am. We'll see if emacslisp can do better.

Smalltalk is like this quirky, fairly attractive girl who on paper seems like a good match for you ("Wow, you like continuations? Me too!") but for whatever reason just doesn't ignite that spark. In addition, she has lots of eccentricities that if she was "the one" would be charming/braggable. But if you are indifferent to her they just come off as irritations/oddities.

I also wonder if watching this video put the final nail in the coffin: "what killed smalltalk" (also). Just hearing about smalltalk being dead (I'm not defending this thesis) sort of activated my pragmatism module which some how grabbed the reins of my brain and executed veto power.

The thing that bugs me is that I don't see myself as a practical dude. In fact, I find the thought of being practical a little bit horrifying. I'm always doing things just because they are interesting, not because I'm trying to be more efficient, etc. And here I am making a practical decision. Ugg.

I wonder what the rules are about abandoning your "one new language a year" language halfway through the year. Is there some governing body I should contact? Do I need to ask permission first?

For what it's worth, smalltalk is not permanently abandoned; it is just pushed down on the list. If I stick with the language a year program I will almost certainly head back to smalltalk at some point. We'll see what happens next time around.

Does working with python as your primary language makes it harder to learn new languages? Every language that I've learned so far has been a productivity boost or has some other sweetener (my language history: basic -> pascal -> c -> c++ -> java -> perl -> python). emacslisp is a booster just because it is the only option in emacs (I will be looking into pymacs though). haskell looks like it might be a next step in the awesomeness hierarchy. But smalltalk doesn't *seem* (in my limited experience) like a significantly more productive tool past python. smalltalk feels like a language from which many great ideas have been lifted. And the ones that haven't yet are either in the process of being stolen (pypy - turtles all the way down) or aren't worth the trouble (image files, etc).

What's funny is that I've talked to others who have had the same initial fascination with smalltalk and then given up on it due various pain points or lack or practicality.

I guess in the end part of it is that smalltalk doesn't seem to solve a problem that I'm having in a vastly superior way and (surprisingly to me) isn't more fun. On paper it seems it should be. And I actually feel guilty for not liking smalltalk. I've failed the gods of language geekdom.

Any way, on to the new hotness of, er, emacslisp.

So why emacslisp? At the start of the year I was trying to decide between smalltalk and haskell. I was in sort of an object-y mood at the time so smalltalk won out. I'm not defaulting to haskell at this time primarily because I want to give it the full year (and it just feels like it has to be a calendar year starting in January). emacslisp works because I know it will be useful. It will make a real difference to my productivity immediately (I mostly live in emacs) and I like lispy languages (though I've only dabbled and/or used them for a semester in an ai class). *And* I don't feel too bad about giving it only half a year.

My goal with emasclisp is to get to the point where I can program in it as easily as I think (how it is for python with me now). I want to be able to whip out useful helper functions like this and be able to write my own modes and/or hack on existing emacs packages.

Any way, smalltalk, it's not you it's me. When I get my head right maybe we'll meet again and give it another shot. I wish you all the best. (but please don't call - I'm not sure emacslisp would understand us "just being friends").

emacs python mode from scratch: stage 18 - miscellany

stage 19 - miscellany

A grab bag of functions to finish us up:

python-jython-packages
python-maybe-jython
python-fill-paragraph
python-shift-left
python-shift-right
python-outline-level
python-current-defun
python-mark-block

• python-jython-packages
  
• python-maybe-jython
  

  I don't really care too much about jython support so this won't get much more that a brief inspection. python-jython-packages seems to be a list of module names that if found in a package will cause jython mode to be used
  

  python-maybe-jython is the function that uses the above list among other things to magically determine whether or not to use python or jython mode.
  

• python-fill-paragraph
  

  Provides a python aware fill-paragraph-function. The usual process would be to define paragraph-start and paragraph-separate and let fill do it's magic, but apparently this doesn't work quite right for the last paragraph in a multiline string. So instead This function narrows the string itself and does the fill action on that.
  

• python-shift-left
  
• python-shift-right
  

  Move some code left or right by the requested number of columns (defaulting to the prima facie indentation level). If there is no region selected then just work with current line. This works by iterating over every line and running indent-rigidly. There is less code for the python-shift-right variation since you don't have to look out for the case where you run out of room for shifting.
  

• python-outline-level
  

  get outline-level. This is just the multiple of python-indent of the current indentation level.
  

• python-current-defun
  

  Work up an indentation level at a time (ie def or class) and keep track of the names of the def and class entities along the way.
  

  This is used by add-log (add-log.el). apparently this is some sore of functionality for working with change logs. this function gives a way to refer to the name of the function where point currently resides. But honestly I'm not really sure how this fits into the big picture. This may well be some feature i depend on, but I'm having trouble figuring out what features use this and why.
  

  If you run M-x add-change-log-entry it opens up something called a change log. I guess I'll just keep my eyes open for this in the future.
  

• python-mark-block
  

  Mark the block. Just as you'd expect.
  

So that's pretty much the end except for some little details here and there. This was an interesting exercise though if I had thought about how long this was going to take I'm not sure I would have started down this path. I definitely learned a lot, but the main thing I learned was how little I really know about emacs. Now that I'm finished I find a nice little tutorial on how language modes work. Well at least I know what they are talking about now. My main lesson I guess is that I really need to learn emacslisp if I want to become a master. But even then I will still have a lot to learn about the actual environment. But I'm getting there.

emacs python mode from scratch: stage 18 - imenu

So first of all, what *is* imenu.

... read, read ....

So it partly seems like a one file only form of etags. But if you do M-x imenu-add-menubar-index it will add an "index" menu to the menu bar with a list of all files for easy access. Being a bit of a mouse-ophobe, I'm not so sure how I'd like that. Also you have to hit rescan whenever you change the contents of the file (or set up imenu-auto-rescan).

Seems like another case of having more than one way to do things. I'll probably stick with etags, but for completeness here is a brief overview of the imenu support functions.

So there is really just one function:

• python-imenu-create-index
  

  Go to the beginning of the file. look for "def and "class" references (unless in comments or string) and add them to the list.
  

Hmm, I wonder if something else more wonderful is going on here. Doesn't really seem like functionality worth supporting.

Perhaps if you had it set up to go off a simple mouse motion as described here.

(if (featurep 'xemacs)
     (global-set-key [(shift button3)] 'imenu) ; XEmacs
   (global-set-key [S-mouse-3] 'imenu)) ; GNU Emacs

Also looks like it integrates in with ido to some extent. Maybe there is some goodness from that.

Any way, that's enough on that topic.

emacs python mode from scratch: stage 17 - ffap

This is a small bit of code, but it's another little gem that I didn't know exists. There is only one function:

• python-module-path
  

  This function uses emacs.modpath to determine the module path for use with ffap-alist
  

What this gets us is support for M-x find-file-at-point.

It's not *quite* as cool as I first anticipated because the file path finder seems to get confused about what is actually a module, but if point is on a module name on an import line then it will pop you directly over to the python lib directory and open the module. If you already have etags set up to do this then you probably won't be *too* amazed, but this is a nice extra way to accomplish the same thing.

emacs python mode from scratch: stage 16 - completion

This is a collection of functions that uses emacs.py to find a list of likely completions for module.

python-imports
python-find-imports
python-symbol-completions
python-partial-symbol 
python-complete-symbol
python-try-complete

• python-imports
  

  A list of python import statements in the buffer
  

• python-find-imports
  

  Populate the above list. search by looking for "^import" or "^from" (skipping those in comments or strings).
  

  Then reverse the list, clean out text properties and change \n to \\n so that output doesn't end up wrong.
  

• python-symbol-completions
  

  Run emacs.complete(symbol, python-imports) and return the list of completions (in another buffer)
  

• python-partial-symbol
  

  Use a regular expression to go backward and thereby find the complete symbol before point. This seems a little odd to me. I seems like it would be easier to just trust the syntax-table that has already been defined for the mode and just do backward-word from the current position (with a save-excursion). As a guess I'd say it's so that it can treat "." as part of the symbol and skip over that to the beginning of a module/class name.
  

• python-complete-symbol
  

  Find the list of completions (using python-symbol-completions) or just scroll the completions window if they've already been found.
  

• python-try-complete
  

  "hippie-expand" version for doing symbol completions. Basically "he-" does all the heavy lifting hereit just needs to be
  given the python-symbol-completions and python-partial-symbol functions.
  

emacs python mode from scratch: stage 15 - info look

Probably the main lesson of my explorations of the python mode is that there is a lot of functionality available that I had never heard of before. info-look is a good case in point.

• python-after-info-look
  

  This function essentially let's you hook into the info system to look for documentation.
  

  As far as I can tell the main use is for calling info-lookup-symbol (C-h S) and having the info page for the python entity show up.
  

  You need to make sure the info flavor of python documentation is installed on your system. (And hopefully it takes you less time than it did me to realize it wasn't).
  

Not much else to say on this.

emacs python mode from scratch: stage 14 - context sensitive help

Here are the relevant variables and function in the context-sensitive help section

VARS:
(defconst db-python-dotty-syntax-table
(defvar view-return-to-alist)
(defvar db-python-imports)

FUNCS:
python-describe-symbol
python-send-receive
python-eldoc-function

• db-python-dotty-syntax-table
  

  This creates an altered syntax table that treats "." as "symbol constituent". In other words, "." is treated as part of the variable name rather than as an other type of syntax.
  

• view-return-to-alist
  

  Used by python-describe symbol. Consists of a window and help-return-method
  

• db-python-imports
  

  Used below when calling emacs.* helper functions.
  

• python-describe-symbol
  

  C-c C-f is bound to this and will do a python help(module|func|etc) for thing at point. It dumps out the standard python documentation in another temporary buffer. It uses emacs.py to get the help. Somewhat useful, but i usually have a python prompt open already.
  

• python-send-receive
  

  Convenience function for sending a command to the python interpreter and reading the result.
  

• python-eldoc-function
  

  Use emacs's own documentation reader to populate the message line with brief description of function and args. this is something that I had not noticed before and am glad I came across it while perusing the code.
  

  If you have a python session going and you either run M-x eldoc-mode or you set up a hook such as
  

  
    (add-hook 'python-mode-hook 'turn-on-eldoc-mode)
  

  
  

  then this will automatically run the eldoc help facility. Not sure how useful it will be day to day (maybe it's just developer eye candy), but I've seen this sort of thing while running in slime or in the haskell mode and now at least I see where it is coming from. Seems like it only really works with core python libraries. Strangely doesn't seem to know about the functions in the actual file you are currently working in.
  

Only 4 sections left to walk through. I'm actually pretty eager to be done with this, but I definitely want to finish (however shallow my exploration of this mode is).

emacs python mode from scratch: stage 13 - python inferior mode

My plan was to look at context sensitive help next but it dependeds on a python inferior process so let's get that working next.

Here are the function/variables of relevance:

(defcustom python-python-command "python"
(defcustom python-jython-command "jython"
(defvar python-command python-python-command
(defvar python-buffer nil
(defconst python-compilation-regexp-alist
(defvar inferior-python-mode-map
(defvar inferior-python-mode-syntax-table
(define-derived-mode inferior-python-mode comint-mode "Inferior Python"
(defcustom inferior-python-filter-regexp "\\`\\s-*\\S-?\\S-?\\s-*\\'"
(defun python-input-filter (str)
(defun python-args-to-list (string)
(defvar python-preoutput-result nil
(defvar python-preoutput-leftover nil)
(defvar python-preoutput-skip-next-prompt nil)
(defun python-preoutput-filter (s)
(defun run-python (&optional cmd noshow new)
(defun python-send-command (command)
(defun python-send-region (start end)
(defun python-send-string (string)
(defun python-send-buffer ()
(defun python-send-defun ()
(defun python-switch-to-python (eob-p)
(defun python-send-region-and-go (start end)
(defcustom python-source-modes '(python-mode jython-mode)
(defvar python-prev-dir/file nil
(defun python-load-file (file-name)
(defun python-proc ()
(defun python-set-proc ()

The main mechanism for creating a python inferior process appears to be creating a derived mode based on comint-mode (defined in comint.el). Most of the code then is defining helper functions for making this mode python aware.

So without further ago, here is a brief look at the above functions:

• python-python-command
  
• python-jython-command
  

  Variables which allow customization of what command to use for invoking python (jython)
  

• python-command
  

  The actual command (probably one of the above) that will actually be executed by run-python
  

• python-buffer
  

  The python buffer that will be the target of code issued from files in python-mode
  

• python-compilation-regexp-alist
  

  compilation-error-regexp-alist is set to this value. This is set by inferior-python-mode and is used by compile.el. This regular expression basically matches a python exception stacktrace:
  

  
     Traceback (most recent call last):
       File "foo.py", line 13, in 
         1/0
     ZeroDivisionError: integer division or modulo by zero
  

  
  

  Honestly I'm a little confused by this since I don't think this regex actually matches the above and I'm not clear what purpose "compile" would need with it.
  

• inferior-python-mode-map
  

  Add a few keys for loading a file and doing pychecker. The key sequence for pychecker seems superfluous here. I never feel like kicking off pychecker while i'm in an interacive session.
  

• inferior-python-mode-syntax-table
  

  Adjust syntax table so that single quotes don't mess up the syntax coloring. "." is the syntax table entry for punctuation-like things.
  

• inferior-python-mode
  

  This is where the python interactive mode is defined. It overrides the default commint-mode. There is a comment here that the "python-mode" type things (keybindings, etc) should be inherited from python mode itself.
  

  The basic customization is to adjust some regular expressions that are used by commint to decide what sort of things get maintained in the history, what the "prompt" looks like (>>>), etc.
  

• inferior-python-filter-regexp
  

  The default is not to save things in the history if they are 3 or less in length. I hadn't noticed this behavior before.
  

  I'm not sure if there is some rhyme or reason why sometimes they use the rx style and sometime they use traditional emacs style regexps (in this case "\\`\\s-*\\S-?\\S-?\\s-*\\'").
  

• python-input-filter
  

  This basically is a wrapper for ignoring inferior-python-filter-regexp
  

• python-args-to-list
  

  This will be used by run-python below to collect args for kicking off a python process. Doesn't work with quoted white space. Don't know emacs lisp well at all but it seems like a complicated way to just tokenize on SPACE and TAB.
  

• python-preoutput-result
  

  Not sure what "_emacs_out" is about, and not sure I care to spend the time tracking this down. (Possibly for emacs.py)
  

• python-preoutput-leftover
  

  related to _emacs_out
  

• python-preoutput-skip-next-prompt
  

  This (as well as previous 2 functions) are used by python-preoutput-filter
  

• python-preoutput-filter
  

  This function appears to clean up the output a bit and prevent spurious >>> ... ... >>> from littering the output. I'm going to punt on this. I'm not especially interested in the logic here but it is interesting passing to see a complex function needed to make the python mode behave in a friendly fashion.
  

• run-python
  

  Start a new python process or use an existing one if available. runs python-command with -i.
  

• python-send-command
  

  A wrapper around python-send-string that is used by python-send-region and python-load-file
  

• python-send-region
  

  Copy a section of code to temporary file and evaluate it. Can't do it directly in the interpreter since empty lines will confuse it
  

• python-send-string
  

  Evaluate a python string in the buffer. This function checks for trailing \n or intermediate \t and throws in an extra \n so that the command is properly terminated in a way that the interpreter is expecting.
  

• python-send-buffer
  

  Wrap python-send-region but use the entire buffer as the region
  

• python-send-defun
  

  Send a region but use beginning-of-defun/end-of-defun to delimit the region
  

• python-switch-to-python
  

  Create a new python process if necessary and switch to it. Giving it an argument makes it go to the end of the buffer
  

• python-send-region-and-go
  

  Combine python-send-region and python-switch-to-python
  

• python-source-modes
  

  A list of mode names so we can automatically tell if the buffer has python code in it
  

• python-prev-dir/file
  

  A cache of the directory and file used by python-load-file so that it has a default value if necessary
  

• python-load-file
  

  Import a python file in its entirety into the python process. Uses emacs.py if the file ends in .py otherwise uses "execfile"
  

• python-proc
  

  Return to or create and move to a python process
  

• python-set-proc
  

  Associate the python-buffer with the current buffer (which is a python process). This is used by things like python-send-region so they know where to send output.
  

I think unless you understand the details of how comint works (which I don't) it's hard to get a strong feel for how the inferior-python-mode works. As always it's interesting to see all of the crazy details that have to be taken care of to get a simple looking thing like an embedded python interpreter working.

That was a very cursory overview of how the python process inferior mode works (even by my already lax standards). I admit it, I'm getting a little exhausted by this project. But I'm going to keep slogging through and finish doing at least a cursory inspection of every line in python.el.

Another thing that is becoming clear is that, while this process is helping me understand emacs and emacslisp, I have a lot of work to do, and at some point I really need to just sit down and learn emacslisp in a focused way.

emacs python mode from scratch: stage 12 - bicycle repair man

For this session we'll get bicycle repair man support working.

First let's see if we can get brm working by itself and then see how it integrates into emacs.

To start it I ran:

M-x python-setup-brm

It took a little googling and experimentation to figure out how to use this. But I finally figured out that if you select some text then the menus do their thing.

This is a strange little tool. I guess I'm so used to refactoring with global search and replaces that this seems overly constraining. But perhaps it's just unfamiliar. I suspect that if you got used to it then you could learn to like it, but I'm skeptical. One behavior that was very counter intuitive is that you can't use the normal emacs undo to revert the refactoring change. You have to do it via the menus (or presumably the brm console). In any case let's look at the code that runs brm.

Firstly it does an autoload of pymacs.

I've never really played with this before but you can see that it is loaded correctly and working by doing the following in *scratch*:

(pymacs-eval "2 + 2")
4

The second prep step is to get the bicycle repair man module itself loaded:

(autoload 'brm-init "bikemacs")

Which I'm assuming works since I was able to do a refactoring

• python-setup-brm
  

  This code is mostly just busy work to set up of menus.
  

  Strangely this code ends with an (error ... ) embedded in an (error ...).
  Not sure what that is about.
  

Interestingly in the comments the author of python.el expresses doubts that brm is useful.
Note to self, I should also look at ropemacs. Looks like rope has more features and if I recall an earlier experiment correctly does autocomplete of python variables/modules as well.

Any way, onward and upward. Next stop, context sensitive help.

emacs python mode from scratch: stage 11 - skeleton

We are actually getting to the point where this mode would actually be useful for developing with.

Next let's get skeleton support working now.

Here are the new functions I've copied over:

python-use-skeletons
python-skeletons
python-mode-abbrev-table
def-python-skeleton
def-python-skeleton if
define-skeleton python-else
def-python-skeleton while
def-python-skeleton for
def-python-skeleton try/except
define-skeleton python-target
def-python-skeleton try/finally
def-python-skeleton def
def-python-skeleton class
python-default-template "if"
python-expand-template

• python-use-skeletons
  

  This controls whether or not skeleton functions are added to the python-mode-abbrev-table
  

• python-skeletons
  

  A list of all the skeletons. apparently used when working on compound statements.
  

• python-mode-abbrev-table
  

  This is where the actual abbrevs are stored
  

• def-python-skeleton
  

  A macro for getting a skeletons "registered" in the abbrev system. First it creates a name for the skeleton of the form: python-insert-foo. Then if adds this name to the pyton-mode-abbrev-table. Finally it calls define-skeleton and creates the skeleton.
  

• Each of the following create the appropriate skeleton by name
  

  
  def-python-skeleton if
  def-python-skeleton while
  def-python-skeleton for
  def-python-skeleton try/except
  def-python-skeleton try/finally
  def-python-skeleton def
  def-python-skeleton class
  

• define-skeleton python-else
  

  else clause used by if and other else-able statements. One reason I would find it hard to get excited about using this particular skeleton/template system is that it seems sort of ridiculous to be prompted after a for loop or a while loop, etc for an else clause. I can count on 1 hand how many times i've ever used an else clause other than part of an if cascade. It seems like an unnecessary burden to make the user consider this case every time they are writing a for loop.
  

• define-skeleton python-target
  

  Helper used by try/except
  

• python-default-template
  

  Variable that defines what template to use by default in python-expand-template
  

• python-expand-template
  

  This function is for allowing a skeleton to be used directly as a key sequence rather than as an abbrev. Interestingly if defaults to the "if" template but then let's you update what the default is.
  

I'm not totally sold on the value of skeletons for python. The syntax is already so minimal that the distraction of writing code through the minibuffer doesn't seem like a big win but it's something i'd like to explore in detail sometime and see if it works for me if i give it a chance.

Next let's get bicycle repair man support working.

emacs python mode from scratch: stage 10 - pychecker

My next step is to get pychecker working. Turns out it's not much code. Here are the functions:

• python-check-command
  

  the default string to use as the command for running pychecker
  

• python-saved-check-command
  

  a cache of the command above and the name of the file we are
  currently working with
  

• python-check
  

  Pretty straightforward but a couple of things to note.
  

  I wasn't aware that the "interactive" command took straight elisp as an argument. I had just assumed you had to use the magic argument strings.
  

  It's a little strange that they use an old school emacs lisp regular expression ("(\\([^,]+\\), line \\([0-9]+\\))") rather than the rx style they seemed to use everywhere else.

A note on pychecker. I find this utility fascinating. For good and bad reasons. It's really quite useful. C-c C-w is reflexively typed after saving a file. And the fact that it is so useful makes it extra confusing that it it seems sort of abandoned (at least last I checked).

As a future project I'd like to look at it (or one of it's cohorts, pylint, pyflakes, etc) and see how it works and perhaps complain less and help more,but it seems strangely abandoned now. I did start to take a look at one point and it seemed very complex so I ran away with my tail between my legs. But fortunately I have a short memory for scary things and I'll likely return to it someday.

Getting pychecker working required less code than I suspected.

Next I'll do skeleton support which if nothing else is a lot more lines of code.

emacs python mode from scratch: stage 9 - rest of the python indent code

So my idea was to move onto something completely new but wouldn't you know it, I found a couple more indentation related functions that needed dealing with.

The functions/variables in question are:

python-guess-indent (both variable and function with this name)
python-guess-indent
python-beginning-of-defun
python-end-of-defun
python-backspace

And just for fun we'll throw in the key mapping for python-backspace

(define-key map "\177" 'python-backspace)

So let's quickly browse through these functions and see how they work and make sure each works as promised.

• python-guess-indent (variable)
  

  Seems weird to me that there can be a variable and function with the same name. I'm guessing this is a due to emacslisp being a lisp-2
  

  This guess seems to be confirmed by: http://coding.derkeiler.com/Archive/Lisp/comp.lang.lisp/2008-01/msg00683.html
  

  In any case having this being true means that python mode will be using the following indentation guessing function.
  

• python-guess-indent (function)
  

  This function does what it's name implies.
  

  Looks for a ":" character and then moves to the beginning of that line. Then it moves to the following line and takes the difference. If the different is >= 2 and <= 8 then it take that as a guess. Otherwise it continues to the next ":".
  

  If the value it found is different from the default value then it sets it to this new value.
  

• python-beginning-of-defun
  

  Look for a regular expression consisting of space and either def or class.
  

• python-end-of-defun
  

  This is a little trickier than python-beginning-of-defun. Here's another place where python's whitespace makes life hard. Good thing it's pretty to look at.
  

  First find the beginning of the block we are currently in. If we are at the beginning of a line then but not at the start of a block, move forward until we find a block starting line.
  

  From here python-end-of-block does the rest of the work.
  

• python-backspace
  

  Look at the list of valid indentation levels for the current line and then back track to the next smallest.
  

  Also present a message explaining what block you just closed.
  

I'm now 40% through. Almost starting to seem like I might make it all the way through this puppy.

emacs python mode from scratch: stage 8 - python-indent-line

OK, now that we have python-calculate-indentation finished let's see who calls it (and hopefully work our way up to the point that tab cycling actually works).

So here are the call chains for python-calculate-indentation and a couple of related functions.

python-indent-line
   python-indent-line-1
      python-calculate-indentation
   python-block-end-p

python-indent-region
   python-indent-line-1
      python-calculate-indentation

python-electric-colon
   python-calculate-indentation

So our new functions we will bring over this time are:

python-block-end-p
python-indent-line-1
python-indent-line
python-indent-region
python-electric-colon

That's about 90 lines so that's good for one session. In addition to these functions we'll add a in the key mapping for python-electric-colon and the indent assignments within define-derived-mode (indent-line-function, indent-region-function).

Well, I don't know yet know *why*, but I can confirm that tab cycling works as I expected. Hopefully it will be clear as I go through the code what exactly is happening.

As for the electric colon, after experimenting around I was able to figure out a scenario which makes a line of code "outdent" as promised:

if foo:
    bar
    else:  

My first instinct is to try to understand the tab cycling by digging down from the top and seeing how the tab key itself is bound.

TAB (translated from ) runs the command indent-for-tab-command
  which is an interactive compiled Lisp function in `indent.el'.
It is bound to TAB.
(indent-for-tab-command &optional arg)

Indent line in proper way for current major mode or insert a tab.
Depending on `tab-always-indent', either insert a tab or indent.
If initial point was within line's indentation, position after
the indentation.  Else stay at same point in text.
The function actually called to indent is determined by the value of
`indent-line-function'.

So this points us to indent.el which is where the magic happens and defines why the major mode needs to define indent-line-function and indent-region-function.

My next step was to look for descriptions of this indent functionality in the emacs manual but I quickly got tired of poring over that. So back to looking at the code in the library.

So let's work through the functions above:

python-block-end-p
python-indent-line-1
python-indent-line
python-indent-region
python-electric-colon

• python-block-end-p
  

  Check to see if the current line is the last line of an indented block. It's sort of surprising that this functionality hasn't been needed so far.
  

  It checks that the current line is not a comment and that that either: it's for sure the end of a block (e.g. raise, etc) or the current indentation is less that the previous line's indentation.
  

  That seems a little loose on the meaning of what I would expect a block-end checker to do. But from looking at how it is used in context I see that it probably is sufficient for it purpose. Basically it is simply being used to determine whether or not to display a message about what block is being closed by the position that the current indenting level. So close enough. Might be confusing if someone tried to use this function more generally.
  

• python-indent-line-1
  

  The comments for this function indicate that this is how indenting would be done if there was no cycling. Which essentially means it will take the last value from the full list of possibilities and just use that.
  

• python-indent-line
  

  So this is essentially the money function for doing tab cycling and now that I know so much more about the work going on underneath the covers it seems comically short. And I certainly wouldn't have really understood just how magical this function is without looking at the gory details myself.
  

  The basic logic is to check if we are already cycling. If so then we move to the next known position and try to show what code block we are closing if that makes sense.
  

  If we are just starting a tab cycle then we just call python-indent-line-1 and set up python-indent-index to be ready for cycling.
  

• python-indent-region
  

  So this function is a little strange. According to the emacs manual indent-region-function is used to act as a fast way to indent lines. The default behavior is to just run the indent-line-function on each line. But this code just goes ahead and loops over every line and just runs python-indent-line for each. Nothing obvious in this code to indicate what the advantage of walking over each line by hand would be. A mystery.
  

• python-electric-colon
  

  So this guy is mapped to ":" and usually just inserts a ":" unless we have the case I outlined above. The basic logic is to check if we are at the end of a line, whether we are in an outdentable context (python-outdent-p), not in a string or comment and we are past the point that it is predicted we should be indented to.
  

  This function is followed by:
  

  (put 'python-electric-colon 'delete-selection t)

  
  

  Which as far as I can tell is related to delsel.el. Apparently delete-selection is used by this mode when you are deleting selected text a la windows behavior (ie delete selected text when you type).
  

Now that I have tab cycling pretty much wrapped up I have this hope that much of the rest of the code will be a little more straight forward.What can I say, I'm an optimist at heart.

As far as code coverage goes, I've copied over 33% of the code. Wow, I'm a third of the way done.

emacs python mode from scratch: stage 7 - python-calculate-indentation

We continue with our task of trying to get tabbing working by copying over python-calculate-indentation and examining how it works.

First thing that is apparent is that we also need these variables to be defined and set:

• python-indent-string-contents
  
  • if true, then for doc strings do alignment
  
  
• python-continuation-offset
  
  • how many spaces to add if indenting within a line following a continuation (ie "\")
  
  
• python-honour-comment-indentation
  
  • go ahead and align a comment with a previous comment even if it's not itself in a "normal" position
  
  

So on a first glance python-calculate-indentation doesn't seem to work as i would have expected. I just get one position back and the value doesn't change on successive runs even if there are various logical candidates. So my guess is that this function returns the best first guess and then a higher level function calling this will changes some global variable so that this function will return the next best and so on until it cycles through.

So let's see what is really going and and how good my guess is.

This function (similarly to python-indentation-levels) is just one big "cond". The various sections are:

• multi-line string
  
• after a backslash or in brackets (ie continuation context)
  
• beginning of buffer
  
• non-indentable comment
  
• the rest
  

The number of new cases here is a bit of a surprise since I had half expected that the previous function we looked at (python-indentation-levels) already handled the main cases. But I guess not.

• multi-line string
  

  If we are currently in a string then do one of two things. If we have set our preferences so that we don't want to indent string contents, then just take the current indentation.
  

  Otherwise if we are on a line that starts the string, then just keep the current indentation. The actual work horse of this section is to keep walking backward a line at a time until we find a non-blank line and then use that as the indentation level.
  

• after backslash or in brackets
  

  The logic here might be best summarized with pseudo code
  

  
     if in bracketed expression
         find first item in list and align with that
         otherwise align with the start statement (plus a "step" per bracketing level)
     else # must be backslash continuation line
         if this is not the first continuation line
            take indent of previous line
         else
            indent one "step" (plus even one more if previous line
            is a opening block)
  

  
  

  Fortunately this function is pretty well documented or it would be a bear to get through. Well, it's still a bear, but it's less of a bear. I wonder if instead of good documentation this would be a good place to create well named smaller functions. My personal style is to aim for functions that are the width of my hand (or two). Not being able to see the function in its entirety on one screen is pretty obnoxious to me.
  

  Note: needed to remember that "if" has one clause for the then statement (hence we find a progn) but the else can be any number of clauses. Strange asymmetry.
  

• beginning of buffer
  

  No choice here. It's gotta be 0.
  

• non-indentable comment
  

  There is a note here that basically says that they copied this behavior from python-mode.el but that it's not necessarily a good idea to have done so.
  

  A non-indentable comment seems to be a comment line that has no space-like characters between the comment starter ("#") and the text
  

  
      e.g.
         # hello
         #there
  

  
  

  That seems weird. But now I know to look for this behavior when editing.
  

• the rest.
  

  So this is weird. This is where the last function we looked at (python-indentation-levels) will get called. but at this point I'm confused about how how they interact. Any way let's wade through this and see what we can figure out
  

  There are four main items here
  

  
  
  • move backward over white space if need to
    
  • call our friend python-indentation-levels
    
  • prefer indenting of comments with a following statement
    
  • take the caar (car of the car) of the last thing in python-indent-list
    
  
  

I have to admit at this point that I can mostly grok the code but the actual details are quite daunting. I would be hard pressed to recreate this logic from scratch. Also I'm still completely nonplussed as to how the magic cycling of tabbing options occurs.

Peeking around the original file I see a reference to cycling so I hope things get more clear when I get to that. But, once again we've been through a bunch of code and we still don't have a working tab cycler yet. But I think we are within spitting distance now.

For what it's worth, I'm now about 30% of the way through the code.

emacs python mode from scratch: stage 6 - python-indentation-levels

OK now let's see if python-indentation-levels works as advertised when it is copied over.

For the following test code block (with cursor at position POINT):

class Foo(object):
    def __init__(self, *args, **kwargs):
        print "hi"
        print 'qwerty' [POINT]

it returns the following list:

((0 . #("class Foo(object):" 0 5 (face font-lock-keyword-face fontified t) 5 6 (fontified t) 6 9 (face font-lock-type-face fontified t) 9 18 (fontified t)))
 (4 . #("def __init__(self, *args, **kwargs):" 0 3 (face font-lock-keyword-face fontified t) 3 4 (fontified t) 4 12 (face font-lock-function-name-face fontified t) 12 13 (fontified t) 13 17 (face font-lock-keyword-face fontified t) 17 36 (fontified t)))
 (8 . #("print \"hi\"" 0 5 (face font-lock-keyword-face fontified t) 5 6 (fontified t) 6 10 (face font-lock-string-face fontified t))))

This is a list of lists where each internal list consists of a pair of tab position and the object with which it is matching.

So everything seems to be functioning correctly,now we just need to delve into how this works exactly.

Basically the whole function is a cond with three cases:

• statement following a block open statement
  
• comment following a comment
  
• everything else
  

Before we go through these three cases I have to say that the style is very new to me. The "predicate" part for the first two cases of each case branch is a block of code that acts as if what it is testing is true. If it succeeds then it is true and it sets the indent list appropriately, otherwise it tries the next case. It is just an unusual style (to me) to run code in a "if" statement like this. But it obviously works, so I'll just adjust my expectations accordingly.

• statement following a block open statement
  

  The logic here is to that check all of the following actions/tests work:
  

  
  
  • move to a previous statement
    
  • check that it is an opening block statement
    
  • save the value of the indent
    
  • move to the end of the statement
    
  • skip comments and blanks
    
  • make sure we are at a ":"
    
  • add the fixed indent amount to the indent of the previous block statement
    
  
  

  I don't really understand all the machinations here.
  

  Intuitively I would have stopped after the first 3 steps.
  

  Ahh... now I see. The comments are useful here.
  

  
     ;; Check we don't have something like:
     ;;   if ...: ...
  

  So if we go to the end of the statement and don't find a ":" we have the above scenario and the "normal" indenting rule won't work.
  

• comment following a comment

  This is more straightforward. If the current line is a comment and the previous line is also a comment, then there is only one choice for indent levels: the indent level of the previous line.
  

• all other cases.

  This logic doesn't look as bad as I would have at first suspected.
  

  The first thing added to the list of indentation levels will be the position of the previous lines indentation *if* the previous line is part of a pair like if/else that makes sense to line up with AND it is not a block closer (e.g. return) that doesn't make sense to line up with
  

  Next we are going to crawl up a block at a time and collect indentation levels on the way up. We only skip a level if we had a word like "else" and the block we are examining doesn't match our "start" word
  

  Even if we had nothing we throw a 0 position on the list for good measure and then set a couple global values (python-indent-list and python-indent-list-length)
  

So that was one of the first "big" functions I've had to work through and it wasn't too bad. It's amazing how many functions were required do something as simple sounding as get suggested indentation levels.

And of course we *still* can't indent. So the next phase will hopefully be using the output of this function to actually navigate using tab. I would never have guessed so much magic was going on when the tab key was hit.

I will need to keep in mind that these values are being set globally and see if I can guess why. As a guess I'd say its because this operation is expensive and while you are on a line there is no need to recalculate it. If that's true I should at some point see some code that recognizes that point has moved to a new line and invalidates the current python-indent-list

We are now 23% of the way through the code (by lines - including comments).

emacs python mode from scratch: stage 5 - more movement methods

So let's continue with the seemingly modest goal of getting tabbing to work

Presumably my current target is to get python-indent-line working

From some quick browsing this function has the following important dependencies:

db-python-indent-line
  db-python-indent-line-1 ;; which sets global: db-python-indent-list-length
    db-python-calculate-indentation (104 lines)

And then looking at python-calculate-indentation we find it requires

   - db-python-indent-string-contents # global var
   - db-python-indentation-levels  # 59 lines

So I guess this time our target is to get the support functions in place for python-indentation-levels.

- python-indent
- python-block-pairs

and the function(s)

- python-first-word
- python-initial-text
- python-beginning-of-block
- python-end-of-block

• python-indent:

  This is pretty straight forward, set a customizable variable for what the default number of columns for indentation will be. Just to be thorough we need to look up what safe-local-variable means.

  From poking around in files.el

  
  

  
  ;; Safe local variables:
  ;;
  ;; For variables defined by major modes, the safety declarations can go into
  ;; the major mode's file, since that will be loaded before file variables are
  ;; processed.
  ;;
  ;; For variables defined by minor modes, put the safety declarations in the
  ;; file defining the minor mode after the defcustom/defvar using an autoload
  ;; cookie, e.g.:
  ;;
  ;;   ;;;###autoload(put 'variable 'safe-local-variable 'stringp)
  ;;
  ;; Otherwise, when Emacs visits a file specifying that local variable, the
  ;; minor mode file may not be loaded yet.
  ;;
  ;; For variables defined in the C source code the declaration should go here:
  

  So basically it's a way to do some simple type checking on a variable.

• python-block-pairs

  This is an alist of python keywords and the keywords that they are the "closers" for.

• python-first-word

  A simple function that returns to the beginning of code for a line and calls current-word

• python-initial-text

  A function to grab the non-comment code on a line. Interestingly this function doesn't seem to work as advertised. Both in my hand copied subset of code and in the full working original both seem to keep the comments at the end of the line.

  I will have to keep this in mind when looking at how it is used to see if this will affect the functionality.

  
  
• python-beginning-of-block

  Move point to beginning of containing block. It starts by moving past any blank space and/or comments and then going to start of current statement.

  Then while point is not on the 0th column and/or the arg passed in is not 0 continue recursively until one of the target conditions is true

  The logic here is a bit of a challenge (for me) to follow. Its a twisty maze of whiles, whens, ands, recursion and throw/catches.

  It seems to be doing something like:

  Move up a line and continue doing it until either we can't go up any more or we hit the condition where we generate a 'done (ie the conditions match a new outer block have been hit).

  
  

  This is probably the most "lispy" code so far and seemed excessively weird when I first tried to wrap my brain around it. But it seems halfway sensible to me now

  
  
• python-end-of-block

  For some reason I was expecting this function to be a simple variation of python-beginning-of-block, but the logic is fairly different. It *does* use many of the same function but the way they are combined is not a simple reverse of the above logic.

  This function and the above are definitely a bit more sophisticated in their logic and the code is more strange (to me). As an example:

  within a let* expression there is the following "variable assignent" which is really not a variable assignment.

  
  

  
     (_ (if (db-python-comment-line-p)
            (db-python-skip-comments/blanks t)))
  

  At this point I'm not sure whether to think this is an ugly hack or just to try and get used to this as idomatic emacslisp. Seems like it would be cleaner just to do this test and function call before the let*, or within a nested let. Hmmm... Can't do it before because we need the value of point before moving and having a nested let makes the code more complicated. I guess I'm already starting to see this more as a clever hack than first blush.

  More strangely, python-beginning-of-block is called recursively and this function is an iteration. I wonder if these were done by different people or in different moods. Or perhaps there is some compelling reason not to do it recursively. Not obvious to me in any case why the difference in style.

Anyway, we are now closer to being able to implement some tab bevhavior. With any luck next phase will have us implementing python-indentation-levels

By lines of code I'm about 20% of the way through.

emacs python mode from scratch: stage 4 - utilities (the rest)

Let's continue from the last entry and finish off the utilities.

The rest of the utilities:

(defun python-comment-line-p ()
(defun python-blank-line-p ()
(defun python-beginning-of-string ()
(defun python-open-block-statement-p (&optional bos)
(defun python-close-block-statement-p (&optional bos)
(defun python-outdent-p ()

python-comment-line-p

• Go to the end of the line, if the emacs parser says we we are in a comment, go to the beginning of the line per indentation and check if we are looking at a comment or the end of a line.
• This is weird. Seems like there is no way for this to ever be an end of line.
• comment-start seems to be a symbol required by rx not necessarily a value set in define-derived-mode (which I'm not setting yet)
• As an aside, the regex for start of comment is \s<
• I don't think that line-end is a necessary part of that regular expression or at least if i take it out it doesn't seem to change the behavior but I'll just leave it in for now.
• Presumably we are just depending on the comment character set in python-mode-syntax-table

python-blank-line-p 

• This is about as straight forward as you get. Go to the beginning of the line and check if you are looking at 0 or more white space chars followed by an end of line. \\s- is the ever so strange looking way emacs regular expressions represent white space.

python-beginning-of-string 

• Hey, this stuff is starting to look familiar. First we determine what state a parser would be at the current position. Then if the parser says we are in a string, we go the the starting point (8th item of the state list)

python-open-block-statement-p

• So this and python-close-block-statement-p and python-outdent-p call a number of things that aren't already defined which is strange since you'd think some functions described as utilities would be self contained. The additional functions are:

  python-beginning-of-statement and python-previous-statement

  So we'll assume these functions do what they claim for now and investigate them more closely down below
• This logic is pretty straight forward. Go to the beginning of the statement and then use a regular expression match to see if we are at the beginning of a block.
• NOTE: there is an optional parameter that let's you skip the movement to the beginning of the statement. Seems like a simple performance helper.

python-close-block-statement-p

• This is the exact analog of python-open-block-statement-p. Except here we check to see if it is something that for sure is the last member of a block.

python-outdent-p

• Check if current line should be "outdented". Again pretty straight forward
  code. Move to the beginning of the the current indentation and check if all of the following are true: looking at (else, finally, except or elif) and not in a comment or string and check that the previous statement is neither a close block nor an open block.
• In other words if on something like an else statement and the previous statement is not something that requires indentation you should outdent.
• Seems like this is just sort of a heuristic and not quite accurate.
  E.g. for an improperly nested "else" following a return. It will say not to outdent. Of course, what *should* you do when the code is invalid?

python-beginning-of-statement

• (this requires python-skip-out so we add that to the list as well)
  Here we will move to the beginning of the line and then if on a continuation of some sort we'll either check if we are on a backslash style of continuation and move backward over these or we will move backward over strings and "skip out" (jump up) from nested brackets.

python-previous-statement

• (this needs python-next-statement so we add that to the list as well)
  If it receives a negative argument they really want to go forward, so pass off to python-next-statement. Otherwise, go to the beginning of the statement and skip over comments and blanks and continue going to the beginnnig of the statement until counter is 0 (or reach beginning of buffer)

python-skip-out

• Pop up out of nested brackets to the front by default and to the
  end if "forward" is set. Additionally if "syntax" of point is already available, then that can be passed in. For well formed nesting we just call backward-up-list with the depth. For ill-matched brackets we try to go backward up the list over and over until we get an error.

python-next-statement

• This is pretty much an exact analogue of python-previous-statement.

There were a surprising number of functions that the utilities them self depended on. I continue to be amazed by the complexity of making a language sensitive mode for emacs.

I also noticed that I'm over 10% of the way done. Making progress.

emacs python mode from scratch: stage 3 - some utilities

As I mentioned last time my plan was to do tabbing/indentation next but there was a lot of reliance on the utility section of code so the next step becomes getting these helper functions copied over and understood.

There are about 100 lines in the utility section so let's do half at a time. In the first 50 lines we have the following functions/definitions:

(defsubst python-in-string/comment ()
(defconst python-space-backslash-table
(defun python-skip-comments/blanks (&optional backward)
(defun python-backslash-continuation-line-p ()
(defun python-continuation-line-p ()

python-in-string/comment

• defsubst is a marcro for inlining a function which seems strange. I wonder if this was just a performance booster. (Is there any other reason to inline code?)
  
• syntax-ppss returns info on what the parser state would be at the current position. Below are the states and its clear why nth 8 is the thing we need for determining if we are in a string or comment
  

  
    0. The depth in parentheses, counting from 0. Warning: this can be negative if there are more close parens than open parens between the start of the defun and point.
    1. The character position of the start of the innermost parenthetical grouping containing the stopping point; nil if none.
    2. The character position of the start of the last complete subexpression terminated; nil if none.
    3. Non-nil if inside a string. More precisely, this is the character that will terminate the string, or t if a generic string delimiter character should term inate it.
    4. t if inside a comment (of either style), or the comment nesting level if inside a kind of comment that can be nested.
    5. t if point is just after a quote character.
    6. The minimum parenthesis depth encountered during this scan.
    7. What kind of comment is active: nil for a comment of style â?oaâ?? or when not inside a comment, t for a comment of style â?ob,â?? and syntax-table for comment that should be ended by a generic comment delimiter character.
    8. The string or comment start position. While inside a comment, this is the position where the comment began; while inside a string, this is the position where the string began. When outside of strings and comments, this element is nil.
    9. Internal data for continuing the parsing. The meaning of this data is subject to change; it is used if you pass this list as the state argument to another call.
  

  
  
• syntax-ppss seems like an insanely powerful feature. My respect for the sophistication going on behind the scenes when emacs opens a file for a certain language continues to grow.
  

python-space-backslash-table and python-skip-comments/blanks

• This seems to be a "throw away" data structure for overriding the *real* syntax table temporarily
  
• It provides a syntax table that redefines "\" as a whitespace class. Presumably this is for allowing movement over otherwise blank continuation lines as if they were whitespace.
  
• What this meant wasn't immediately obvious, but it seems to be addressing the type of situation below (which honestly I haven't seen in the wild before, but seems to be legal python code).
  

  
  if x and \
          \
          \
          \
          \
    y:
     print z
  

  
  
• For movement we will use forward-comment which moves over up to X comments. It moves backward if arg is negative. The forward motion is straight forward. The backward motion starts by positioning point so that it is at the *start* of the comment (probably to avoid having it forward-comment move it to the start of the comment itself which would probably seem like a noop to the user)
  

python-backslash-continuation-line-p

• This is pretty straight forward. Check if the last char on the previous line is a \ and make sure not in a comment or string
  
• syntax-ppss-context is a an undocumented function (but trivial) that checks if the current syntax parsing state is string or comment
  

python-continuation-line-p

• is an extension of the above function and checks for the case of a continuation char (ie the above function) or if in a matching paren type context that is allowed to span multiple lines.
  
• The syntax-ppss-depth function tells you how far nested you are in parens, braces, etc. The interesting part of this function is that if you have unmatched parens then it tries to move up the list and assumes that if you succeed then you must be in something close enough to matching lists after all. I wasn't able come up with a set of braces that tickled this case but otherwise the logic makes sense.
  

So that is the first half of the utility functions.

I'm still overwhelmed by the amount of complexity required to get a language mode working. I haven't even gotten something seemingly simple like tab support working. I guess this is like a lot of things. When you start to understand a new system, many of the hard things turn out to be trivial and many of the seemingly trivial things turn out to be quite complex.

I had never really browsed through the emacs lisp manual before and I'm finding it is quite helpful and not too hard to navigate.

Eventually I'll have to understand a mode for some other language(s) as well. I'm curious how much python's significant whitespace makes makes things harder. Presumably the default language mode behaviors are tuned for something like lisp and/or C.

I also wonder if it would be possible to use pythons own parser in place of emacs for doing some of the syntax checking activities. That would be an interesting project to pursue. As will be looking at the pymacs module.

Settle down there tiger. One thing at a time....

emacs python mode from scratch: stage 2 - comments

So my next plan was to look at indenting/tabbing, but I found almost immediately that (a) it's really bizarrely complicated and (b) it depends (in some of the supporting functions) on comments being recognized correctly. So I'm forced to figure that out now. So, rather than reading the emacs manual (I mean *anyone* could do that) I used the tried and true binary code search technique and started commenting things out (starting with the original code) until font-locking for comments stopped working.

That worked like a charm. The missing piece of the puzzle was this little fella:

(defvar python-mode-syntax-table
  (let ((table (make-syntax-table)))
    ;; Give punctuation syntax to ASCII that normally has symbol
    ;; syntax or has word syntax and isn't a letter.
    (let ((symbol (string-to-syntax "_"))
   (sst (standard-syntax-table)))
      (dotimes (i 128)
 (unless (= i ?_)
   (if (equal symbol (aref sst i))
       (modify-syntax-entry i "." table)))))
    (modify-syntax-entry ?$ "." table)
    (modify-syntax-entry ?% "." table)
    ;; exceptions
    (modify-syntax-entry ?# "<" table)
    (modify-syntax-entry ?\n ">" table)
    (modify-syntax-entry ?' "\"" table)
    (modify-syntax-entry ?` "$" table)
    table))

Strangely you don't have to actually *use* this variable anywhere, it just needs to be defined. If I had to guess off the top of my head, I'd say that define-derived-mode uses this syntax-table if it's available. Otherwise it just falls back to some default behavior.

I was surprised also to see that my friends:

;;   (set (make-local-variable 'parse-sexp-lookup-properties) t)
;;   (set (make-local-variable 'parse-sexp-ignore-comments) t)
;;   (set (make-local-variable 'comment-start) "# ")

still don't seem to be necessary. So they remain commented.

Because I'm now using the block of code above in my "from scratch" python mode, technically I'm supposed to understand it now that I've copied it in. So let's look at what this guy does.

Looking in derived.el we find this little guy:

(defsubst derived-mode-syntax-table-name (mode)
  "Construct a syntax-table name based on a MODE name."
  (intern (concat (symbol-name mode) "-syntax-table")))

So it's not too surprising that it's using the name python-mode-syntax-table somewhat auto-magically.

And to really seal the deal we find in the elisp manual that "<" is the syntax class for comment starter. So that's part of what's getting set when this syntax-table variable is defined.

Also of interest is that initially all symbol constituent chars (except "_") are reassigned to the punctuation character class. This makes sense since python only allows numbers/letters and _ in variable names. Everything else is a type of punctuation.

Additionally we change the syntax class of a few more characters. Notably we tell it that "'" is a quote character and "`" is a paired delimiter.

Cool. Now we have comments that get correctly colored as comments.

emacs python mode from scratch: stage 1 - syntax coloring

First a note on my method. In order to keep my regular python mode functional while I'm playing around, my copy of the python mode uses different function names where possible (prepend "db-" to the normal name) and I'm targeting .pydb files (instead of .py).

So far this seems to work fine. I have the following in my .emacs file:

;; (load-library "db-python-1")
;; (add-to-list 'auto-mode-alist '("\\.pydb\\'" . db-python-mode))

After changes, I run the load-library command and it seems to pick up any changes that I've made recently. (In case it's not clear my stage 1 library is in a file called "db-python-1.el" and it provides db-python-mode). I plan to change the library file name as I progress and leave the db-python-mode name constant.

I am surprised on how much stuff is going on to get a mode working. But even more surprising is how understandable it is. I definitely don't understand all the working parts so far and not everything I've tried has worked but in general the pieces are falling pleasantly into place.

For stage 1, syntax coloring *mostly* works. For whatever reason I wasn't able to get comments to color correctly. My first guess was that one or more of the following:

(set (make-local-variable 'parse-sexp-lookup-properties) t)
(set (make-local-variable 'parse-sexp-ignore-comments) t)
(set (make-local-variable 'comment-start) "# ")

would do it, but so far no avail. But since I didn't get it working that means I don't really understand the lines above so by my rules I can't add them to the file. I'm hoping that at some point as I'm schlepping code from one place to another it *will* start working and hopefully then everything will fall into place.

I'm intrigued by the rx syntax for regular expressions. A little verbose but it might be better than the fairly non-standard emacs regex syntax. It's interesting to look at the actual regular expression that rx produces. For instance the keyword list that is ORed together dumps out as:

"\\(?:\\_<\\(?:False\\|None\\|True\\|a\\(?:nd\\|s\\(?:sert\\)?\\)\\|break\\|continue\\|del\\|e\\(?:l\\(?:if\\|se\\)\\|x\\(?:cept\\|ec\\)\\)\\|f\\(?:inally\\|or\\|rom\\)\\|global\\|i\\(?:mport\\|[fns]\\)\\|lambda\\|not\\|or\\|p\\(?:ass\\|rint\\)\\|r\\(?:aise\\|eturn\\)\\|self\\|try\\|w\\(?:hile\\|ith\\)\\|yield\\)\\_>\\)"

Which seems pretty bad. If you look carefully you can see that it has done some consolidating of terms. I'm not sure how useful it is to do it like that, but if nothing else the rx style is quite readable compared to the above final output.

I also removed db-python-font-lock-syntactic-keywords since adding it in didn't seem to do anything and I really want to understand what all the pieces do.

One thing I had never noticed before is that globals at the top level of the module get a special syntax coloring.

So with no further ado, here is what I have so far:: db-python-1.el

(defvar db-python-font-lock-keywords
  `(,(rx symbol-start
         ;; From v 2.4 reference.
         ;; def and class dealt with separately below
         (or "and" "assert" "break" "continue" "del" "elif" "else"
             "except" "exec" "finally" "for" "from" "global" "if"
             "import" "in" "is" "lambda" "not" "or" "pass" "print"
             "raise" "return" "try" "while" "yield"
             ;; Future keywords
             "as" "None" "with"
             ;; Not real keywords, but close enough to be fontified as such
             "self" "True" "False")
         symbol-end)
    ;; Definitions
    (,(rx symbol-start (group "class") (1+ space) (group (1+ (or word ?_))))
     (1 font-lock-keyword-face) (2 font-lock-type-face))
    (,(rx symbol-start (group "def") (1+ space) (group (1+ (or word ?_))))
     (1 font-lock-keyword-face) (2 font-lock-function-name-face))
    ;; Top-level assignments are worth highlighting.
    (,(rx line-start (group (1+ (or word ?_))) (0+ space) "=")
     (1 font-lock-variable-name-face))
    (,(rx "@" (1+ (or word ?_))) ; decorators
    (0 font-lock-preprocessor-face))))

(define-derived-mode db-python-mode fundamental-mode "(DB)Python-1"
  (set (make-local-variable 'font-lock-defaults)
       '(db-python-font-lock-keywords nil nil nil nil
                                   ;;(font-lock-syntactic-keywords
                                   ;; . db-python-font-lock-syntactic-keywords)
                                   ))
)

(provide 'db-python)