Debugging Emacs Lisp

In this post, I'll show how I used basic capabilities of the built-in Emacs Lisp debugger such as suspending execution, stepping through code, and adding watchpoints to solve a day-to-day issue.

Downloading Problems

I recently stumbled upon an Advent of Code utility which lets you authenticate with the advent of code server via M-x advent-login and download your daily input file with M-x advent-input.

When I ran advent-login followed by advent-input, instead of receiving my input file, Emacs printed an HTTP 400 error in the message area.

To find out why, I opened up advent.el and started with the function definition of advent-login:

(defun advent-login (session)
  "Login to adventofcode.com.
Argument SESSION session cookie value."
  (interactive "sValue of session cookie from logged in browser: ")
  (url-cookie-store "session" session "Thu, 25 Dec 2027 20:17:36 -0000" ".adventofcode.com" "/" t))

I wasn't familiar with the url-cookie-store, or Emacs's URL capabilities in general, so I placed my cursor on url-cookie-store and hit M-. to hop to its definition. I skimmed the source file it's found in, noting the top-level functions and variables and reading their docstrings.

From the source code, I noted that cookies stored in variables url-cookie-storage and url-secure-cookie-storage, and that you can list currently stored cookies with M-x url-cookie-list.

Step Debugging a Function

Next I decided to step through the execution of the function url-cookie-store. To debug a Lisp function in Emacs, you can move your cursor inside the function and enter C-u C-M-x, which executes eval-defun with a prefix argument.

The next time the function is called, Emacs will switch to a source buffer showing the code you are debugging. Execution is stopped, and the editor is awaiting a debugger command.

The function will continue to trigger debugging each time it's run until you re-evaluate it without a prefix argument by placing your cursor inside the function and entering just C-M-x.

After instrumenting url-cookie-store for debugging, I triggered it by invoking advent-input. Here's how that looks in practice:

Stepping through Lisp code follows the rules of applicative order evaluation, meaning form arguments are evaluated first, followed by the form itself.

(There are a few exceptions to the evaluation order, such as the special form if or the macro when, which modify the flow of control, as you can see above.)

Each time an argument or form is evaluated, the result is displayed in the bottom message area.

By repeatedly pressing n, I was able to step through and see that url-cookie-create called, and its return value (stored in the variable tmp) is bound to url-cookie-secure-storage in a setq expression.

After the function finished, I ran url-cookie-list and saw that a cookie for domain .adventofcode.com was indeed created.

With the cookie visible, advent-input still failed with a 400 error.

At this point, I re-ran url-cookie-list and found that the cookie was gone and had been replaced by an unrelated StackOverflow session cookie at some point during the execution of advent-input even though I hadn't visited the site between the two commands.

(defun advent-input (&optional day)
  "Load todays adventofcode.com input in other window.
Optional argument DAY Load this day instead.  Defaults to today."
  (interactive "P")
  (let* ((year (format-time-string "%Y"))
         (day (or day (advent--day)))
         (url (format "https://adventofcode.com/%s/day/%d/input" year day))
         (dir (format "%s/%s/%d" (expand-file-name advent-dir) year day))
         (file (format "%s/input" dir)))
    (if (not (file-exists-p file))
        (url-retrieve url 'advent--download-callback (list file))
      (find-file-other-window file))))

The problem was almost certainly going to be found somewhere in url-retrieve. But rather than hunt through the source, I decided to use a watchpoint to more easily track down what was changing url-cookie-secure-storage.

Adding a Watchpoint

In debuggers, a "watch" refers to a mechanism that helps you observe when a variable changes.

In Emacs, adding a watch to a variable is done with M-x debug-watch, and it will cause the editor to enter the debugger when the target variable changes. You're provided with a stack trace that you can use to explore the problem.

I set a watch on the variable url-cookie-secure-storage, where I knew my advent cookie was stored, and then called advent-input to trigger it:

From the stack trace, I could see that a file ~/.emacs.d/url/cookies was being parsed and loaded, and as a result expression (setq url-cookie-secure-storage 'nil) was being evaluated, which was blowing away the session token I had previously entered.

I opened up ~/.emacs.d/url/cookies and saw what was being loaded and executed:

;; Emacs-W3 HTTP cookies file
;; Automatically generated file!!! DO NOT EDIT!!!

(setq url-cookie-storage
 '((".stackoverflow.com"
  [url-cookie "prov" "dramatic-reenactment" "Fri, 01-Jan-2055 00:00:00 GMT" "/" ".stackoverflow.com" nil]))
)
(setq url-cookie-secure-storage
 'nil)

;; Local Variables:
;; version-control: never
;; no-byte-compile: t
;; End:

I'm not sure how the file was created, but I suspected that removing it would resolve my issue. Once I did, I was able to successfully download my input.

As a bonus, I created a new cookies file that contained my advent of code session cookie to save myself the trouble of logging in.

Thanks for reading!