Owen’s Notes on Clojure

Lisps are a horrible language family because it is entirely up to the programmer to decide how to think about their problem. This problem becomes worse when the poor programmer tries to use libraries - because then they have to figure out if the foreign ideas of other programmers are good or bad.

Here stands my compiled opinions on how to think about Clojure programs. May I not forget to read it from time to time.

Setting up a computer for Clojure Development

I installed babashka and it didn’t work out very well - it proved clunky to integrate with deps.edn. Nevertheless, for projects that can be done with no dependencies (rare for me, since I want access to orchestra) bb avoids the slow startup time of clojure. It requires a local bb.edn rather than a deps.edn. I installed babashka with:

To install certain key libraries, I set up a user-wide deps file in $XDG_CONFIG_HOME/clojure/deps.edn aka ~/.config/clojure/deps.edn. CIDER in Emacs doesn’t know about this file, so I will typically need to duplicate it at the project level to get everything working.

{:deps {cheshire/cheshire {:mvn/version "5.10.0"} ; namespace: cheshire.core
        org.clojure/core.match {:mvn/version "1.0.0"}
        clojure-lanterna/clojure-lanterna {:mvn/version "0.9.7"}
        orchestra/orchestra {:mvn/version "2021.01.01-1"}
        org.clojure/tools.cli {:mvn/version "1.0.206"}
        opar/opar {:local/root "/home/owen/devel/x-clojure/opar/"}
        }}

The last one is a set of personal functions I use a lot. Mainly imperative which combines let, do and cond in an imperative style. You can see the code. This is a later development of the function which started as a variant of cond.

I needed to put a simple deps.edn in the opar folder - {:paths ["."]}. It took a bit of mucking around to verify that this worked. It looked like an absolute path was needed for a while. Eventually a caching problem was surfaced that could be resolved by running clj -Sverbose -Spath -Sforce | sed -e 's/:/\n/g' (which obviously does more than that, it is useful incantation). Deps remains confusing and questionably documented.

Setting Up a Clojure Project

Interface

Clojure in Scripts

Clojure has a serious problem in shell scripts. A shell script will traditionally start with a shebang line like #!/usr/bin/env clojure. This is problematic. It will try to find a deps.edn in the file where the script is being executed, which is probably not where the script lives.

There may be some way to overrule this behaviour by loading the deps in with the interpreter like #!/use/bin/env -S clojure -Sdeps /home/owen/the/deps.edn. I just got an

Error while parsing option "--config-data /home/owen/the/deps.edn": java.lang.RuntimeException: Invalid token: /home/owen/the/deps.edn

so that didn’t work for me.

It looks like the easy way is a bash script, which starts the Clojure interpreter in the directory with the deps.edn, then pass it the directory that the bash script was run in. Ugly, but it seems this is a tradeoff that lets me keep orchestra so I’m willing to make it.

This is also an opportunity to more completely set up the script’s environment, incorporating lessons from 5.1.

#!/usr/bin/env bash

# REMINDER: We're doing this in a bash script because the clojure interpreter
# needs to be executed in the same directory as `deps.edn`

# Where the script was run from
RUN_DIR=$PWD

# Where the clojure interpreter should start, absolute path
SCRIPT_PATH="`dirname \"$0\"`"
SCRIPT_PATH=`cd $SCRIPT_PATH && pwd`

# The clojure script will need to work with the *command-line-args* to work out
# what directory it should be operating in.
cd $SCRIPT_PATH
clojure check.clj $RUN_DIR

# Execute the interpreter in the script directory, pass it the working
# directory and any other arguments.
# The clojure script will need to work with the *command-line-args* to work out
# what directory it should be operating in. `clojure` seems like it might be
# sensitive to argument order.
cd $SCRIPT_PATH
clojure -i main.clj --report stderr -- $RUN_DIR $@

CLI

{:deps {org.clojure/tools.cli {:mvn/version "1.0.206"}}}

There is a minor practical problem to overcome since the command line parameters won’t be set up when doing interactive development. The command line args aren’t even nil if CIDER is being used and might look something like ("--middleware" "[cider.nrepl/cider-middleware]"). If there is an elegant way it isn’t obvious so solve with a crude approach from the opar namespace.

    (defn command-line-args
      "Returns clojure.core/*command-line-args*. However if there is evidence that
      this code is being executed in a CIDER REPL then instead it will evaluate to
      `default`. CIDER is detected by checking if the command line args load
      CIDER's middleware."
      [defaults]
      (cond
        (not= *command-line-args*
              '("--middleware" "[cider.nrepl/cider-middleware]"))  *command-line-args*
        :else defaults))

There is a second unhelpful situation - sometimes called Buridan’s Ass - where it is difficult to make a deterministic choice between equal alternatives. Interfaces can be a bit like that for me, there are a lot of options and it isn’t obvious how to be systemic about making choices. What does seem clear is that making some choice is better than thinking for too long. Here is my flow chart for choosing an interface:

#!/usr/bin/env clojure

(ns template
  (:require [clojure.tools.cli :refer [parse-opts]])
  (:gen-class))

(def cli-options
  ;; An option with a required argument
  [["-p" "--port PORT" "Port number"
    :default 80
    :parse-fn #(Integer/parseInt %)
    :validate [#(< 0 % 0x10000) "Must be a number between 0 and 65536"]]
   ;; A non-idempotent option (:default is applied first)
   ["-v" nil "Verbosity level"
    :id :verbosity
    :default 0
    :update-fn inc]
   ;; A boolean option defaulting to nil
   ["-h" "--help"]])

(println (parse-opts *command-line-args* cli-options))

cli.matic

It isn’t very obvious how to use it from the README, and by the time you remember how to use it you’ll have lost all motivation to get anything done. Don’t bother.

TUI

Use Lanterna for creating a terminal user interface. Well documented and easy to use.

{:deps {clojure-lanterna/clojure-lanterna {:mvn/version "0.9.7"}}}

GUI

It isn’t easy to outdo Java’s Sling - it works everywhere and is comfortingly basic. Best to use a wrapper-like library to make it a bit terser; try Seesaw. seesaw is an instant hit with a great code style and well documented. Dave Ray, who wrote it, obviously knows how to build a GUI.

{:deps {seesaw/seesaw {:mvn/version "1.5.0"} ; namespace: seesaw.core}}

Programming Aids

Note that these libraries are all added in my user-wide deps.edn that is mentioned under the 2.1 heading.

Dependencies

Despite initial impressions, there is actually useful information about deps.edn in the clojure reference pages. Pretend that the document was written spiralling out from the centre; start with the “deps.edn sources” heading.

Note that the base data structure here is a map and therefore keys must be unique.

There is but little rhyme or reason to the sub-keys of :deps. Copy someone elses work or just arbitrarily name stuff something/or-other. Include {:paths [“.”]} in your deps.edn to include the current folder in the classpath.

Static Analysis

core.typed

This project would, by itself, get Ambrose Bonnaire-Sergeant into a hall of fame. Unfortunately the effort is spoilt by a few weaknesses:

I honestly wanted to use this family of libraries. At the end of the day the ergonomics didn’t seem to be smooth enough but persisting paid off and it became apparent that I was too nervous about nailing down the keys of my maps.

Spec

Use orchestra. The clojure devs don’t seem to have implemented using the :ret key - it is clear why spec is an alpha library!

{:deps {orchestra/orchestra {:mvn/version "2021.01.01-1"}}}

(ns example
  (:require
    [clojure.spec.alpha :as s]
    [orchestra.core :refer [defn-spec]]
    [orchestra.spec.test :as st]))

(ns named-space
  (:require [orchestra.core :refer [defn-spec]]
            [orchestra.spec.test :as st]
            [clojure.spec.alpha :as s]))

;; later

(st/instrument)

defn-spec stresses clj-kondo, the usual linter. Add {:lint-as {net.danielcompton.defn-spec-alpha/defn schema.core/defn}} to the project’s .clj-kondo/config.edn file

Rookie Errors

Consider a collection of elements [{:a 1} {:a 2} {:a 3 :b 1}] with the intent of creating a spec to ensure that this is a collection of maps with an :a key.

(ns example (:require [clojure.spec.alpha :as s]))

;; I could use (s/keys). But I won't since all that matters is that the objects
;; meet a spec
(s/def :example/entry
  #(contains? % :a))

;; Method 1
;; The problem with this implementation is the messages print the entire
;; collection x when there is a problem. It is better to just print the failed
;; entry. Or better yet jsut the part of the failed entry that failed to meet
;; the spec.
(s/def :example/spec
  (s/and
    vector?
    (fn [x] (every? (partial s/valid? :example/entry) x))))

(s/explain :example/spec [{:a 1} {:b 2}])
;; => [{:a 1} {:b 2}] - failed: (fn [x] (every? (partial valid? :example/entry) x))

;; Method 2
;; Right idea? We might expect this approach to be buggy since (s/coll-of)
;; takes a predicate and :example/entry is a keyword. However, there is an
;; undocumented feature where coll-of can be passed spec references.
;; This approach has risks. coll-of will conform the elements of the vector
;; being passed and so may transform the data if more clauses are added to the
;; and.
(s/def :example/spec
  (s/and
    vector?
    (s/coll-of :example/entry)
    ))
    
(s/explain :example/spec [{:a 1} {:b 2}])
;; => {:b 2} - failed: (contains? % :a) in: [1] spec: :example/entry
    
;; Method 3
;; Superficially this looks ok; but it is basically buggy. (s/every) is
;; misnamed and doesn't check that every element of the collection meets the
;; spec - it only takes a random sample (?!). It also has undocumented
;; behaviour that it can take a spec-reference. Best avoided.
(s/def :example/spec
  (s/and
    vector?
    (s/every :example/entry)
    ))
    
(s/explain :example/spec [{:a 1} {:b 2}])
;; => {:b 2} - failed: (contains? % :a) in: [1] spec: :example/entry

This specific example suggests (s/coll-of :example/entry :kind vector?) is the most appropriate spec to use.

Spec-ing Complex Objects

There is an ambiguity that is important to recognise - a predicate like string? and a spec like (coll-of string?) are very different. A spec is in fact some sort of Java class and doesn’t seem to implement IFn (ie, can’t be called). That means a predicate is a spec, but a spec cannot be used as a predicate.

This creates a slight awkwardness between s/and, and and every-pred. Since predicated are also specs, s/and can mix the two. But since some spec conformances transform the data this function has unexpected properties. For example, (s/and string? ::something) and (s/and ::something string?) are completely different specs, and what is valid for one might be invalid for the other. After conforming to ::something the data might be structured differently.

every-pred is a bit more reasonable, applying a set of predicates in parallel. But it is poorly integrated into the spec library, so on failure the specific failing predicate isn’t properly identified. This undermines the basic value proposition I’m looking for in spec which is clear error messages describing how functions are failing. It also requires specs to be transformed back into predicates: #(s/valid? ::something %). Forgetting this produces crazy error messages as the spec library tries to explain that a spec is not a function.

(s/def ::somespec
  (every-pred
   vector? ; this line is OK. (vector?) is a predicate.
   ;; This fails - it treats a spec as a predicate. This spec is not a predicate.
   (s/coll-of #(s/valid? ::otherspec %)) ; this like fails.))
    
;; A sample of a crazy error message when every-pred tries to use a spec as a predicate.
class clojure.spec.alpha$every_impl$reify__2255 cannot be cast to class clojure.lang.IFn (clojure.spec.alpha$every_impl$reify__2255 and clojure.lang.IFn are in unnamed module of loader 'app')

and from Clojure core is not useful here, being inferior to every-pred. It just exists to confuse us by having wildly different semantics to s/and. s/& and the regex operators get little acknowledgement here. It isn’t clear to me how they work and it is late at night as I write this, having been fighting with the other options all evening. Regex probably aren’t a solution, so I’m not touching them.

Spec-ing JSON

I had reason to be working with some JSON objects that were read and manipulated in Clojure. I tried spec but struggled with the assumption that map keys were keywords, not other objects (like strings, which is hugely common ion JSON).

After careful thought, I concluded that it was better to use json-schema to validate these objects. The =luposlip/json-schema` library claims to generate JSON schemas from simple Clojure data structures. It does work, although the README file leaves a little to be desired.

;; Generate a schema
(require 'json-schema.infer)
(require '[json-schema.core :as json-schema])

(def json-object {:the "json" "object" 111})

(spit "schema.json" (json-schema.infer/infer->json {:title "Good Schema"} json-object))

; Fails, 2 should be "2"
(json-schema/validate
  (json-schema.infer/infer->json {:title "yes"} json-object)
  {"the" 2 "object" 3}
  )

{:deps {luposlip/json-schema {:mvn/version "0.3.2"}}
 :mvn/repos {;; relies on everit-org/json-schema which is distributed on jitpack
             "JitPack" {:url "https://jitpack.io"}}}

Handy libraries

Builtins

External

{:deps {org.clojure/core.match {:mvn/version "1.0.0"}
        cheshire/cheshire {:mvn/version "5.10.0"} ; namespace: cheshire.core
        }}

General setup

Editor

(use-package cider)
(use-package clojure-mode
  :mode "\\.clj\\'")
(use-package company)
(use-package rainbow-delimiters
  :hook ((clojure-mode    . rainbow-delimiters-mode)))
(use-package blamer
  :hook ((clojure-mode    . blamer-mode)))

;; We are going to install lsp-mode later. Run
;; M-x lsp-install-server RET clojure-lsp
(use-package lsp-mode
  :hook ((clojure-mode) . lsp-deferred)
  :commands lsp)

Notes On Debugging

Log Clojure errors to the terminal

So for reasons various I want to use Clojure to write a shell script. The script is fairly simple, say:

#!/usr/bin/env clojure

(throw (new java.lang.Exception "!"))

$ script.clj
WARNING: When invoking clojure.main, use -M
Syntax error compiling at (./script.clj:1:1).
!

Full report at:
/tmp/clojure-16000908815811689537.edn

That report is troubling! I want my errors logged straight to the console. Having to open up a tmp file be too slow; I make a lot of mistakes. A little reading suggests that I want to set -Dclojure.main.report=stderr.

$ script.clj -Dclojure.main.report=stderr
WARNING: When invoking clojure.main, use -M
Syntax error compiling at (./script.clj:1:1).
!

Full report at:
/tmp/clojure-16000908815811689537.edn

That didn’t work. But I’ve always wanted to know what these stack traces mean, so I open up /tmp/clojure-16000908815811689537.edn and use it as a guide to main.clj. The journey starts at [clojure.main main "main.java" 40] which is

(defn root-cause
  "Returns the initial cause of an exception or error by peeling off all of
  its wrappers"
  {:added "1.3"}
  [^Throwable t]
  (loop [cause t] ; <= Line 40
    (if (and (instance? clojure.lang.Compiler$CompilerException cause)
             (not= (.source ^clojure.lang.Compiler$CompilerException cause) "NO_SOURCE_FILE"))
      cause
      (if-let [cause (.getCause cause)]
        (recur cause)
        cause))))

This must be the bottom of the stack for doing error triage. Scanning up the stack, the next interesting frame is [clojure.main$main doInvoke "main.clj" 616] - not very useful, that is (defn main ...) - and one level higher is [clojure.main$main invokeStatic "main.clj" 664]. Aha!

; ... in (defn main ...

         :main-opt
         (try
           ((main-dispatch opt) args inits) ; <= Line 664
           (catch Throwable t
             (report-error t :target (get flags "report" (System/getProperty "clojure.main.report" "file")))
             (System/exit 1)))))

; ...

Clojure must be accessing the property at the last minute with (System/getProperty). I bet there is a System/setProperty.

#!/usr/bin/env clojure

(System/setProperty "clojure.main.report" "report")

(throw (new java.lang.Exception "!"))

$ ./script.clj
WARNING: When invoking clojure.main, use -M
{:clojure.main/message
 "Syntax error compiling at (./script.clj:5:1).\n!\n",
 :clojure.main/triage
 {:clojure.error/phase :compile-syntax-check,
  :clojure.error/line 5,
  :clojure.error/column 1,
  :clojure.error/source "script.clj",
  :clojure.error/path "./script.clj",
  :clojure.error/class java.lang.Exception,
  :clojure.error/cause "!"},
 :clojure.main/trace
 {:via
# continues ...

Debugging Spec

Clojure specs are actually a debugging nightmare for someone developing in CIDER. It is very difficult to figure out why a spec is misbehaving. For example

(s/def ::example
  (s/and
   map?
   #(contains? % :a)
   #(int? (% :a))
   (s/coll-of int?)
   ))

(s/explain ::example {:a 1})

The problem here is clear and the spec is impossible to satisfy. The spec has to be a map that contains something like {:a 1} and when that is viewed as a collection to check the s/coll-of that entry of the map will be [:a 1], which is not an int. The programmer has made a mistake. They probably meant to check an element of the map.

The problem here is that there is nothing to instrument to step through the spec and find out why it is going through the map entries and expecting them to be integers. The situation is confusing for someone who doesn’t already understand the problem. Is it the last or second-to-last line of the spec that is the problem? It becomes harder to figure this out the larger the spec is. Bugs here could be hard to find.

We can’t instrument the function because specs are checked before invocation or after evaluation. We can’t instrument the spec because it is data and never executed. We can’t really instrument the checking functions because we don’t know where they are and it isn’t obvious CIDER can instrument clojure.core.

You can look at the code for conforming coll-of. It doesn’t look fun to step through that. This might have to be shelved as an unresolved problem. No complex specs for me.

Sequences are to be made up of specable components. If the sequence is specced ::seq then the elements are speced ::seq+.

Summary