Full AI Suite for LispE: llama.cpp, tiktoken, MLX and PyTorch

I have presented LispE a few times in this forum. LispE is an Open Source version of Lisp, which offers a wide range of features, which are seldom found in other Lisps.
I have always wanted to push LispE beyond a simple niche language, so I have implemented 4 new libraries:

1. lispe_tiktoken (Openai tokenizer)
2. lispe_gguf (encapsulation of llama.cpp)
3. lispe_mlx (Mac OS's own ML library encapsulation)
4. lispe_torch (An encapsulation of torch::tensor and SentencePiece, based on PyTorch internal C++ library)

I provide the full binaries of these libraries only for Mac OS (see Mac Binaries).

What is really interesting is that the performance is usually better and faster than Python. For instance, I provide a program to fine-tune a model with a LoRA adapter, and the performance on my Mac is 35% faster than the comparable Python program.

It is possible to load a HuggingFace model, to load its tokenizer and to execute inferences directly in LispE. You can also load GGUF models (the llama.cpp format) and run inference directly within LispE. You can download models from Ollama or LM-Studio, which are fully compatible with lispe_gguf.

The MLX library is a full fledged implementation of the MLX set of instructions on Mac OS. I have provided some programs to do inference with specific MLX compiled models. The performance is on par and often better than Python. I usually download the model from LM-Studio, with the MLX flag on.

The whole libraries should compile on Linux, but if you have any problems, feel free to open an issue.

Note: MLX is only available for Mac OS.

Here is an example of how to load and execute a GGUF model:

    ; Test with standard Q8_0 model
    (use 'lispe_gguf)
    
    (println "=== GGUF Test with Qwen2-Math Q8_0 ===\n")
    
    (setq model-path "/Users/user/.lmstudio/models/lmstudio-community/Qwen2-Math-1.5B-Instruct-GGUF/Qwen2-Math-1.5B-Instruct-Q8_0.gguf")
    
    (println "File:" model-path)
    (println "")
    (println "Test 1: Loading model...")
    
    ; Configuration: uses GPU by default (n_gpu_layers=99)
    ; For CPU only, use: {"n_gpu_layers":0}
    (setq model
       (gguf_load model-path
          {"n_ctx":4096
             "cache_type_k":"q8_0"
             "cache_type_v":"q8_0"
          }
       )
    )
    
    ; 2. Generate text only if model is loaded
    (ncheck (not (nullp model))
       (println "ERROR: Model could not be loaded")
       (println "Generating text...")
       (setq prompt "Hello, can you explain what functional programming is?")
       ; Direct generation with text prompt
       (println "\nPrompt:" prompt)
       (println "\nResponse:")
       (setq result (gguf_generate model prompt {"max_tokens":2000 "temperature":0.8 "repeat_penalty":1.2 "repeat_last_n":128}))
       (println)
       (println "-----------------------------------")
       (println (gguf_detokenize model result)))

One of the first important things to understand is that when you are using Python, most of the underlying libraries are implemented in C++. This is the case for MLX, PyTorch and llama.cpp. Python requires a heavy API to communicate with these libraries, with constant translations between the different data structures. Furthermore, these APIs are usually pretty complex to modify and to transform, which explains why there is a year-long backlog of work at the PyTorch Foundation.

In the case of LispE, the API is incredibly simple and thin, which means that it is possible to tackle a problem either as LispE code or when speed is required at the level of the C++. In other words, LispE provides something unique: a way to implement and handle AI both through the interpreter or through the library.

This is how you define a LispE function and you associate this function with its C++ implementation:

        lisp->extension("deflib gguf_load(filepath (config))",
                        new Lispe_gguf(gguf_action_load_model));

On the one hand, you define the signature of the library function, which you associate with an instance of a C++ object. Once you've understood the trick, it takes about 1/2 hours to implement your own LispE functions. Compared to Python, there is no need to handle the life cycle of the arguments, this is done for you.

        Element* config_elem = lisp->get_variable("config");
        string filepath = lisp->get_variable("filepath")->toString(lisp);

The name of your arguments is the way to get their values on top of the execution stack. In other words, LispE handles the whole life cycle itself, no need for PyDECREF or other horrible macros.

One of the most striking features of LispE is that it is very close to the metal in the sense that a LispE program is compiled as a tree of C++ instances. Contrary to Python, where the code in the libraries executes outside of the VM, LispE doesn't make any difference between an object created in the interpreter or into a library, they both derive from the Element class and are handled in the same way. You don't need to leave the interpreter to execute code, because the interpreter instances are indistinguishable from the library instances. The result is that LispE is often much faster than Python, while proposing one of the simplest APIs to create libraries around.

The lispe_torch library is still a work in progress, for instance MoE is not implemented yet in the forward. In the case of tiktoken, gguf and MLX, the libraries are pretty extensive and should provide the necessary bricks to implement better models.