Back to the schedule
Previous: Emacs Application Framework: A 2021 Update
Next: CLEDE: the Common Lisp Emacs Development Environment

Imaginary Programming

Shane Mulligan

Q&A: live Q&A or IRC
Duration: 10:17

If you have questions and the speaker has not indicated public contact information on this page, please feel free to e-mail us at emacsconf-submit@gnu.org and we'll forward your question to the speaker.

Talk

Q&A

Description

Imaginary Programming (IP) is both methodology and paradigm. It is an extension of literate programming and a way of creating software without the use of imperative, functional or even declarative code. Yet IP employs all disciplines to achieve the miraculous. The only contingency is on one or more language models, known as foundation models. The real value of IP is not found by abandoning sound logic altogether, but in weaving the real with the imaginary. The future of imaginary programming is one in which almost all of computing is inferred. I have built a suite of tools based on emacs for interfacing real programming languages with imaginary ones; all of this in order to demonstrate what I mean; a ‘complex’ terminal that lets you imagine what happens no matter how nested you are within interpreters, an example-oriented language, a file format that encodes the provenance of text and a library for imaginary functional programming primitives called iLambda. It is important to recognise IP because, for lack of a better term, it has far-reaching implications for intellectual property and the GPL. Please keep an open mind.

Discussion

IRC nick: libertyprime

Pad:

• Q1: Do you have a site we can follow more of your writing on?
  • A:Pen.el Tutorial: https://semiosis.github.io/posts/pen-el-tutorial/
  • https://semiosis.github.io/posts/ilambda-tutorial/
  • https://emacsconf.org/2021/talks/imaginary/
• Q2: re slide 27, would it mean that 2 such "idefined" functions would be the "same", meaning do the same thing the same way, given that they are defined without a "body"? (i'm trying to get a better grasp on the objects that get so "imagined" under the hood)
  • A: The first time a function is run with given parameters, the results are remembered. I use the memoize library. You can update the function every time by surrounding the call the the function with the (upd ...) macro. The body evaluation is completely short-circuited with idefun. The imacro works a bit differently. It will generate real code. You can use the normal macro-expand on an imacro.
• Q3:Opalvaults :What are some underlying concepts/papers, that we could read to become more familiar with your overarching ideas? (i.e. for instance things that inspired your ideas)
  • A: paper: pretrain, prompt and predict
• Q4: Sorry, I just don't get it: How is a function that does something different each time it's called useful?
  • A: Each time you run one of these functions, you are getting the computer to imagine for you. It's a bicycle for the imagination. You can automate the filtration of the results you want, say by doing many generations and applying grep, or other prompts such as the semantic search prompt to the results. The functions are memoised, so they technicaly do the same thing every time if you want them to. Also, if you use a temperature of 0 for the prompt functions (I demonstrate how to override that, somewhere in the slides), it will be deterministic too, even when bypassing the cache.
• Q5: How on earth do you ensure that what ilambda gets back from GPT-3 is Lisp and not, say, Harry Potter fanfic?
  • A: A combination of good prompt design, filtering the results, and validating the results. Also, you can fine-tune models to the task you want to eliminate the possibility of unwanted generations.
• Q6: Your views on the pluses and minuses of GPT-3?
  • A:It's something we have to live with because of its transformative nature on computing. These language models unfortunately are license-blind.
• Q7: Any interesting ideas about potential applications of GPT-3 to Emacs itself (or Emacs-adjacent things)?
  • A: Emacs is the ultimate text-centric operating system. It will become a kernel for AGI, I think. That's what I plan on making. The power-user's terminal of human-ai interaction. I'm trying to extend as many modes in emacs as possible. Org-brain, eww browser, org-mode, comint, emacs lisp primitives, etc.
• Q8: Follow-up on Q2: how does infering functions in this manner differ from, say, how in the Haskell ecosystem functions are infered by specifying inputs and return type (such as when searching for a suitable function for a given purpose)?
  • A: Where in haskell, type-declarative function search look through a discrete set of functions by type, the domain of possible functions that are search for using language models is qualatively and quantatively infinite.
• Q9: Are you deriving functions from their names? What do you do when this is ambiguous - for example, when the name of the function is "get-element-from-pair"?
  • A: idefun will infer computation and short-circuit the code. Given either 'function name', alone, function name + args, or function name, + args + docstring, or function nae + args + docstring + function body, it will make use of the context you have provided and imagine evaluation. It will create functions which infer rather than properly evaluate, based on merely the name of the function, for example.
  • A (re: ambiguity): If you had an imaginary defun for this, you'd need to send the final list

BBB:

• libertyprime: What kinds of software is IP (imaginary programming) not suitable for?
  • libertyprime: Good question. IP is great for things like mocking API calls, because you can imagine the API call output. It's great for code generation where you can then do a macro-expand to generate code as you are programming. It's great for coming up with functions that might be difficult to write (idefun color-of-watermelon), for example
• Hey libertyprime, where do we follow up to find out more?
  • libertyprime: it's not really good for scaffolding code. I consider emacs to be 45 years of scaffolding to build imaginary functions around
  • libertyprime: Because IP needs a rigid complimentary code.
• So how does an IP user verify that the imagined code does what is intended?
• I like the word 'imaginary' to describe the paradigm
• libertyprime: How does an IP user verify that the imagined code does what is intended? Through a combination of 'validator functions', imaginary validation functions and language model fine-tuning. So you may also choose an underlying language model to use when running code. That model may have been trained to do the task you are giving it. If you're trying out the docker container you can run pen config or do M-x pen-customize to force the language model, or chage it in the imagine-evaluating-emacs-lisp .prompt file
• libertyprime: Haha. The brilliance of emacs, and the reason this stuff is so easy to do with emacs, is that emacs provides intelligible modes and abstractions with which to build prompts. Otherwise you have an amorphous blob of a language model.
• libertyprime: So the value is absoltely not in replacing emacs entire, as I've come to understand it, but in combining real and imaginary.
• (wish i could give you back just a fraction of the time you saved just this one person here!)
• I would love to see the result of imaginary major modes and keymaps
• libertyprime, is the idea for the first draft of the gpt output to be final, or do you expect to edit some?
• There seems to be a lot of jargon in this context, like validators, prompts, language models, etc. It's really hard for someone who doesn't already use these things to understand what these pieces are and how they fit together.
  • well prompts seem to be the input you give to the language model, which it then generates a follow up to
  • validators sounds like tests? language models are neural language models like GPT-3/j etc.
  • libertyprime: http://github.com/semiosis/glossaries-gh/blob/master/pe-prompt-engineering.txt http://github.com/semiosis/glossaries-gh/blob/master/prompt-engineering.txt http://github.com/semiosis/glossaries-gh/blob/master/pen.el.txt
  • libertyprime: Here are some glossaries for the subjects
  • So like, a prmpt would be "Marco!" and GPT-3 would of course say... "Polo"
  • libertyprime: @alphapapa, I also have a much matured prompt format readme, here: https://github.com/semiosis/prompts
  • libertyprime: which can explain 'validator'', etc.
• aindilis: So uh... does GPT-3 know... everything? in every human and computer language? I don't understand its role exactly, or its limitations.
  • GPT-3 knows a lot, but not all, from my experience. It's pretty scary, in a good way. I think libertyprime wants to keep it libre.
  • libertyprime: the latest language models such as Codex are world language + codex, and they know everything at an abstract level, like a human does, in a way. So their depth may be superficial. They're pretty good knowledge aggregators.
• so libertyprime can you just tab complete and it completes on like the previous sentence, region, buffer, etc?
  • libertyprime: Yes, it has basic autocompletion functions, (word, line, lines). I'm also making more interesting autocompletion functions, which do beam-search on downstream generations, -- calling it desirable-search. http://github.com/semiosis/pen.el/blob/master/src/pen-example-config.el
  • libertyprime: There are some key binding definitions here which will work for the docker container
• Does GPT-3 "know" how to transliterate from say public code written in JS / Other-Lang to elisp if you were trying to imaginary code similar function names?
  • libertyprime: yes, it absolutely can. transpilation is one thing it is very good at. But more bizarrely, you can also transpile intermediary languages, that are composed of multiple different language chimerically. For example, you can smash out your algorithm with a combination of elisp and bash and it will understand when it transpiles into a real language.
• How well does it actually work to write a function in a mishmash of Bash and Elisp? I can't imagine that working well in practice. There are too many semantic differences in the languages and implementations
  • libertyprime: it's a very new sort of thing, but feels natural as you are doing it, to generate code. the results of generating code should most probably be looked at before running. that beign said, you can also run 'ieval' around it to run it in inference. I think the takeawaay should be that these models are getting better and better and show no signs yet of reducing quality of results or ability -- no sign yet
• how does lexical binding affect things, if at all?
• How about going from a CLOS/EIEIO style of OO to Java / C++ style? Or Erlang style of parameter pattern matching?
• so IIUC GPT-3 is a service run on a remote system, right? And it's proprietary? How big is it in terms of bytes?
  • libertyprime: yes, aggregated language models are not good in my opinion. GPT-3 is around 170 GB, approximately 1GB per million parameters, IIUC
  • libertyprime: There are libre models, and you can connect one to penel to run the inference etc. My goal is to decentralise them though
  • libertyprime: Because I don't think that 170GB is accessible enough. The issue is actually running the models though. You need a very large computer indeed for that
  • libertyprime: I can do a customized demo if anyone wants
• can someone here provide some sample input, and you run it and paste the result, just to give an idea of the quality? or do you already have samples online?
• here's an idea for a demo... something like (idefun (string target-language) "Translate STRING from its source language into TARGET-LANGUAGE and output it to the echo area.")
  • oops I forgot to name the function, was thinking of ilambda
  • I have a feeling that such a large scope for the function will exceed the max output size of the model. maybe we work on a more realistic example?
  • I was hoping the model would solve all the messy problems for me
  • libertyprime: Oh crud. I hope I havent broken Ilambda. Lol I added support for 0 arguments, it makes it variadic. This will work
  • doesn't seem like it quite understood the purpose but I can see the connection
• what happens if you change target lang to "Elisp" >:)
  • look at the echo area if you didn't notice it
  • oh wait, I missed the echo area
  • libertyprime: Yup, exactly, that will work too. One sec
• can you run the function again or show "C-h e"? And can we see the resulting source code?
  • libertyprime: translate python to elisp
  • libertyprime: just with (idefun translate)
  • libertyprime: No docstring, etc. or arguments.
  • libertyprime: ccrud. It didnt work haha
  • libertyprime: Sigh.
• libertyprime: I need to fix the 2-ary argument thing. :S Really sorry I think I broke it
• I'd like to see the generated (or "imagined") Elisp source code, assuming it does some HTTP API queries to do the translation and such
• libertyprime: Yup, I can show that. It works much better when I use OpenAI Codex. Here are some generated functions
• libertyprime: That's how it works under the hood. Then it cuts out the bit that you want
• This reminds me of the classical AI paradigm of "generate and check."
• libertyprime: Sigh. I really cry when demos break. Sorry. I demo'd the underlying prompt though. I broke ilambda, i think
• I think I saw it generate a huge fibonacci function, is that still in your kill-ring?
• okay, well thanks for demoing, the code is pretty stable though at this point right? this is just the norm with any demo.
• I bet people would be glad to watch/read something later on if you want time to work on it.
• libertyprime: https://semiosis.github.io/cterm/ This is what I call the complex terminal. Essentially you prefix any terminal program with ct and you get autocompletion etc. for anything. it uses emac's term-mode
• libertyprime: https://semiosis.github.io/ii/ And this, ii, it's fully imaginary terminals, so you can import imaginary libraries, etc. and work with them.
• libertyprime: https://semiosis.github.io/apostrophe/ This one here, which imagines conversations with anyone that the model knows about. So I'm demoing having a 3way conversation with amber heard, tom cruise and chris nolan.
  • so you used GPT to generate a compliment, and now GPT generates the convo from that prompt?
  • libertyprime: Yeah, so the best way to interact with these types of chatbots is to imagine the situation you are in before hand. the initial phrases can be anything you can think of really. Why are you in the bath tub?, for example. But I tend to open with something like, may I interrupt? What were you just working on? so by choosing the prompt very carefully, you can tease out the information you require.
• libertyprime: https://semiosis.github.io/nlsh/ and this, which is a natural language shell
• libertyprime: I also have a way to filter results semantically, with my semantic search prompt http://github.com/semiosis/prompts/blob/master/prompts/textual-semantic-search-filter-2.prompt
• libertyprime: YOu can run all these prompts also from bash like so: pl "["It's cool. I used to dance zouk.","I don't know.","I'm not sure.","I can't stop dancing to it.","I think it's ok.","It's cool but I prefer rock and roll.","I don't know. It sounds good.","Nice but a bit too fast,"Oh, I know zouk, you can teach it to me.","Zouk is nice."]" | "penf" "-u" "pf-textual-semantic-search-filter/2" "positive response". That will pipe json results into Pen.el, and have it filtered. all prompting functions are also available as shell commands.
• well I think this is the coolest thing I've seen in a long time. how do we follow up with you and get involved? run it etc?
• libertyprime: hehe thanks aindilis: i'm on #emacs as libertyprime. Feel free to hit me up any time. Otherwise, the setup for pen.el is fairly straight forward. If you have any issues demoing, I'd be very interested, so I can make Pen.el more reliable. I have a discord server. I'll copy the link. One sec
• Do you think you could run an IRC channel too?
  • libertyprime: https://discord.gg/sKR8h9NT
• Thanks a lot, very interesting and I am excited to learn more later!
• yeah this talk was crazy good, ty!

IRC:

• What Shane is saying right now reminds me a lot of the SICP opening words, about how programming, and computing ideas in general are all about dreams and magic. Creating an idealized solution from abstractions and building blocks.
• This also reminds me of the concept of Humane Tech. Technology, and frameworks that are inherently conducive to human curiosity, intelligent, and all the best traits. https://github.com/humanetech-community/awesome-humane-tech
• I think this is like semantic auto-complete on steroids, like tab completion of whatever your typing, or translation of something you've written into code for instance.
• If you're worried about these kind of advances in AI, just remind yourself of how easily technology breaks
• oh my god, executing code derived directly from GPT-3?! that's lunatic curl | bash, eat your heart out
• idefun definitely helped by a docstring
  • yeah that's a use-case, gen from docstring
• Man, I really think it would be awesome to have shane be able to explain some of these ideas more in depth as they are obviously very deep topics. I'd love to help contribute next year to possibly creating a way to have multiple talks going on at once so people have more time to speak. I believe it was sachac who mentioned it yesterday.
• This vaguely reminds me of that one Python package that generates a CLI parser from the help string except that that python package actually made sense
• re slide 27, would it mean that 2 such "idefined" function would be the "same", meaning do the same thing the same way, given that they are defined without a "body"?
• the full abstraction would look something like an interactive proof program where you could repeatedly refine the results until it matched what the user wanted
• it started incomprehensible and then moved straight to impossible magic.
• wow...mind blown even though that went by a bit too quick.
• Hmm, I do think we could do test-driven imaginary programming tho i.e. you only define the ERT testcases and then do the rest with idefun
• So (imacro with-clifford-algebra (p q r)) would just "work"... this does feel too magical
• I am really happy that someone is trying Deep Learning stuff with emacs and not just for writing Python code
• well I've had pretty good success with GPT-3, I think this also supports GPT-j which is I think free/libre.
• most users of GPT-3 do it via calls to a web api
• is it still invite only?
  • no, it's been opened recently

From YouTube:

• Is this Emacs with smooth scrolling? How is that possible? I tired that really hard. Or is it just a PDF reader?
• Lets go!!!!! Imaginary Programming all the way.

Outline

• 5-10 minutes:
• a 5 minute introduction to imaginary programming, followed by
  • a demonstration of iLambda.
    • iλ, a family of imaginary programming libraries https://mullikine.github.io/posts/designing-an-imaginary-programming-ip-library-for-emacs/

IRC libertyprime at #emacs on libera

Shane Mulligan

Links

• Pen.el Tutorial: https://semiosis.github.io/posts/pen-el-tutorial/

Transcript

Hi everyone! My name is Shane, and it's truly a real honor to be able to speak at EmacsConf. I love Emacs. I love Emacs Lisp, and this is also another topic that really excites me ever since I got my hands on OpenAI's GPT-3. Six months ago, I knew that it was important to research this for Emacs, you know, asides from being somebody who never wants to leave Emacs myself, and I hope that…. Being one of the few Emacs users, possibly the only one that had access to GPT-3 for this long, I hope that I can contribute. So, Emacs, sorry. Imaginary programming is…, it's based on prompt engineering currently, but it's an abstraction over prompt engineering. You can read about prompt engineering and language models, that's quite easy to google for. And imaginary programming is a subfield of imaginary computing, which is just the larger domain of employing of computing that's based on imagination, artificial imagination, so, basically, dreaming up user interfaces instead of…, or at least partially dreaming them up. I suppose that it's a fringe paradigm, but it's extremely useful. Imaginary computing, if you're willing to call it that, would be what's used to imagine simulations for self-driving cars, as they're trained for example. But this technology finds its way to the public, and it should be in the public domain. So, Demis Hassabis he's the founder of DeepMind, and he did his Ph.D. in human imagination, now he's working on artificial imagination. So, just a couple of days ago I took AlephAlpha's world model API and I plugged it into Emacs's eww browser, and now I have a way of generating text for the images instead of actually…, so I can stay in my text only Emacs, which is the way it should be. Intelligent NFTs, I'll leave this for you guys to look at. Information bubbles. So, there's some potential bad outcomes from the runaway empowering of these large language models and other models in commercial hands. It's causing information bubbles and ways of controlling people. So, for example, micro tasks and stuff that…, furthermore just automating away, or abstracting away the role of a programmer, and the automating more and more increasingly abstract tasks, and I think the solution is to decentralize and break up these tasks. I have a potential way of doing that, but firstly I'll talk about the imaginary web briefly because the thing about these language models is, they can replace basically everything on the internet. So, like, replace your Wikipedia, or Stack Overflow, replace conversation if you want it with, from real people to chatbots instead, replace basically anything there's a website for. And that means that rich media has gone from becoming images and video and even from paywalls now into intelligent and truthful, because generating fictional websites is going to become a very easy thing to do, and actually the best way to do it, and the most useful way to do it, so then you need a source of truth. The imaginary web is a thing, imaginary interpreters are a thing, so you imagine your interpreter, or you overlay prompting on top of a real interpreter to see what you might want to do in an interpreter, and what you might want to say to somebody if you're talking to them. If you're inside, say, ERC in Emacs, an IRC client, and you prompt after somebody says something, then the prompt will probably suggest, what you might say in return, and then you can prompt like a multiverse, and you can pick from them. There's a bunch of crazy utilities for these language models. Paracosm vs Metaverse. So, Mark Zuck wants you to live in his virtual reality as defined by him, and he's going to do it by using these models to make you a fictional world that you can live in. You can do that, or you can use them yourself, and you build your own Paracosm. And I think that's an important ability to be able to have, otherwise we will be like the borg, and we'll be connected to Mark Zuckerberg. Truth is a hot topic. So, the way that I think we should do this to decentralize the language models is, to use Structuralism. Universal grammar, template metaprogramming, and GPT-3 what do they have in common? Well, you have some kind of basis, like, you train your GPT-3, and then you do all your prompting on top of it. Like a person is born with this grammar, and then they quickly learn language, and like with C++ templates, you pre-process and then the run time runs on that. So, anyway, that was a slide. Structuralism. I think you can decompile…, think you can break, so you can decompose the language models into units, but those units won't look like neurons, they would look like these, and you put them onto a blockchain. But you can look at that later if you want anyway. I'm going to skip straight to ilambda, running out of time. I'll just quickly show you the ilambda primitive in ilambda. So, it evaluates instead of run, for example, here's the reduced function, and you've defined your imaginary lambda here. And it doesn't have a body, it's just got the comment and the parameters, and that's enough for…. Once you have that ilambda that runs now as a function, you can stick it into a reduced function, for example, and it'll reduce this list. You could even remove…, yeah, so you kind of need the comment otherwise it's too hard to imagine what would happen next, but for a function you can literally have an idefun even without the argument list. It was just like generate-fib-sequence, and most likely when you run that defun, it would work the way you want it. The more information you give the idefun, the imaginary defun, the better it would capture the task which you're trying to do, in this case you want to generate a Fibonacci sequence. And yeah, you can define functions without having a body, and they run an inference instead. Here's a way of overriding the language model that's used, for example, using dynamic scope. So, under the hood, idefun just uses an ilambda. This function here just doubles things. So, here's a function that gets you a hexadecimal color just from the name. And you can create arbitrary functions like this, so, what we need is like a library of imaginary functions, I think, that match a language model. Macros on the other hand, as different from functions, they actually macro expand and generate code. So, when you macro expand this, you'll get this, and that's because this has an arity of 3, and then when you macro expand that underlying macro, it generates the actual source code. You can actually run these macros, and it will cache the output…, it will catch the source code, so the macro runs the same every single time, or generates the same code, but you can just use it to generate code really easily while you're programming. I hope that this has been informative, it wasn't too much time, but there's plenty of material for you to dig into it more if you're interested. Thank you very much for letting me talk today. Peace out! captions by bhavin192 (Bhavin Gandhi)

Back to the schedule
Previous: Emacs Application Framework: A 2021 Update
Next: CLEDE: the Common Lisp Emacs Development Environment