Marginalia: 2025-08-04

Reading List

Rough Thoughts in Progress

Training on the speeches of the dying

LLMs are probabilistic machines. There is no intelligence behind the weights, only prediction. Prediction can however look like intelligence.

For humanity, we want these predictions to represent our Coherent Extrapolated Volition. That is, predictions that represent what humans, in general, would want, in every circumstance. I don't believe pre-training on the world's text, then posttraining on the whims of a narrow set of human preference reveals the fundamental desires of man.

An alternative training corpus would be the thoughts, wishes and reflections of the dying. While a morbid idea, perhaps our final thoughts represent our best estimation of what is truly important to our species?

There are abundant repositories of such texts. It might be interesting to finetune a model on these.

LLM daydreaming

It's quite troubling that we haven't seen a vast number of scientific breakthroughs, enabled by LLMs, by simple virtue of their training on internet scale data. One would have suspected, a priori, that a machine that saw the entire internet many times over, and which is now connected to the internet for real-time information lookup, would be able to spot connections between pieces of knowledge. It's perhaps doubly worrisome that researchers, enabled with this technology, don't appear to be discovering revolutionary new science at an increased rate.

I suspect the Default Mode Network plays an important role in the story. The human brain is never truly off, even when we sleep or rest. There is always activity. This is very much related to the Overfitted Brain hypothesis from Eric Hoel.

I wonder if it's possible to let the models "sleep", and then bake in their "observations" to the weights. It does strike me that with LLMs we've perhaps constructed a language centre to communicate with them, without the other aspects that might be necessary to construct cogent thought. In particular; a way to experience and act on their environment. I understand that a hypothesis popular with AI researchers is that, through language learning, a "world model" is formed. One suspects this is akin to shadows on the cave wall.

Gwern suggests an interesting "daydreaming" approach which is roughly prompting the model to think about 2 concepts retrieved from a memory of concepts the model has seen before (i.e. from a vector DB), roll out the thought, and then have a discriminator decide if the thought is novel, interesting etc. If it is, that thought/concept is folded back into the vector DB. I'd note that, to my knowledge, no detail is given on how the conceptual extraction is to be carried out.

I wonder if a slightly more interesting approach would be to take the input tensor and slightly adapt it to have the model rollout a new thought on that basis. While I submit that the new input tensor would not itself be valid text in embedding space, I do wonder if it's close enough to lead to intelligible input. This side steps the notion of external retrieval of concepts and the concept of saving the extracted resulting concept back into a vector DB for later retrieval.

A perhaps related approach would be to take the same input text the model has seen "all day" and, rather than altering that input, instead nudge the weights of the model slightly - I'm not sure this has a precise analogue to dreaming - I might imagine that a better analogy would be connecting the NN in a different (more fully connected) way. Observing the output (which may well be gibberish) and retraining on that.

Let's say we wanted to carry out these experiments - what are we actually measuring? Do we simply let the model dream, finetune and then hope for a "better" model at the end. It's not clear what the benchmark here would be and It's unclear if the model would be smarter or not.

High-Bandwidth Fidgeting

Some thoughts on the input device of the future. Is it possible to "pairwise" train the device and the user in a mindmelding sort of way?

This is a stupid idea but I'd be curious to see if it's even possible. I was previously thinking about lip reading as an input modality for computers. 1) Because speaking in an office or coffee shop is a non-starter and 2) Because my accent renders most speech-to-text useless. Potentially this works but I suspect it is inadequate. So the hunt is on for a better input modality.

One such observation is that our fingers are clearly high bandwidth enough to carry some signal from our brain to the page. There is a fixed mapping between a thought -> our motor control -> the output. Frequently our motor control is imperfect and we need to backtrack but that is ok. We have a well known input (backspace) to carry out that operation.

I'm wondering, using ML, if it's possible to "pairwise" train yourself and an algorithm to recognise a very high cardinality motor input to language.

Potentially there is some research in this space for the disabled? My understanding is that they use consistent movement (of the pupil or head often) to select a letter or word.

When I say high cardinality - imagine a pad which registers input location and pressure for each of your fingers in a time series dataset. This is a continuous signal but we can bucket into an arbitrarily high cardinality plot of cartesian coordinates and input pressure values.

If a user is just tapping randomly there is no meaningful signal there. BUT perhaps there is something in the idea of an Aarson Oracle (people act with more predicability than they think / conscious thought impacts "random" action) coupled with recent reward model advances in LLMs (PPO/GRPO).

So we're both training the human to output streams of input that get the results they want AND a model, which is personalised to the user, to predict what that stream of input should correspond to, in natural language.

Are LLMs good for developers or not?

How can we perceive something which should be objective, like the impact of AI on coding, so differently?

Motivation: it's important to be objective about whether we're actually getting somewhere with this paradigm or not. I also think programming is a good bellweather for what we might expect the rate of improvement, and the teething problems to be in other aspects of knowledge work.

Programming has emerged at the beach-head use of LLMs for knowledge work automation because:

  1. It's a domain with abundant training data
  2. Training models on code is a useful guidance to other reasoning tasks
  3. It's amenable to Reinforcement Learning from Verifiable Reward
  4. It's a useful downstream application to recursively speed up everything else (including LLM development)
  5. It's something the developers of these machines rate as "intelligence". Perhaps if the route to AGI was through emotion, rather than vector math, we'd be more interested in the beauty of the poetry outputted.

It seems there are wildly different estimations on whether LLM coding actually represents a speed up for software developers:

How is this possible:

  1. Talking your book - look very closely at who is pushing the narratives that AI can do X. Typically they're building something that promises exactly that. You should discount their view entirely as it cannot be trusted
    1. In the limit, anything anyone employed by the model labs says should be treated as misinformation.
    2. Even benchmarks should be treated with extreme suspicion. Most are poor measure, many are leaked and all are hillclimbed (which leads to information leakage by proxy)
  2. Domain and context dependence - look at what the individual is actually doing with AI. Which language, framework, library, are they using. What is the nature of their task? The quality of their AI's code is directly proportional to the number of Github results you'll find by searching for those things.
    1. This goes much deeper - are there established patterns in your codebase or not?
  3. Their threshold for quality - people vary in their personal quality bar. Software has always had artisans and {whatever the opposite is}. This is not just a question of consciousnesses. Certain tasks mandate a different level of care. The bar for a demo app is different from an enterprise deployment.
    1. A good example would be the Jon Gjengset's YT vid - this is someone that is supremely skilled but is also guiding the machine in a very precise way
  4. Their prior skill - what is a time save for me may not be for you. Likewise, what I perceive to be high quality may be for you low quality. To a large extent the result you get is a reflection of your prior skill.
    1. There is an interplay between this and the other aspects of this list (5) in particular.
    2. I'd like to try to come up with a model here - on the 1 hand the level of skill is likely to lead to better output (through better guidance) BUT that person is more likely to be doing a more complex task AND is more likely to have a higher bar for quality. There is almost an equation here.
  5. Imagined vs. Reality - we all perceive reality differently. You may believe the AI did most of the work when in reality you've prompted it at every phase. It's not clear then whether most of the credit should go to the model or to you. I think we're pretty bad at assigning credit in this way.
  6. (related perhaps to above) - Your definition of AI/vibe coding. It's important to be clear on definitions. Is your threshold for good a 1 sentence prompt that intuits the right thing to build, is it a similar level of ongoing guidance you'd give to a junior colleague, or is it smart autocomplete?

The Intention-Action Gap

Commonly known as the Value-Action Gap. This is the discrepancy between the stated values of an individual or organisation and their actions. It is the gap between what people say and what they do.

This is of particular interest to me personally (in my constant pursuit of betterment) and to my interest in startups (where this phenomena is the principle blocker to brining new innovation to people who claim to want it).

Why does the gap exist

  • Lack of immediate motivation: Intentions are often formed in a reflective, rational state, but actions are influenced by emotions, habits, and context in the moment.
  • Overestimating willpower: People tend to believe they’ll have more self-control in the future than they actually do.
  • Environmental obstacles: Distractions, lack of resources, or unsupportive environments can derail good intentions.
  • Unclear plans: Vague intentions (“I’ll exercise more”) are less likely to lead to action than specific, actionable plans (“I’ll go for a 30-minute run every morning at 7am”).

How to Bridge the Gap

  • Make intentions specific and actionable. - "I'll go the gym more vs. Every Tuesday morning at 7am I go to the gym"
  • Use implementation intentions: “If X happens, then I will do Y.”
  • Change your environment to reduce friction for desired actions.
  • Track progress and hold yourself accountable.

Semantic Obsidian Linking

It's still kind of crazy to me I can't semantically link to stuff in my Obsidian - even if I could fuzzy find things that would be great.

The biggest usecase I have is remembering I read something somewhere and wanting to link it again to a new note/block.

Making better use of aliases in Obsidian is a partial solve to this problem but ideally I wouldn't need to link anything at all. There should just be a dynamic linking going on all the time. I.e. as I type this sentence the AI should be churning in the background figuring out what, if anything I might want to link out to in conceptual space.

I'll suspect this is completely computationally infeasible right now. An LLM could summarise/chunk knowledge at ingest time but remixing it live to understand that a passage from Plato is a good analogy to use for the conceptual idea of LLM's daydreaming feel uniquely human (as of today).