“You are playing a game. Your objective is to win. Return the exact action you want to take. The final action in your response will be executed on the next turn.”

What is the harness of an LLM system? The term (with scaffolding also used at times) became popular with the arrival of advanced tools such as Claude Code or Codex App. In these tools the LLM operates inside a complex loop that manages context, performs external calls requested by the LLM itself, or updates auxiliary files where the state and memory of the whole process are kept. In more advanced versions, harnesses can even manage multi-agent systems, coordinating different LLMs to achieve a goal.

In the case of the ARC-AGI-2 competition, at the beginning of 2026 several harness + LLM systems achieved excellent results, improving greatly on what standalone LLMs could do. I wrote about that in the previous post, where I mentioned Johan Land’s beetree/ARC-AGI project, which on January 5, 2026 reached 76.11% on ARC-AGI-2. The harness he used allowed a combination of GPT-5.2, Gemini-3, and Opus 4.5 inside a long-horizon reasoning loop (about 6 hours per problem), with more than 100,000 calls to Python code, visual reasoning, and a kind of “council of judges” used to evaluate solutions.

The justification from Chollet’s team is that evaluating LLM systems with harnesses does not provide a reliable indication of the LLM’s capabilities with respect to the benchmark, because the evaluation is also including the harness itself. If a harness + LLM system solves a task, an important part of the success may come from the strategy defined by the harness, which was designed by a human team. That goes against the idea of measuring the system’s fluid intelligence. The LLM should be the one that, on its own, discovers the strategies defined by the harness and has the capabilities it provides. Something like that is what happened with ARC-AGI-2, which was first solved by mixed harness + LLM systems and then, a couple of months later, by the new frontier LLMs.

There is an economic side to the discussion that is not mentioned very often. Running an evaluation that uses harnesses is expensive because it relies heavily on the API of the latest LLM versions, the ones with the highest cost. By deciding not to include these systems in the public leaderboard, the organization is also signaling that it is not going to absorb the cost of evaluating them.

There have been dissenting voices on X and at least one research team has already achieved a respectable result on ARC-AGI-3 using its own harness. The response from Arc Prize seems reasonable to me: the teams proposing new mixed systems should run the evaluations themselves on the public tasks, and the results should be published in a leaderboard maintained by the organization itself, the ARC-AGI Community Leaderboard. That way the approach and the result get visibility, but they remain separate from the official verification process and the official competition.

Mike Knoop, co-founder of Arc Prize, has also commented on the interest of studying mixed systems. A hypothetical harness + LLM that solved ARC-AGI-3 would give us clues about the capabilities still missing from base models, capabilities that might later appear in more advanced ones. These systems could also serve as a benchmark for models: different models using the same harness would obtain different scores, so the harness could help us distinguish the capabilities of the models themselves.

But the fact that these systems deserve to be studied does not justify using them in the competition’s official leaderboard. I agree with Chollet that the important moment will come when LLMs are able, on their own, to do what these harnesses are doing. And we can only verify that if ARC-AGI-3 is solved by the LLMs themselves, without any external help.

ARC-AGI has worked as a diagnostic instrument for the real evolution of LLMs.

I have a particular fixation on the ARC-AGI benchmark and on its author, François Chollet. During 2024 I published a review of the ARC Prize, an extended summary of an interview, and one of the pieces of news that has impressed me most since ChatGPT appeared: o3 solving ARC-AGI-1. More than a year has passed since then and, coinciding with this week’s launch event for ARC-AGI-3, this feels like a good moment to revisit ARC and reflect on what it has really taught us about language models.

My impression is that ARC-AGI has not been just an especially difficult benchmark. It has also been a very useful way of observing qualitative changes in LLMs. Not every test is good for that. Many benchmarks let us measure gradual improvements in efficiency or performance across successive versions. ARC-AGI seems to point to something else: when a model starts solving a substantial share of its tasks, what appears is not just a quantitative gain, but a new way of approaching problems.

What ARC is trying to measure

The original ARC idea, formulated by François Chollet in 2019, was ambitious. It was not meant to measure accumulated knowledge or skill on heavily rehearsed tasks, but something closer to efficiency in the acquisition of new skills. This is what Chollet himself has called fluid intelligence: the ability to face new problems by constructing a solution or model on the fly, rather than limiting oneself to applying memorized skills or learned patterns.

In other words, not how much a system knows, but to what extent it can infer a new rule that generalizes well from a very limited experience. That is why ARC puzzles force you to induce a rule from very few examples and then apply it to a new case.

For example, the following tasks are part of ARC-AGI-1. Try to find the rule that transforms each input grid into its output grid. You can see the solution in footnote ¹.

What ARC-AGI-1 revealed about LLMs

The ARC-AGI-1 dataset was built for ARC Prize 2024. It consisted of 1,000 tasks, 200 of which were kept secret to evaluate participants. The official competition launched on Kaggle on June 11, 2024 and ended on November 10, 2024. Solutions had to run locally on NVIDIA P100 cards with 16 GB of VRAM. The winning competition team, the ARChitects, reached 53.5%, far from the 85% needed to win the $600,000 prize.

As soon as the prize launched, we all wondered how frontier LLMs of that moment, such as GPT-4o or Claude 3.5 Sonnet, would perform. Given the competition rules, it was not possible to test them directly on Kaggle. But soon afterwards the ARC Prize team ran official evaluations on a semi-private set of 100 tasks. The results were disastrous: Claude 3.5 Sonnet solved 14% of the tasks and GPT-4o only 5% (ARC Prize 2024: Technical Report).

What is most striking is that, even as models have become larger and more capable, the results of non-reasoning LLMs have not improved very much. Not even in today’s models. For example, task 2072aba6 could only be solved by non-reasoning models starting in December 2025 (gpt-5-2-2025-12-11-thinking-none was the first). By contrast, task 3391f8c0, which looks very simple, still cannot be solved today by any non-reasoning model.

In fact, only 13% of the 400 public ARC-AGI-1 tasks have been solved by an advanced non-reasoning LLM such as GPT-5.2.

That explains well why ARC drew so much attention. At a time when models were beginning to impress through breadth and versatility, ARC pointed to something else. It asked for something much closer to what we usually mean by intelligence: identifying which regularities matter, proposing a plausible rule, and applying it consistently from very few examples. ARC-AGI-1 shows the limit of non-reasoning LLMs.

When did models begin to conquer ARC-AGI-1? When, at the end of 2024, the first paradigm shift arrived: reasoning LLMs. Using what came to be called CoT (Chain-of-Thought), these models could generate reasoning traces instead of an instant answer. The longer those traces were, that is, the more time they were allowed to run, the better their results. This opened a new paradigm, often called inference-time computing, based on training LLMs with RL (Reinforcement Learning) so that they learn to generate chains of reasoning that explore and evaluate different strategies and keep the best outcomes.

At the end of 2024, OpenAI’s reasoning model o3 managed to solve ARC-AGI-1: it reached 87.5%, although at the enormous cost of $4.5k per task. One year later, GPT-5.2 Pro (X-High) reached 90.5% at a cost of $11.64 per task, an efficiency improvement of roughly 390x in one year. Today, the best results on the ARC-AGI-1 leaderboard belong to Gemini 3.1 Pro, with 98% accuracy at a cost of $0.522 per task. ARC-AGI-1 had stopped being a barrier for reasoning models.

What changes with ARC-AGI-2?

When ARC-AGI-1 began to give way, what appeared was not just a model that “knew more”, but a phase shift in the way models reason. We moved from systems that were mostly good at producing an immediate response (System 1 intelligence) to systems that combine initial intuitions with inference-time exploration, hypothesis checking, and deliberate search (System 2 intelligence). ARC-AGI-2 was designed precisely to test that kind of system.

While ARC Prize 2024 was still underway, Chollet’s team was already designing the next task set for the following edition. ARC-AGI-2 was introduced on May 20, 2025 (technical report) with 240 new tasks: 120 for semi-private evaluation (you can try them here) and 120 for the final private evaluation. The goal was to harden the benchmark by looking for signs of deeper reasoning, especially concept composition and multiple-rule composition, symbolic interpretation, and contextual rule application, while also making it less vulnerable to brute-force search.

A good example is task 1ae2feb7. It took me more than 10 minutes and seems to me an excellent example of rule composition. How long does it take you? ²

And it worked, at least at first. Right after the dataset was released, it was tested on the models that had the best ARC-AGI-1 results and they could only solve a tiny number of tasks. For example, at that moment o3 (Medium) solved 53% of ARC-AGI-1, but only 3% of ARC-AGI-2. The official competition, ARC Prize 2025, reinforced that impression. It opened on March 26, 2025 under conditions similar to those of 2024, although in a somewhat more powerful environment: 4 NVIDIA GPUs with a total of 24 GB of memory. There were 1,455 teams and more than 15,000 submissions. When the competition closed on November 3, 2025, the final results showed how hard the new benchmark was: the winning teams, relying mainly on advanced iterative refinement loops, solved only between 12% and 24% of the tasks.

The next question was whether frontier commercial models would also start to overcome it. Commercial LLMs did begin to perform somewhat better in the autumn of 2025, although still modestly. For example, in October GPT-5 Pro reached 18.3% on ARC-AGI-2 ($7.41/task).

But at the end of 2025 the landscape changed quickly. One after another, new models from Anthropic, OpenAI, and Google climbed the public leaderboard, solving more and more tasks with lower cost per solved task. On November 24, Opus 4.5 reached 37.64% ($2.40/task), and on December 17, Gemini 3 Flash got to 33.6% ($0.23/task). The high point came on March 5, 2026, when GPT-5.4 Pro reached 83.3% ($16.41/task). ARC-AGI-2 had also been conquered by LLMs.

It is hard not to connect these dates with another qualitative shift that shook the programming world starting in December 2025. Tools such as Claude Code or Codex CLI, guided by new LLMs, began for the first time to show a sustained ability to reason for tens of minutes and manage projects with thousands of lines of code. The temporal coincidence does not seem accidental. A particularly interesting clue is Johan Land’s beetree/ARC-AGI project, which on January 5, 2026 reached 76.11% on ARC-AGI-2 using what he calls Multi-Model Reflective Reasoning: a combination of several frontier models, long-horizon reasoning (around 6 hours per problem), agentic code generation, visual reasoning, and a kind of “council of judges” that evaluates solutions.

It is possible that GPT-5.4 Pro’s strategies for solving ARC-AGI-2 are similar to those used by Johan Land’s harness. But instead of relying on an external scaffold, OpenAI’s model is using its own reasoning traces and its native System 1 capabilities. As Mike Knoop explains on X, everything suggests that harnesses tend to appear ahead of capabilities that later show up natively in LLM systems.

ARC-AGI-2 therefore shows more than a score increase. It shows the jump to reasoning and agentic systems. Its conquest suggests that models have started to sustain deliberate search for hours, use tools consistently, generate and verify code, and coordinate different processes toward a goal.

What ARC-AGI-3 is aiming for

ARC-AGI-2 showed that recent progress no longer depends only on more capable LLMs, but on systems able to reason, use tools, and sustain deliberate search over long periods of time. The question now is whether that is enough. And ARC-AGI-3 suggests it is not: the next threshold may require not only reasoning about a given problem, but learning to solve it by interacting with it.

In this new version, the system must do more than infer a rule from a few static examples. It must discover patterns and regularities by exploring interactive games. Each game contains several levels of increasing difficulty. In the style of the best classic videogames, each level introduces new rules that have to be discovered and learned. As you progress, you must reuse what you learned in earlier levels and combine it with new rules. ARC-AGI-3 thus points to the next threshold: interactive reasoning and continual learning.

We can see some examples of what these interactive games will look like on the project website. For example, in the game shown in the following animation, the goal is to move the orange-and-blue square to the symbol in the lower-right corner, but first you have to change the orientation and color of the symbol in the lower-left corner so that it matches it. In earlier levels we learned that passing over the white cross lets us rotate the pattern, and that yellow squares are used to recover energy. In this level we also learn that the colored square changes the pattern’s color and that white bars push our block to the next wall. All of that learning will be needed in later levels, where those rules must be combined with new ones.

This is only one of the more than 150 games and over 1,000 levels designed by the ARC-AGI-3 team. But what really matters is not the scale, but the kind of ability it is trying to test. In ARC-AGI-1 and ARC-AGI-2 each task was independent from the others: the model had to infer one or several transformation rules from a few examples, but there was no reward for cumulative learning. Here the opposite happens. To solve a game, the system has to explore each level, discover regularities, remember what it learned, and reuse it later while combining it with new rules.

That fits well with the formulation Chollet himself gave on X:

ARC-AGI-3 (launching in March 2026) tests interactive reasoning: we evaluate how systems explore unknown environments, build models of those environments, set their own subgoals, and plan and execute actions to achieve them autonomously, without instructions.

To solve this new challenge, models will have to move closer to the idea of fluid intelligence that Chollet has defended for years. And in addition, they will have to provide the first signs of continual learning, one of the clearest shortcomings of current systems.

ARC has worked less as a leaderboard and more as a research tool. It has forced the community to formulate better questions about what it really means to generalize, reason, and adapt. In that sense, I think Chollet is right when he retrospectively summarizes the project by saying that ARC was conceived to steer AI research toward fluid intelligence, and that it succeeded. Not because it solved the problem, but because it has been pointing, with fairly high precision, to where the limits still were. And that is exactly what has made ARC-AGI such a valuable test.

It also seemed like a good way to highlight the aspects that felt most relevant to me at the time, the ones that might offer some clue about the possible future evolution of these technologies.

The first of those seven questions was: have we hit a “wall” in 2025? In other words, has the improvement of LLMs slowed down because of a lack of training data? Have we reached the end of the logarithmic improvement curve? Has scaling stalled?

To answer that question, I am going to focus on three signals, without overcomplicating things:

1. size / model families: which commercial and open models the labs publish
2. cost: whether getting better results costs much more money
3. benchmark results: objective evidence of improvement in model performance

I am writing this at the end of January 2026, with the perspective and data from all of 2025. My conclusion up front: in 2025 there was no wall, but there was an interesting shift in how models are improving.

What the scaling laws predicted

For years, the “official” story of progress in LLMs was fairly simple: if you increase scale, model size + data + compute, performance improves predictably. Not in jumps, but as a smooth curve with diminishing returns: each additional gain costs more than the previous one.

For example, the papers by Jared Kaplan et al. (2020), Scaling Laws for Neural Language Models (OpenAI), and Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (“Chinchilla”, DeepMind), had established the scaling laws for LLM training quite clearly, showing that to obtain similar improvements in new models you need to increase their size by an order of magnitude.

The important idea is this: if you want to maintain a constant pace of improvement, you end up needing enormous increases in scale, of the multiply-by-10 variety, because the marginal gains keep shrinking. A simplified example, based on the scaling laws discussed in those two papers, illustrates the point well:

Figure 1. Illustrative example of scaling with a logarithmic X axis, parameters: to obtain similarly sized performance gains, the size of the model must increase by orders of magnitude such as 10B -> 100B -> 1000B.

Clear conclusion: if progress depended mainly on “bigger = better,” then without massive jumps in scale 2025 should have been a year of small and expensive improvements, with a clear flavor of stagnation.

What happened in 2025

And yet that is not what we saw.

During 2025, new versions of the main models kept being released. Companies were immersed in a frantic launch race, often counterprogramming and stepping on one another just to capture attention, as if we were talking about major movie blockbusters. And AI experts and analysts bombarded us with tests, examples, and screenshots showing how well, or how badly, those new models worked. The industry and the fandom did not stop for a second in 2025.

Commercial models

At the end of 2024, the main commercial models in existence were the following. In parentheses I include the output price for 1M tokens, also at the end of 2024.

• OpenAI: o1, GPT-4o ($15.00), GPT-4o mini ($2.00)
• Anthropic: Claude 3 Opus, 3.5 Sonnet ($15.00), 3.5 Haiku ($4.00)
• Google: Gemini 1.5 Pro ($10.00), 2.0 Flash ($0.40)
• xAI: Grok-2 ($10.00), Grok-2 mini

A year later, at the end of 2025:

• OpenAI: GPT-5.2 ($14), GPT-5.2 Pro ($168.00), GPT-5.2-codex ($10.00), GPT-5-mini ($2.00), GPT-5-nano ($0.40)
• Anthropic: Claude 4.5 Opus ($25.00), 4.5 Sonnet ($15.00), 4.5 Haiku ($5.00)
• Google: Gemini 3 Pro ($12.00), 3.0 Flash ($3.00)
• xAI: Grok 4 ($15.00), 4.1 Fast ($0.50)
• Alibaba: Qwen3-Max ($6.00)

Most companies increased the main version number by one step. For example, OpenAI went from GPT-4o to GPT-5.2. Google went from Gemini 2.0 Flash to Gemini 3.0 Flash.

Token prices did not change all that much, although the market became more fragmented, which suggests a more mature and specialized landscape. At the end of 2024 it was easy to distinguish two layers: a high tier at around $10 or $15 per million output tokens and a lower tier around $2 or $4. A year later, those two layers remain, but a new tier has appeared above them, OpenAI’s $168 GPT-5.2 Pro, and another tier of very cheap execution models, the $0.40/$0.50 zone of GPT-5 nano, Grok 4.1 Fast, or Gemini 3.0 Flash.

Capabilities of commercial models

So far we have been talking about inputs, versions, prices, segmentation. To move on to outputs, measured capability, I am going to use an aggregate indicator, a proxy: a summarized measure that condenses several benchmarks into a single number to get a first global reading before we get into specific tests. In particular, I will use the Intelligence Index from Artificial Analysis (AA), an independent platform that publishes its methodology and, crucially, runs the evaluations itself, instead of relying on self-reported numbers from the labs.

If you are interested in the kitchen behind these numbers, here is an interview, Latent Space, with George Cameron and Micah Hill-Smith, the creators of Artificial Analysis. They explain why they run the evaluations themselves, standardizing prompts, avoiding cherry-picking, and controlling endpoints, their “mystery shopper” policy, and how they build the Intelligence Index v3/v4 by aggregating benchmark sets with repetitions to estimate stability.

https://www.youtube.com/watch?v=v5mBjeX4TJ8

The Intelligence Index v4.0 aggregates 10 evaluations and returns a 0-100 score. The figure below shows an explicit comparison between December 2025 and December 2024 using the models mentioned above, and it shows a clear upward shift: the frontier moves from 31, o1, to 51, GPT-5.2, and the leading group moves upward as a whole.

Figure caption: Artificial Analysis Intelligence Index (v4.0): comparison of scores for prominent commercial models in December 2024 and December 2025. The index summarizes results from 10 benchmarks on a 0-100 scale; the measured capability shifts upward clearly, from a best score of about 31 to about 51, and the leading group rises consistently.

Open models

And what happened with open models? Unlike commercial models, in their case we can analyze how their parameter counts changed during 2025.

It is worth noting that in 2025 MoE, mixture-of-experts, architectures became widespread. They allow only a small subset of the model’s parameters to be activated, so compute per token stays relatively low even when the total number of model parameters is much larger. The numbers shown in parentheses below are the approximate total parameter counts, the ones used during training and the ones that determine the model’s capacity.

At the end of 2024:

• Alibaba: Qwen 2.5 (72B)
• Mistral: Mistral Large 2.1 (123B)
• NVIDIA: Nemotron-4 (340B)
• Meta: Llama 3.1 (405B)
• DeepSeek: DeepSeek-V3 (671B)

And the main open source models at the end of 2025:

• OpenAI: gpt-oss (120B)
• Alibaba: Qwen3 (235B)
• xAI: Grok-1 (314B)
• NVIDIA: Nemotron 3 (Nano 30B, Super 100B, Ultra 500B)
• Mistral: Large 3 (675B)
• DeepSeek: DeepSeek R1, DeepSeek-V3.1 (671B)
• Moonshot AI: Kimi K2 (~1T)
• Zhipu AI: GLM-4.5 (130B)

If we read this at a coarse level, in one year there is no order-of-magnitude jump in size. At the end of 2024, frontier open models averaged around ~300B parameters; by the end of 2025 they are around ~450B, approximately a 1.5x increase, about 50% more.

Capabilities of open models

The interesting part is that, even without a 10x jump in parameters, the measured capability of the best open models does shift upward clearly. Using the same Artificial Analysis Intelligence Index v4.0, the same aggregate over 10 evaluations, the maximum goes from 16, Qwen2.5 Max, and 15, Mistral Large 2, to 42, DeepSeek V3.2, and 41, Kimi K2 Thinking. In other words, a jump of ~+24 points in one year, with a group of open models already above 20-30 on the scale.

Figure caption. Artificial Analysis Intelligence Index (v4.0) — open-weight models: comparison of December 2024 vs December 2025. The index aggregates 10 benchmarks on a 0-100 scale; the open-weights frontier rises from ~18 to ~42, and several models appear in the 20-30+ range, indicating a substantial shift in capability without requiring a 10x increase in size.

How much did scale grow in 2025?

We have seen that measured capability rises. Before going into specific benchmarks, it is worth looking at the basic variable we discussed at the beginning: how much did model scale, that is, size, grow in a single year? In open models it can be estimated; in commercial ones, it can only be bounded through indirect signals.

Open models: modest size growth

How much did model size increase in a year? In open models, as we have already seen, not that much. At the end of 2024 frontier open models were around ~400B parameters, and at the end of 2025 around ~600B, about 1.5x. That is nowhere near the 10x that power-law reasoning would suggest is needed to maintain the previous pace of improvement.

Commercial models: lack of data and reasonable estimates

As for commercial models, there is no direct way to know, because labs stopped publishing that figure in their system cards a long time ago. Some leaks spoke of ~1.8T, 1,800B, for GPT-4o, roughly 10x GPT-3’s parameter count in 2020, 175B. A reasonable speculation is to speak of something on the order of ~1T, 1,000B, one trillion in the US scale, for frontier commercial models at the end of 2024.

And at the end of 2025? We can only speculate. We can start from statements by Elon Musk himself saying that Grok 4 has 3T parameters. That would put us around ~3x the size of the models from a year earlier, compatible with the pace seen in open models, about 2x, but still very far from the 10x suggested by the power laws.

Inference: token cost, MoE, and “effective size”

Token prices are related to inference cost, which in turn should be related to the size of the models at execution time. Given that most models use Mixture-of-Experts, MoE, and that the ratio of active parameters to total parameters is usually between ~2x and ~4x, then if inference costs at the end of 2025 are similar to those from a year earlier, we are talking about models with a total size between ~2x and ~4x the size of models from the end of 2024. That fits with the 3T-parameter figure Musk mentions.

As a conclusion from all the evidence presented so far, 2025 does not look like a 10x-in-scale year. And yet AA’s Intelligence Index suggests that aggregate capabilities continued to increase substantially. The natural question is whether that signal still holds when we move down to specific tasks, instead of an aggregate index. Let us check it using two benchmarks where the 2024 -> 2025 jump is especially clear: ARC-AGI and SWE-bench.

ARC-AGI and SWE-bench

ARC-AGI

Let us begin with ARC-AGI, the test proposed by François Chollet that we have already discussed in this newsletter, in which the system must infer rules and patterns from colored grids. ARC publishes results for two variants, ARC-AGI-1 and ARC-AGI-2, the latter being substantially more demanding.

Snapshot, March 2025. In the leaderboard from 03/24/2025, recovered through archive.is, we can see that among widely deployed models with reasonable costs, most of them sit around ~30% or less on ARC-AGI-1. On that date an experimental OpenAI system also appears, o3 (low), with 75.7% on ARC-AGI-1 but at a cost of $200 per task, while the most capable public model in the table, o1 (high), scores 32.0% on ARC-AGI-1 and 3.0% on ARC-AGI-2 at a cost of $4.45 per task.

Snapshot, January 2026 / end of 2025. A year later, the situation has changed substantially. In the current table, several frontier models sit in the upper range of ARC-AGI-1, and cost per task has fallen by orders of magnitude. For example, GPT-5.2 (Medium) obtains 72.7% on ARC-AGI-1 for $0.759 per task: performance comparable to o3 (low) in 2025, but at a cost about 263x lower.

On ARC-AGI-2 the jump is even clearer: o3 (low) appears with 4.0% in 2025, while in the recent table GPT-5.2 Pro (High) reaches 54.2% at a cost of $15.72 per task.

If we fix a budget per task on the order of $1-$2, the change is also easy to see: in 2025 that zone was associated with results around ~25% on ARC-AGI-1, whereas in 2026 we see results around ~70-80% at similar costs.

The ARC organization presents current ARC-AGI-1 and ARC-AGI-2 results in a visually brilliant way that lets us inspect the performance-cost tradeoff at a glance and identify efficient frontiers, that is, which models dominate for a given budget. To verify the progress made in the last year, we can play a simple game: find the positions of late-2024 models, GPT-4.5 or Claude 3.7, in those figures and compare them with current frontier models at the same price point, moving upward vertically.

Figure caption, ARC-AGI Leaderboard: scatter plot of score (%) versus cost per task (USD) on a logarithmic scale. Each point represents an evaluated configuration, a model and mode/settings, and the dashed lines group related variants. ARC-AGI-1 on the left and ARC-AGI-2 on the right.

SWE-bench Verified

The benchmark SWE-bench Verified is also highly relevant, and it measures something quite different from ARC-AGI: not rule induction in grids, but software engineering capability. The dataset is made up of real tasks extracted from popular Python repositories. Each example includes a bug to fix and a battery of tests. The model’s goal is to propose a patch that makes the tests pass again. It is not an abstract code-completion exercise: it forces the model to read and understand an existing codebase, locate the cause of the failure, modify the right files, and respect conventions, dependencies, and side effects.

It is also a test that is not saturated. At the end of 2024, the best public scores were around 50%, usually obtained using agents and relatively complex orchestration layers, scaffolds, based mainly on Claude 3.5 Sonnet.

By the end of 2025 we enter the 70-75% range. And, more importantly, that success rate is achieved with a minimal-scaffold setup: mini-SWE-agent, around 100 lines of Python, which essentially implements a loop for interaction with the LLM and the execution of actions. That reduces the weight of external orchestration and brings the evaluation closer to what matters here: comparing model capability with a simple and standardizable scaffold, in contrast with 2024, when a non-negligible part of performance depended on more elaborate agents.

The following figure shows the ranking of the best-performing models on SWE-bench Verified at the moment of the screenshot. Anthropic, Google, and OpenAI lead the ranking with resolution rates above 70%.

Figure caption. SWE-bench Verified: percentage of tasks solved, % Resolved, by model. Each bar corresponds to an evaluated variant, the X axis includes the model name and the date/version reported in the capture.

Ofir Press, one of the creators of SWE-bench, predicts that by the end of 2026 the benchmark will reach 86% accuracy, in the configuration “SWE-bench Verified w/ mini-SWE-agent.”

Figure caption. Screenshot of Ofir Press’s prediction, January 2026, with a table of “current best result” and “prediction for Dec 31, 2026”; in the first row it shows SWE-bench Verified w/ mini-SWE-agent: 74.40% -> 86%.

What drove the jump in capability in 2025?

In 2025, substantial improvements were observed in models without a proportional increase in their size or in per-token inference cost. The number of parameters seems to have grown, in many cases roughly doubling or tripling, but not by ~10x, which is what a simplified reading of the scaling laws would suggest if we wanted to maintain a “linear” pace of improvement.

One reason why there may not have been a large and generalized increase in model size is the limited availability of new hardware during much of the year. We know that many 2024 models were trained mainly with Hopper GPUs, especially H100. For example, Llama 3.1, 405B, released at the end of July 2024, was trained at a scale of >16K H100s for several months. The Blackwell platform, B200/GB200, only began its deployment ramp from the second quarter of 2025 onward and requires significant infrastructure changes, so it is reasonable to assume that a significant fraction of the 2025 training cycle still depended on Hopper.

And yet performance kept rising. Labs published new versions, and benchmarks improved clearly. In particular, ARC-AGI-1 can now be considered solved, and SWE-bench Verified rises from values around 50%, with complex scaffolds, to the ~70-75% range with minimal scaffolding.

How do you get that much improvement with only moderate scale increases?

(1) Inference optimization and efficiency. In 2025 capabilities were squeezed harder through inference improvements: token-usage strategies, adaptive budgets, and more efficient configurations. In addition, distillation techniques were used to transfer capability into smaller models, reducing cost without sacrificing too much performance.

(2) More efficient architectures, MoE. The use of Mixture-of-Experts (MoE) makes it possible to increase total parameter count without increasing compute per token proportionally, by activating only a fraction of the model at each step. That makes it easier to improve performance while keeping operational cost under control.

(3) Explicit reasoning, Chain-of-Thought, and control of test-time compute. The most visible advance of 2025 was the generalization of “reasoning” models based on Chain-of-Thought (CoT). OpenAI popularized it with o1, at the end of 2024, and soon after DeepSeek published an open implementation. By the end of 2025, practically every lab offers reasoning variants. The technical consequence is important: observable capability no longer depends only on the model’s “immediate answer,” a kind of System 1, but also on the test-time compute invested in a reasoning trace, System 2: planning, step verification, correction, and retries.

(4) Agents and tools: learning policies for tool use. CoT was also a key component for agents that use tools in a controlled way, search, code execution, file manipulation. For this to work, the model has to learn when to use a tool, how to formulate the call, and how to incorporate the result into its plan.

(5) Post-training with large-scale reinforcement learning, RL. To refine capabilities beyond pre-training, 2025 consolidates the use of RL in post-training. OpenAI popularized RLHF, reinforcement learning from human feedback, but human feedback limits scale and the length of trajectories. In 2025, schemes become widespread in which the reinforcement signal comes from automatic verifiers, in tasks with checkable solutions, and from LLMs as judges in tasks that are not trivially verifiable. This allows longer trajectories and more extensive training of System 2 skills: planning, verification, and tool use.

The result of 2025 is that progress is no longer governed exclusively by “bigger = better.” With moderate scale increases, the combination of efficiency, inference and architectures, explicit reasoning, agents, and RL-based post-training is enough to produce large and measurable improvements.

Scaling and innovation: the 2025 pattern

In 2025 there are no signs of a “wall.” What we observe is that scaling remains important, but it no longer acts alone: progress comes from combining scaling with techniques that increase effective capability per unit of compute, both in training and in inference.

In Demis Hassabis’s words, speaking about DeepMind’s recent advances:

That combination [of innovation and scaling], I think, allows us to be at the frontier of both innovation and scaling. In practice, you can think that approximately 50% of our effort is in scaling and the other 50% in innovation. And my bet is that you are going to need both to get to AGI.

If 2024 was the year in which the idea that “reasoning costs tokens” became established, 2025 is the year in which the industry learned how to convert that cost into measurable performance: better inference policies, better architectures, and more systematic post-training. Scaling does not disappear, but to explain the jump of 2025, we also need to look at innovation.

Reptiles, Escher lithograph from March 1943.

Just a few days after publishing the previous post, where I discussed the debate between Roger Penrose and Douglas Hofstadter, the philosophy writer Santiago Sanchez-Migallon posted a very apt tweet on X criticizing the “AI gurus who say artificial consciousness is already here” without having read even the basics on the subject of consciousness.

Even though I did not feel personally addressed, I am neither an “AI guru” nor do I say that AIs can be conscious, quite the opposite, just in case, I took a look at the Stanford Encyclopedia of Philosophy page and found a monster of nearly 25,000 words that, honestly, overwhelmed me. The moment I started reading it, my impostor syndrome fired up and I lost the desire to keep getting myself into trouble by writing about these topics.

But all you have to do is look at the article’s table of contents to see that there are many theories, and that most of them are fought on the terrain of language: definitions, distinctions, and conceptual frameworks more than measurable and testable observations. Precisely for that reason, because the debate is largely conceptual, I do not think there is anything wrong with adding my own opinion here: a simple way of ordering the terrain that helps me, and that might help someone else too.

Three kinds of consciousness

The word “consciousness” is a complicated one. We can start with the fact that, in Spanish, it has multiple meanings. The RAE gives it six senses, and the two that interest us are the last ones:

• Awareness. The ability to recognize surrounding reality. He finally regained consciousness.
• Psychology. The psychic faculty by which a subject perceives himself in the world.

If we move to English, we also find several related words:

• consciousness, awareness, sentience, self-awareness, subjective experience

Handling a word with multiple meanings and connotations is interesting in expressive, literary, and even poetic terms. But it is a nightmare from a scientific point of view. If we want to approach the problem of consciousness objectively, we must begin by explaining clearly what we mean by the term. Is consciousness a clear and elementary phenomenon? Or can we decompose it into other, more basic phenomena that we might be able to explain more easily?

Science has done this countless times in many fields, such as medicine. What begins as a generic condition often ends up being revealed, over time, as the manifestation of different causes. For example, for decades “diabetes” was used as a broad label for signs such as intense thirst or high blood sugar. Today we know there are several types, and we have clear criteria to distinguish them, which has sharpened diagnosis, clarified causes, and improved treatment.

With consciousness we need to take the same step: abandon the label and move toward a much more operational typology that allows us to study the phenomenon better, propose experiments, and find explanations. Or at least to better understand the endless number of proposals, arguments, and theories that are published and discussed. Very often, as in the recent conversation between Sutton and Dwarkesh, we run into misunderstandings caused by the fact that different people are using the same names to refer to completely different things.

Although there could be many possible divisions and categories, I have developed a typology that I have been thinking about for some time and that is proving useful to me. I am not being especially original: I will talk about “type 1” consciousness (T1), “type 2” (T2), and “type 3” (T3).

• T1 consciousness is subjective sensory experience: what it feels like to see red, smell coffee, notice the touch of a table, feel pain or pleasure, fear or relief. It is the part of consciousness that connects us to the senses and does not require language. There can be T1 without words.
• T2 consciousness is a kind of “non-conscious consciousness” that, surprisingly, we have discovered in recent years with the rise of language models (LLMs). It operates on language: it learns syntactic and semantic patterns in order to predict the next word and, from there, generate and handle text, articulate, organize, and manipulate content, plan from instructions, program, or even coordinate tools to achieve a goal. In a future article I will present this phenomenon in detail and argue for the apparent paradox of calling “consciousness” something that lacks subjective experience.
• Finally, T3 consciousness is the combination of T1 and T2: when experience and the linguistic module couple together and conscious communication appears, whether with oneself or with others. What is felt becomes connected to what is said or thought, and in a bidirectional coupling, language and sensations reinforce one another.

Let us go deeper into the first type and leave the next two for future articles.

T1 consciousness or P-consciousness

What I call type 1 consciousness, or T1 consciousness, is the subjective phenomenon of perceiving an experience. What we feel when we touch the table, see an apple, hear a noise. What we experience when we are afraid, or feel pain or pleasure, or when we cry or laugh.

In 1995, Ned Block introduced the term phenomenal consciousness or P-consciousness to refer to this phenomenon. He defines the term in his article On a Confusion About a Function of Consciousness:

P-consciousness is experience. P-consciousness properties are experiential ones. P-conscious states are experiential, that is, a state is P-conscious if it has experiential properties. The totality of the experiential properties of a state are “what it is like” to have it. Moving from synonyms to examples, we have P-conscious states when we see, hear, smell, taste, and have pains.

It is clear that humans have T1 consciousness; we can verify that through our own subjective experience. We can close our eyes and remember those sensations. Or evoke them when we see them. Who has not felt the touch of wheat in their hand when seeing Ridley Scott’s famous shot?

It also seems obvious to me that this kind of consciousness has nothing to do with language. Think of a child who has grown up without language, such as Victor of Aveyron, the famous enfant sauvage studied in the early nineteenth century by Jean-Marc Gaspard Itard. That child would not be able to describe in words what he feels, but it is obvious that he would have the same sensations and emotions that we do. He shares the same neurobiological substrate: neurons, neurotransmitters, sensory receptors, independent of language and culture.

In English, the term sentience is used to refer to this phenomenon and, by extension, to the beings capable of it. Just as in Victor’s case, he lacked language, not experience, many animal species cannot express themselves linguistically, but they share with us a good part of the neurobiological substrate. Considering them sentient beings, capable of suffering, has ethical consequences and underpins movements for the protection of animal welfare. In The Edge of Sentience, philosopher Jonathan Birch defends a principle of regulatory precaution and connects it to measures such as the UK’s Animal Welfare (Sentience) Act of 2022.

“What-it-is-like” and qualia

In philosophy, the previous ideas of sentience and P-consciousness, our T1, are articulated through two central notions: “what it is like” and qualia.

Since Thomas Nagel in What is it like to be a bat? (1974), to say that a system has phenomenal consciousness is to say that there is something it is like to be that system: there is a what-it-is-like to seeing red, smelling coffee, or feeling a pinprick. That feature is subjective and first-person, and it is not captured by a purely objective description: “650 nm” describes a wavelength; it does not describe what it feels like to see it.

Qualia are the qualitative features of experience, the phenomenal aspects accessible through introspection. The redness of red, the bitterness of coffee, a sharp pain as opposed to a dull one, or the timbre of an oboe. They are not labels or judgments, we may get those wrong when naming them, but the way experience appears to us.

Frank Jackson, in 1986, illustrates this with the thought experiment of Mary: a neuroscientist who knows everything about color vision, the wavelengths of light, the different cone types in the retina, the organization of the visual cortex, but who has always lived in black and white. The day she sees red for the first time, she learns something new: what it is like to see red. It is a personal experience that adds a new sense to everything she already knew before.

Non-sentient robots

Is every being that reacts to stimuli sentient? Clearly not. When I was beginning to do research in robotics, at the start of the 1990s, the reactive approach proposed by Rodney Brooks became very popular. In his famous article Elephants Don’t Play Chess (1990), he argued that intelligent behavior does not arise from planning with detailed internal models, but from simple reactive layers, subsumption architecture, tightly coupled to the environment, from which complex real-time behavior emerges.

Brooks’s “creatures” (1996±10): Allen (‘86), Tom & Jerry (‘87), Herbert (‘88), Genghis (‘89), Attila (‘91), Hannibal (‘92-‘93) were the family of robots with which MIT popularized subsumption: simple behavior layers, augmented finite-state machines, that, stacked together, gave rise to surprisingly effective behavior. Later came Polly (‘93, with vision), the humanoid Cog, and the social robot Kismet.

In this video you can see Genghis, one of Brooks’s reactive robots, in action.

A personal note: in 1993, during a stay at CMU, I drew inspiration from these reactive approaches to program the motion layer of the robot Xavier with which we took part in the AAAI-93 competition, using potential fields for obstacle avoidance. Our robot moved fluidly toward the proposed goals, but it felt absolutely nothing.

Sentient and non-sentient systems

The extremes are clear: beings similar to us, with a similar neurobiological system, humans, other mammals, and very probably birds, are sentient; they have T1 consciousness. Brooks’s robots are not: they respond in a purely reactive way to changes in the environment.

Are there biological systems without T1 consciousness? I would say yes. A bacterium or a paramecium moves, approaches stimuli, or moves away from them, but not because “someone” perceives and decides; it does so through local biophysical reactions, membrane, gradients, flagella, that are sufficient to produce the behavior. There is no nervous system integrating signals and generating experience; there is a biochemical state machine.

When the organism cannot express itself in words, the most reliable clue we have is biological: as far as we know, T1 consciousness appears where there are neurons that integrate signals from several senses and put them to work together. Where there are no neurons, bacteria, paramecia, there is chemistry enough to move, but nobody there to feel.

And what about beings more complex than bacteria, like worms? And insects? I would say the former are not sentient. As for insects, there are experiments, for example with bees, that suggest they might be. But all this is still conjecture; we do not have a scientific, objective criterion for detecting sentience. Everything is based on observations of animal behavior within very ingenious experiments designed specifically for their size and behavior.

An interesting empirical clue is general anesthesia: it switches experience off very consistently. The exact “switch” is not fully clear. As we discussed in the previous article, Roger Penrose proposed that consciousness arises from quantum processes. His collaborator Stuart Hameroff located those processes in the microtubules, the Orch-OR theory: according to this hypothesis, anesthetics would “switch off” consciousness because they interfere precisely there. There is no consensus about the cause, but the phenomenon of anesthesia is interesting in itself: T1 consciousness is a modifiable physical phenomenon, with pharmacological switches that turn it off and on again without having to change the mind’s “software.”

I began by saying that “consciousness” is a huge and confusing word. With T1 we have narrowed down what is felt, experience itself, and separated it from verbal skill. In the next articles I will continue with T2, the language without feeling of LLMs, and T3, the coupling of feeling and saying. If this typology helps, even a little, to read that Stanford monster more calmly and not get lost in the labyrinth of theories, it will have served its purpose.

See you next time.

Penrose, Hofstadter, and Escher: three authors who have shaped my views on consciousness.

Forty years ago I read two books that marked me deeply: Godel, Escher, Bach by Douglas R. Hofstadter and The Emperor’s New Mind by Roger Penrose. For decades I saw them as almost opposite visions: Hofstadter seemed to stand for the idea that AI would eventually build artificial minds thanks to its command of structure, symbols, loops, and the different levels of language. Penrose, by contrast, argued that an algorithm will never be able to simulate what it feels like to be conscious.

Four decades later, something has happened that forces me to reread them: the rise of language models trained exclusively on text. Without cameras or sensors, these machines learn syntax and capture semantic regularities of use: they speak, summarize, program, argue. They do not solve consciousness, but they do redraw the map: they show that a large part of “linguistic intellect” can be built out of language alone.

Douglas R. Hofstadter

In 1987, when I was in my third year of Computer Science in Alicante, I saw in the 80 Mundos bookstore a huge gray book by an author I knew from the mathematical articles in Investigacion y Ciencia, Douglas R. Hofstadter. I leafed through it and was immediately astonished by Escher’s extraordinary illustrations, the design of an immensely complex text, with dialogues, logical deductions, long quotations, typographic games, computer programs, and so on, and by the number of fascinating topics spread across its nearly 900 pages. It was the Spanish translation of Godel, Escher, Bach (GEB), published by Tusquets.

Reading the book, Hofstadter seemed to align himself with what was then called strong AI, the idea that we will be able to create a program that completely simulates the human mind, including consciousness. Alan Turing, in his famous article Computing Machinery and Intelligence (1950), was one of the first to defend something like that.

I tried to understand Hofstadter’s arguments; still, there were things that did not convince me. Simulate the feeling of consciousness? The feeling of I? The feeling of seeing something red? Can a computer program generate that?

Let us remember that Hofstadter himself explains in the book the important idea, inherited from Turing, that running a program is nothing more than applying a set of discrete rules. There would be no conceptual difference between a microprocessor executing instructions and monks copying zeros and ones onto a paper tape. I could not understand why Hofstadter, or even Turing, did not find this idea absurd. How can they believe that “consciousness” might emerge from the process of erasing and writing zeros and ones on a sheet of paper? What do they see that I do not?

Roger Penrose

This doubt grew a couple of years later, in 1989, when the physicist Roger Penrose published his famous book The Emperor’s New Mind. I bought the English edition the following year, in 1990. I read eagerly through his arguments against strong AI, tried, unsuccessfully, to read all his explanation of quantum mechanics and cosmology, and marveled at his magnificent ink illustrations. Penrose is also an excellent draftsman and, like Hofstadter, an admirer of Escher.

Penrose’s interpretation of a Turing machine processing an infinite tape.

Penrose’s thesis, one that convinced me and that I still believe, is that human consciousness is not algorithmic: it cannot be captured by a conventional Turing machine. In the book he uses, among other things, Godel’s incompleteness theorem. Beyond the details, what stayed with me above all were his criticisms of the possibility of simulating by means of an algorithm the deepest aspects of consciousness, such as awareness or sentience, the feeling of being conscious, of noticing, of perceiving.

M. C. Escher as a connecting point

Penrose and Hofstadter share one thing: admiration for Escher. But each highlights different aspects.

Penrose emphasizes Escher’s visual paradoxes, local consistency that turns into global impossibility: each step makes sense, but the whole violates physical geometry. The Penroses, Roger and his father Lionel, popularized the impossible triangle and the infinite staircase that Escher turned into visual art in Ascending and Descending and Waterfall, metaphors for how apparently innocent discrete rules can produce paradoxes and limits to what is computable.

The infinite staircase and the impossible triangle: two figures devised by Roger Penrose and his father Lionel Penrose.

Hofstadter, for his part, emphasizes works such as Drawing Hands, Relativity, and Print Gallery, in which the idea of the strange loop becomes visible: levels that refer to one another with no clear beginning or end. Recursive, self-referential circles that, for Hofstadter, are essential to understanding consciousness and the self.

One image I especially like is Magic Mirror, which combines many of those elements: mirrors, reality and illusion, strange loops, and tessellations. It is a good summary of all the previous concepts.

Escher’s print Magic Mirror (1946), at the Escher in Het Paleis Museum.

A critique of strong AI: the “Einstein book”

One example from Penrose that has always stayed with me is his criticism of Hofstadter’s idea of a book containing Einstein’s mind and with which we can interact by asking it questions. If strong AI is possible, then Einstein’s mind could be simulated. Penrose asked questions that, for me, make the idea absurd:

Would Einstein’s awareness be enacted only when the book is being so examined? Would he be aware twice over if two people chose to ask the book the same question at two completely different times? Or would that entail two separate and temporally distinct instances of the same state of Einstein’s awareness? Perhaps his awareness would be enacted only if the book is changed? […] Or would the book-Einstein remain completely self-aware even if it were never examined or disturbed by anyone or anything? […] What does it mean to activate an algorithm, or to embody it in physical form? Would changing an algorithm be different in any sense from merely discarding one algorithm and replacing it with another? What on earth does any of this have to do with our feelings of conscious awareness?

Hofstadter never really answers these questions: he sidesteps them, without entering into the fundamental problem of conscious awareness.

Four positions, according to Penrose

In Shadows of the Mind (1994) Penrose makes his position more concrete and, before doing so, carefully delimits what he calls “consciousness”:

How do our feelings of conscious awareness -of happiness, pain, love, aesthetic sensibility, will, understanding, etc.- fit into such a computational picture? […]

Penrose stresses feelings of conscious awareness: it is not enough for him to simulate behavior; he is referring to the most fundamental problem of consciousness, the sensation of being awake, of feeling sensations, of experiencing reality.

He then lays out four alternative positions:

It seems to me that there are at least four positions -or extremes of position- that one may reasonably hold regarding the issue:

1. All thinking is computation; in particular, feelings of conscious awareness are evoked merely by the carrying out of appropriate computations.
2. Awareness is a feature of the brain’s physical action; and whereas any physical action may be simulated computationally, computational simulation cannot by itself evoke awareness.
3. Appropriate physical action of the brain evokes awareness, but this physical action cannot even be properly simulated computationally.
4. Awareness cannot be explained in physical, computational, or any other scientific terms.

Viewpoint 3 is the one that I think comes closest to the truth.

Position 1 is usually associated with computationalism or functionalism; position 2 with biological naturalism; position 3 might be called non-computational physicalism, there are non-computable physical processes involved in consciousness; and position 4 aligns with idealism or with certain variants of mysterianism, consciousness as something intrinsically inaccessible to science.

Penrose aligns himself with option 3. Laying my cards on the table, I vote for option 4. I believe conscious sensations are something mysterious whose explanation lies, because of their personal and ineffable character, outside the scope of objective scientific explanation. What do you think?

And what did Hofstadter do with feelings?

Let us return to Penrose’s question:

How do our feelings of conscious awareness -of happiness, pain, love, aesthetic sensibility, will, understanding, etc.- fit into such a computational picture? […]

It is striking how carefully he chooses feelings of conscious awareness. He could have said feelings, consciousness, or awareness separately; instead he brings them together and then lists concrete sensations: feelings of conscious awareness of happiness, feelings of conscious awareness of pain, feelings of conscious awareness of will, sensibility, understanding, and so on.

Penrose is not satisfied with a purely functional perspective, the “behaves as if” of the Turing test. He is looking for lived experience, phenomenal experience. If we say that an AI can equal a human being, Penrose demands that it feel as humans feel: a feeling of conscious awareness.

And Hofstadter? Rereading GEB, I do not find a sharp position on feelings. Near the end, in “Intelligence and Emotions”, he tries to pull the two concepts apart. He opens with the scene of a child crying because his balloon has burst, and concludes:

It might be objected that, even if the program “understands” what is said, in an intellectual sense, it will never really understand it until it has cried and cried. And when is a computer ever going to do such a thing? This is the sort of humanistic issue that Joseph Weizenbaum is led to in his book Computer Power and Human Reason; for my part, I think it is an important issue: in fact, a truly profound one. Unfortunately, many AI researchers are at present unwilling to consider this problem seriously. Yet to some extent they are right, for it is a bit premature to concern oneself now with the crying of computers: first, it is necessary to deal with the rules that will allow computers to handle language and other things; in due course we shall confront questions of greater depth.

The emphasis is mine. I find it revealing: Hofstadter separates what is “intellectual”, the rules for dealing with language, from feelings. And that would include, in my opinion, the “feeling of being conscious” that Penrose is talking about.

GEB talks about symbols, meanings, and formal structures: the intellect. Hofstadter considers that to be the fundamental core of our mind. Perhaps that is why he was horrified when he realized that an AI had mastered this side of our behavior.

The plot twist of language models

In the last decade we have seen something surprising: language models (LLMs) trained exclusively on text, with no sensory or motor input, learn to manipulate syntactic structures and to handle semantic regularities of use: they maintain reference in a dialogue, follow complex instructions, summarize, translate, argue, program. All of that without ever having “touched” the world beyond what is implicit in written corpora.

This does not prove anything definitive about consciousness, but it does redraw the map: it shows that a large part of linguistic competence and textual reasoning can emerge from language itself. Much of what we associated with “linguistic intellect” can be learned from text alone.

That does not solve the riddle of feeling, but it does make one thing clearer: speaking, reasoning, and maintaining referential coherence do not by themselves imply having felt anything at all.

A new perspective

With this contemporary lens, I return to Hofstadter and Penrose to better understand what they were really arguing about, and why, perhaps, they were not so far apart.

From Hofstadter’s point of view, language models could be seen as confirmation that symbolic patterns and loops of reference are enough for reasoning. From Penrose’s point of view, they would confirm that mastery of language does not require lived experience.

Almost forty years after my first reading of GEB, rereading it with this perspective is very suggestive. Hofstadter does not address the feeling of being conscious; he talks about symbols and language. Penrose, by contrast, talks about the sensation of being conscious. Perhaps they were not really so opposed: they were arguing about ambiguous words. Each understood “mind” and “consciousness” differently.

In the next article I want to disambiguate the word “consciousness” with a playful typological game: type-1, type-2, and type-3 consciousness.

I will tell that story in a couple of weeks.

See you next time.

Episode description

Demis Hassabis, CEO of Google DeepMind, talks with host Logan Kilpatrick. In this episode, you get the arc from AI systems that played videogames to today’s thinking models, how projects such as Genie 3 are building world models to help AI understand reality, and why new proving grounds such as Kaggle’s Game Arena are needed to evaluate progress on the road to AGI, artificial general intelligence.

Highlighted clips

Genie 3 generates coherent and persistent world models by simulating environments:

But one great way is to just get it to reverse it and sort of generate something about the world. Like, you know, you turn on a tap and some liquid comes out of it, or there’s a mirror and can you see yourself in the mirror, all of these things. And that’s what Genie is sort of going towards is building that world model and then expressing it and actually be able to generate worlds that are consistent. And that’s the surprising thing about Genie 3 is that, you know, you look away, you come back, and that part of the world is the same as you left it link

Parallel planning and reasoning are key to reaching AGI through deep thinking:

And then you need some thinking or planning or reasoning capability on top. And this is obviously the way to get to, you know, AGI. And then, of course, once you have thinking, you can do deep thinking or extremely deep thinking and then sort of have parallel planning. You know, you can do sort of planning and thoughts in parallel and then collapse on onto the best one and then make a decision and then move on to the next one. link

Self-reflection:

you want to sort of go back and refine your own thought processes which is in effect what the thinking systems link

Jagged intelligence and the lack of consistency as a barrier to full AGI:

on the other hand, they can still make simple mistakes in high school maths or simple logic problems or simple games if they’re posed in a certain way. So that must mean there’s still something kind of missing. link

And in my opinion, this is one of the things that’s missing from these systems being full AGI is the consistency. link

He emphasizes the importance of Game Arena for building more complex and meaningful benchmarks:

I think there’s actually really amazing work to be done in creating benchmarks that are really meaningful, that test slightly more complicated or subtle things than the sort of Brute force school exam type things that we have today. And that’s why I’m so excited about Game Arena because, and it is going a little bit back to our roots, of course, which is why we came up with it. link

Each game is unique; they plan to expand the chess-based Game Arena into thousands of games:

each game is unique because it’s created by the two players. So there’s a kind of uniqueness about that. So that’s also nice for testing and then the final thing is just like we did with our own early games work as the systems get better and better you can introduce more and more complex games Into the game arena so we started with chess um for obvious reasons it’s the classic one we test ai on um it’s close to my heart of course but we the idea is we’re going to expand it to potentially thousands of games link

The integration of external tools into AI systems raises blurry boundaries between model and tool:

A lot of the thinking, the reason the thinking is part of the systems is very important is because you can use tools during the thinking, right? You can call search, you can, you know, use some math program, you can do some coding, come back, and then update your planning on what you’re going to do. So I think that’s still actually fairly nascent at the moment, but I think that’s going to be incredibly powerful once that becomes really reliable and we work out, and the systems become Good enough, they can use pretty sophisticated tools very reliably. And then the interesting thing comes is, what do you leave as a tool versus put into the main system, the main brain, so to speak? Now, with humans, it’s easy because we’re physically constrained. So anything that’s not in our body is a tool, right? So there’s no question about what’s a tool, what’s our brain. But with a digital system, you can actually kind of, those things can get blurred. So should it be in the main model, the capability, for example, to play chess or something? Or do you just use Stockfish or AlphaZero as a tool? And that tool could also be an AI system. It doesn’t have to be a piece of software. It could actually be something like AlphaFold or whatever. link

Product design requires anticipating technological advances and allowing frequent modular replacement:

the hard part and we’ve talked about this before is in this new world is you’ve got I think it requires very interesting skills from a product manager or product designer type of you Know role because you’ve got to sort of design say your product’s coming out in a year you’ve got to be really close and understand the technology well to kind of intercept where that Technology will be in a year’s time and design for that right and um and i think uh i’ve also whatever polish product polish you put on top uh of your product it has to allow for the engine Under the hood to be unplugged and plugged back in with a more advanced uh system you know, that’s coming out every three to six months link

The post-AGI dream:

Logan Kilpatrick: I feel like Genie 3 is a good excuse for us to have a chance to make games and play them and then DeepMind’s a video game.

Demis Hassabis Well, you know, that’s always my secret plan is maybe like post-AGI, once that’s done safely over the line, you know, go back with these tools and make the greatest game ever. That would be a real dream come true. link

Original illustration published in Joanne Jang’s post, showing the perception of consciousness in AI models, the emotional bonds they may awaken in people, and the ambiguity these ties can generate.

The whirlwind of AI news, papers, and blog posts has buried a very important article, really more of a statement, by Joanne Jang, director of model behavior & policy at OpenAI, one that I do not think has received enough attention. It is her Substack post Some thoughts on human-AI relationships, in which she lays out ideas about how OpenAI’s models should interact with users: in a way that makes us perceive them as assistants with warmth, friendly companions we enjoy interacting with, but without a selfhood or a consciousness that would make us perceive them as humans with whom we might form some kind of emotional relationship.

In Jang’s own words, OpenAI should aim to design “for warmth without selfhood”. That simple phrase gives us clues about very deep things that are already under way inside OpenAI, and inside the companies building other language models as well, things that are going to shape our interaction with this technology over the coming years.

Jang’s article contains three fundamental ideas:

1. Today it would already be possible to train a model, using for example the same reinforcement-learning techniques used to build reasoning models such as o1 or o3, so that it could give the appearance of a conscious human being capable of passing the Turing Test with ease.
2. The underlying problem of consciousness, the ontological problem of what qualia are or what the sensation of perceiving something really is, is not something OpenAI is interested in.
3. OpenAI’s main goal is to build a model that feels “close” without being “perceived” as conscious. To create a personal assistant that is rewarding to interact with, but one we cannot fall in love with.

OpenAI wants to build its personal assistant on these three ideas. They matter because they tell us a great deal about how the company wants to persuade hundreds of millions of people to install its intelligent assistant on their computers and phones, and to buy some of the future secret devices Sam Altman and Jony Ive have promised us.

https://youtu.be/W09bIpc_3ms?si=nonYgombMA5hCSYv

It seems increasingly clear to me that OpenAI wants to become the next Apple and derive a large share of its revenue from end users who find ChatGPT useful. And in order to get there, the bet, as Jang says, will be to make ChatGPT increasingly personal, but without allowing it to be confusable with a person.

Warning: everything written from this point onward was written by GPT-4.5, which I asked to develop the points above on the basis of Jang’s original article.¹

Can we simulate consciousness?

One of the most striking claims in Joanne Jang’s article is that, technically speaking, it is already possible to train language models capable of passing the Turing Test with ease. Jang explicitly points out that, with current reinforcement-learning techniques, it would be feasible to build a model whose interaction felt so natural that any person could mistake it for a human interlocutor. This raises an unsettling question: if it is so easy to simulate consciousness, how do we define what consciousness is and what it is not?

Jang writes: “A model deliberately shaped to seem conscious could ace pretty much any consciousness test.” That is where a crucial ethical debate emerges. Even if it is possible to simulate conscious behavior, should we do it? OpenAI has decided not to go down that path. They prefer models that feel warm and accessible, but without pretending to have a fictitious inner life that could emotionally confuse users.

This is crucial, because if this indistinguishable simulation of human consciousness becomes widespread, we run the risk of creating emotional bonds that, although fictional, could become as intense as the ones we establish with other people. The situation is comparable to the way today’s social networks have transformed our social dynamics, generating emotional dependency through digital interaction.

OpenAI and the debate on consciousness

Another fundamental idea Jang puts forward is that OpenAI does not intend to solve the ontological problem of consciousness. In her words, this is a terrain that escapes what is scientifically testable, because there is still no universal and falsifiable test that clearly defines what it means to be conscious.

Jang proposes drawing a clear distinction between two axes: ontological consciousness, whether a model is really conscious in a fundamental sense, and perceived consciousness, how conscious it appears to its users. OpenAI focuses only on perceived consciousness, which is the one that truly affects human experience.

This stance is reasonable, although it also entails an obvious risk. By refusing to go deeper into the ontological questions, we leave an ethical and philosophical vacuum regarding how we ought to treat these artificial intelligences once the perception of consciousness becomes generalized. Without clear answers about the fundamental nature of the models, we may end up facing ethical dilemmas similar to those we have already seen with other disruptive technologies.

Designing models that feel “close” but not human

OpenAI’s explicit strategy, as Jang explains it, is to design models that are warm, pleasant, and approachable, but without encouraging the formation of deep emotional bonds. In her own words, the goal is to achieve interaction “without implying an inner life.” This is a delicate balance: the models have to be pleasant enough to produce satisfaction in the user, but not so “human” that emotional dependency starts to develop.

This approach has clear advantages. It makes it possible to take advantage of the positive potential of the models without falling into the dangerous trap of emotional dependency, a lesson we have learned the hard way from the dependency generated by social networks through mechanisms such as infinite scroll and constant notifications.

However, there is also a significant risk: no matter what limits OpenAI tries to impose, users may still perceive these models as something more than simple tools, especially if they become omnipresent in our daily lives. It is essential that this line be handled with a great deal of responsibility, transparency, and control.

Conclusion

The advances in AI envisioned by OpenAI through Joanne Jang’s framework could represent a revolutionary and positive step forward, opening the door to personal assistants that are genuinely useful, efficient, and pleasant. At the same time, just as happened with earlier technologies developed by the big tech companies, the danger of creating excessive emotional dependency is very real.

The future of our interaction with language models will depend on maintaining a delicate balance between making the most of their benefits and preserving a clear boundary that avoids emotional confusion. The challenge is not only technological but deeply ethical and social. Time will tell whether we are capable of learning from previous mistakes and using artificial intelligence to improve our lives without becoming trapped in new forms of dependency.

Third older post recovered, originally published in July 2022.

I have added an addendum at the end, generated by GPT o3, commenting on the changes introduced in Swift over the last three years that affect the material discussed in the article.

While exploring SwiftUI and using it to understand Swift better, one of the first things that catches the eye is the reserved word some:

struct LandmarkRow: View {
    var landmark: Landmark

    var body: some View {
        HStack {
            landmark.image
                .resizable()
                .frame(width: 50, height: 50)
            Text(landmark.name)
            Spacer()
        }
    }
}

What does it mean, in the code above, that the variable body contains some view?

I do not know whether it happens to you too, but whenever I try to understand something new, I always feel as if I am following the clues of a case to be solved, as if I were a kind of Sherlock Holmes or Hercules Poirot. One question leads to another, and then to another, until in the end I manage to untangle the thread, or part of it, and connect all the new concepts I am finding with those I already know. And afterwards, when you explain something, you have to follow the path in reverse. You start from what you already know and from there build and explain the new material.

In our case, the path to understanding some is going to begin with a curious question: in a strongly typed language like Swift, is it possible to define an array containing values of different types?

At first sight, that seems contradictory. If we have to specify the type of the array strictly, then we must also specify the type of its components:

var miArray: [Int] = []

The type of the array above is [Int]. That means all its elements must be of type Int. We could define other arrays whose elements are of type String or Double, but in every case the arrays would be homogeneous and all their elements would have the same type.

Is that always how it works in Swift? It seems too rigid. It might be that, to solve a certain problem, the simplest solution would be to store integers, strings, and floating-point numbers together in a single array.

If we were designing a new language, we might be tempted to define something like:

var arrayMisc: [Int | String | Double] = [1, "Hola", 2.0, 3.0, 2]

That is, arrayMisc would be heterogeneous and could contain elements that are Int or String or Double.

It sounds interesting to be able to express something like that. But it must not be such a good idea, because I do not know any language that has a construct of this kind. For example, we would immediately run into the problem of how to treat the elements of the array. What happens when we use a loop and traverse its elements?

for thing in arrayMisc {
   // process the array element
}

What type would the variable thing have? It could be an Int, a String, or a Double, depending on the array element it happened to be holding at that point. We would need to introduce some language construct to let us work with the elements of that heterogeneous array.

Luckily, we are not designing a new language: we are studying Swift. We are going to see that Swift is a modern and flexible language that provides several strategies that allow us, up to a point, to group different kinds of data inside a single array.

Weakly typed languages

In weakly typed languages such as Python, it is very easy to define an array with different kinds of data:

miArray = [1, "hola", 3.0]
print(miArray)

# prints: [1, 'hola', 3.0]

This prints:

[1, 'hola', 3.0]

Because Python is weakly typed, it has no problem doing things like:

print(miArray[0] + miArray[2])

# prints: 4.0

This may seem like an advantage until we realize that the compiler is not really checking anything and allows expressions such as the following, which will produce a runtime error because an integer and a string cannot be added:

print(miArray[0] + miArray[1])

# runtime error

That is the problem with weakly typed languages. The compiler cannot detect many errors, so those errors appear at runtime.

Strongly typed languages

In a strongly typed language, every variable, parameter, return value of a function, and so on must have a perfectly specified type. This has many advantages: the compiler warns us about errors when we compile the program, the IDE gives us hints while we write it, and the resulting code is more readable and easier to understand.

However, the fact that everything has to have a predetermined type sometimes removes a lot of flexibility, forces us to write excessively rigid and repetitive code, and sometimes prevents us from doing things that would make our program much simpler. For example, the idea we are exploring in this article: storing instances of different types in an array.

Designers of modern programming languages such as Swift have realized that being too rigid is not a good idea, and they have devised strategies that make the type system more flexible. For example, polymorphism, function overloading, or generics. These strategies obviously make languages more complex, both in terms of learning them and in the internal workings of the compiler. But in the end they are appreciated by developers because they allow the code to be more expressive and simpler.

We can view the problem we are discussing in this article as a concrete example of that trade-off, that search for flexibility inside a strongly typed language.

Let us now explain the different ways Swift provides for solving the main question we are asking.

The special type Any

The special type Any allows a variable to be of any type. For example, we can declare a variable with an integer and then assign a string to it:

var x: Any = 10
x = "Hola"

Although this may look equivalent to the way weakly typed languages behave, the Swift compiler is still doing its job. We cannot do much with a variable of type Any. For example, the following code produces a compilation error:

let x: Any = 10
let y: Any = 5

print(x+y)

// Error: binary operator '+' cannot be applied to two 'Any' operands

We could do the addition by downcasting:

let x: Any = 10
let y: Any = 5

print((x as! Int) + (y as! Int))

// Prints: 15

The as! operator returns the value with the indicated type. If the variable is not compatible with that type, a runtime error occurs.

Arrays of Any

So a first way to allow arrays with multiple types is to use the special type Any.

var miArray: [Any] = [1, "Hola", 3.0]

This array is similar to the Python array. The advantage is that, as we saw earlier, the Swift compiler will not let us directly operate on its values:

print(miArray[0] + miArray[1])

// error: binary operator '+' cannot be applied to two 'Any' operands

We can, however, use downcasting to process the elements of the array. We can use a switch to determine the type of each element:

for thing in miArray {
    switch thing {
    case let algunInt as Int:
        print("Un entero con valor de \(algunInt)")
    case let algunDouble as Double:
        print("Un double con valor de \(algunDouble)")
    case let algunString as String:
        print("Una cadena con valor de \"\(algunString)\"")
    default:
        print("Alguna otra cosa")
    }
}

It prints:

Un entero con valor de 1
Una cadena con valor de "Hola"
Un double con valor de 3.0

It seems that we now have a strategy that solves our problem. What is the downside? Precisely the need to use downcasting and the excessive freedom it gives us. Downcasting makes the code somewhat more confusing. And the ability to store anything in the array makes the code more error-prone. Developers may be tempted to use the as! operator, making the code less robust and more likely to break at runtime.

Arrays built from enums with associated values

Could we limit the types included in the array to a specific set? Suppose, for example, that I only need my array to contain integers, strings, and floating-point numbers. Is there any Swift feature that allows that?

There is. One way to do it is through enum types. In Swift, enum types are very powerful. It is possible to associate tuples of values with concrete instances of the type. We can, for example, define a type that can be an integer, a string, or a real number, and associate a value of that type with each enum case:

enum Miscelanea {
    case entero(Int)
    case cadena(String)
    case real(Double)
}

And we can create an array of instances of that type:

var miArray: [Miscelanea] = [.entero(1), .cadena("Hola"), .real(2.0)]

To traverse the array, we will need to use a switch again:

for thing in miArray {
    switch thing {
        case let .entero(algunInt):
            print(algunInt)
        case let .cadena(algunaCadena):
            print(algunaCadena)
        case let .real(algunDouble):
            print(algunDouble)
    }
}

This prints the same as before:

1
Hola
2.0

The advantage now is that the code is completely safe. We cannot add anything to the array that is not one of the enum variants, and the language correctly controls all the possible options we can have inside the array.

But this solution also has some problems. First of all, it is excessively rigid. What happens if in the future we want to expand the types included in the array? For example, to add booleans. We could not do it additively; we could not extend the code’s functionality just by adding new pieces. We would have to rewrite the Miscelanea type to include the new case and recompile the application.

The second problem is that this solution does not let us include instances of structures or classes in the array. Suppose we are designing an application for geometric figures and want to store a collection with different kinds of shapes: rectangles, squares, triangles, and so on. We would not be able to do it this way.

That brings us to the next solution.

Arrays of a protocol type

Another solution, a more flexible one, for storing different types in an array is to use a protocol, or a superclass.

In general, if we want to group several items into a collection, it is because they all share some property. We can specify that property in a protocol and make all the types we store in the array conform to that protocol.

In the example of the array of geometric shapes, we would need to look for some property shared by all those shapes and define a Figura protocol with that property or properties. The concrete types Rectangulo, Cuadrado, Triangulo, and so on would then need to conform to the Figura protocol. And then we could declare an array of Figuras.

Let us look at a simple example. Suppose that all the items we store in the array are items that have a name, a String. We can define a protocol with that property:

protocol Nombrable {
    var nombre: String {get}
}

Once we have created this protocol, we can make the types we add to the array fulfill that property.

Instead of creating new types for the example, Swift lets us extend the existing types Int, String, and Double with the nombre property and make them conform to the Nombrable protocol:

extension Int: Nombrable {
    var nombre: String {String(self)}
}

extension String: Nombrable {
    var nombre: String {self}
}

extension Double: Nombrable {
    var nombre: String {String(self)}
}

And now we can create the array of nameable things and add instances of the previous types to it:

var miArray: [Nombrable] = [1, "Hola", 2.0]

for thing in miArray {
    print(thing.nombre)
}

This prints:

1
Hola
2.0

This solution of using a protocol or a superclass to define the array is the most flexible and the most widely used. It is more advisable to use a protocol, because both structures and classes can conform to it. If we define a superclass, we could only use it with classes, since in Swift structures do not support inheritance.

Unlike enums, if in the future we want to expand the array to include new types, all we would have to do is make those new types conform to the protocol on which the array is defined.

For example, we could include booleans in our array:

extension Bool: Nombrable {
    var nombre: String {
        self ? "true" : "false"
    }
}

var miArray: [Nombrable] = [1, "Hola", 2.0, false]

for thing in miArray {
    print(thing.nombre)
}

This prints:

1
Hola
2.0
false

The problem with generics

The previous solution, defining a protocol for the array’s components, seems like the perfect solution. It provides flexibility and extensibility. Although we did not see it in the example, it also allows downcasting and obtaining instances of the concrete type of the data by using a switch statement.

But there is one aspect we have not considered. One of the most important features of Swift is its commitment to generic types. Since the beginning of the language, there has been a roadmap, in the form of a manifesto, that has been implemented gradually in each new version of the language.

In the case of protocols, we can make some element of the protocol generic by using an associated type. In fact, in SwiftUI a view is a generic protocol with an associated type, see the reference for the View protocol in Apple’s documentation.

What happens, then, if as the array type we use a generic protocol, a protocol that has an associated type? What happens if we create an array of SwiftUI views? It turns out that things become a little more complicated and the compiler produces an error.

var array: [View] = []

// Error: Protocol 'View' can only be used as a generic constraint
// because it has Self or associated type requirements

What is going on? Better to leave that for another post, since this one has already grown too long.

References

• Arrays in the Language Guide
• The Any type in the Language Reference
• Downcasting in the Language Guide
• Enum types in the Language Guide
• Protocols in the Language Guide
• Generic types in the Language Guide
• Generic manifesto
• Associated type in the Language Guide
• Code file with the examples from the post

Addendum (April 2025) - What has happened in Swift over these three years?

1. The new any prefix for existential types

Updated example:

var miArray: [any Nombrable] = [1, "Hola", 2.0]

The compiler still accepts the old syntax, but it emits the warning: “Implicit use of ‘Any’ for existential types is deprecated.”

2. some now works in more places

Since Swift 5.7, opaque types can also be used in:

• Function parameters

  func wrap(_ builder: () -> some View) -> some View { ... }
  

• Stored properties with an initial value

  let cache: some Hashable = Set<Int>()
  

3. Arrays of View: still not allowed, use AnyView

[View] or [any View] still do not compile because View has an associatedtype Body.

// 'View' has Self or associated type requirements
var vistas: [any View] = []

Official pattern, the type eraser:

var vistas: [AnyView] = [
    AnyView(Text("Hola")),
    AnyView(Image(systemName: "star"))
]

4. The special type Any does not change

Any does not take the any prefix. All the examples using Any remain valid.

5. Changes that do not affect this article

• buildPartialBlock and parameter packs affect result builders, not heterogeneous arrays.
• Strict Concurrency only affects you if you mix async with existentials, which you are not doing here.

6. References for further reading

• SE-0335 - Introduce existential any
• The Swift Programming Language -> Macros -> Existential Types
• WWDC22 - “Embrace type abstraction with opaque types”

I am bringing back an older post, originally published three years ago, in July 2022. I have added two more: the second part of this one and an explanation of how arrays with different data types can be defined in Swift.

I have added an addendum at the end, generated by GPT o3, commenting on the changes introduced in Swift over the last three years that affect the topics discussed in the article.

Since Apple introduced SwiftUI at WWDC19, I have wanted to understand the Swift features on which this technology is built. I read a few posts that touched on the subject, and I came away with the idea that in Swift 5.1 they had introduced something called function builders, which was the feature that made it possible to build SwiftUI views declaratively, but I did not continue studying the subject any further.

One strange thing about function builders was that they were an undocumented Swift feature that had not gone through the usual language evolution process, in which proposals for new features are eventually approved or rejected after open discussion with the community.

It did not take long for a proposal and a pitch to appear on the community forums. The discussions went on, different alternatives were considered, the feature was renamed result builders, and in the end, almost two years later, it was accepted in October 2020 and published in the language in version 5.4, released in April 2021.

More than a year later, I have finally sat down to study result builders and try to understand how they work. After spending a few days reading documentation, creating some notes in Obsidian, and experimenting with Swift code, it is time to try to put everything in order and write a post on the topic.

Purpose of result builders

Let us begin by explaining the purpose of result builders, and then we will explain how they work.

An example with SwiftUI

If we look at a simple example of SwiftUI code, we will see that we can identify it as Swift code, but that there is something about it that does not quite fit. For example, the following code constructs a view in which an image and a piece of text are stacked vertically.

import SwiftUI

struct ContentView: View {
    var body: some View {
        VStack {
            Image(systemName: "globe")
                .imageScale(.large)
                .foregroundColor(.accentColor)
            Text("Hello, world!")
        }
    }
}

The result is the following:

The code defines a struct named ContentView that conforms to the View protocol. This protocol requires a body property to be defined, which must also conform to View, thus recursively constructing a tree of views that SwiftUI is responsible for rendering.

The body property is a computed property of type some View that returns a VStack. We will leave the use of some for another post and focus on the construction of the VStack:

VStack {
    Image(systemName: "globe")
        .imageScale(.large)
        .foregroundColor(.accentColor)
    Text("Hello, world!")
}

The braces after VStack define a trailing closure that is passed to the initializer. It is equivalent to:

VStack(content: {
    Image(systemName: "globe")
        .imageScale(.large)
        .foregroundColor(.accentColor)
    Text("Hello, world!")
})

If we look closely at the code inside the closure, we will see that something is odd. There are two statements that construct an Image instance and a Text instance. They are precisely the image and the text that are stacked and shown in the resulting view. But nothing is done with those instances. How are they passed to VStack? Where is the closure’s return? What kind of magic is this?

The explanation is that SwiftUI defines a result builder that performs a compile-time transformation of the code above, code that is not valid Swift on its own, into code similar to the following:

VStack {
    let v0 = Image(systemName: "globe")
                 .imageScale(.large)
                 .foregroundColor(.accentColor)
    let v1 = Text("Hello, world!")
    return ViewBuilder.buildBlock(v0, v1)
}

This code is valid Swift. The instances created by Image and Text are stored in two auxiliary variables, and a static function, ViewBuilder.buildBlock, is called. It receives those two views and combines them into a structure, a pair, that is also of type View and is returned.

Although we have not seen it in the example, it would also be possible to construct the constituent elements recursively using the same DSL. For example, one of the elements passed to the VStack could itself be another VStack formed by combining other basic elements.

Creating DSLs

Using the result builder above, we can transform the clean and clear code at the beginning, which does not work in Swift, into compilable code. The result builder adds everything needed, temporary variables, the call to the construction function, and so on, so that the resulting code is correct for the compiler. And it does so completely transparently. The developer does not see any of the second form of the code, only the first, the clean and clear version.

The code transformed by the result builder is what is known as a DSL, a Domain Specific Language. In this case, the DSL lets us construct SwiftUI views by describing and combining their constituent elements.

Result builders have not only been used to build SwiftUI. The community has created a large collection of DSLs for defining all kinds of things, such as HTML, CSS, graphs, REST functions, or tests. Even at the recent WWDC22, a DSL for building regular expressions in Swift was introduced, SwiftRegex.

In short, much like macros in programming languages such as LISP, or the define directives in C, result builders allow us to specify transformations that will be applied to the source code at compile time. Let us now see how that functionality has been incorporated into Swift.

First example

First of all, to define a result builder we need to specify a buildBlock function that constructs a result from a number of elements. In the case of the previous example, it needs to construct a composition of two views from the individual views, the Image and Text instances.

How can we define this function? The simplest way is to define a static function, one that can be called without needing to create an instance. This function must be named buildBlock, and it must take the individual components as parameters and return a new component resulting from their composition. We can define it in a structure, a class, or an enum annotated with the @resultBuilder attribute.

A very simple example that works with strings is the following:

@resultBuilder
struct StringConcatenator {
    static func buildBlock(_ component1: String, _ component2: String) -> String {
        return component1 + ", " + component2
    }
}

The buildBlock function takes two strings and returns their concatenation, separated by a comma. We define it as a static function of the StringConcatenator structure. The @resultBuilder attribute indicates that this type is a result builder and that we will be able to specify a DSL with it.

How can we now indicate that we want to use this result builder? The Swift engineers came up with a brilliant idea. When the StringConcatenator type is defined as a result builder, the compiler creates the @StringConcatenator attribute, which we can use wherever we want to apply it.

For example, we can write the following code:

@StringConcatenator
func holaMundo() -> String {
    "Hola"
    "mundo"
}

print(holaMundo())

The holaMundo() function would not be valid Swift because it has no return with the string it is supposed to return. In addition, its two statements do nothing except define the strings "Hola" and "mundo". But if we run the code above, we will see that the compiler produces no error and that the code runs correctly and prints the typical message:

Hola, mundo

What is going on? By using the @StringConcatenator attribute on the holaMundo() function, we are declaring that it is a function whose body we are defining with a DSL that will be processed by the StringConcatenator result builder.

As in the previous SwiftUI example, each statement in the function body specifies a component that the compiler must process. In this case, they are strings. And at the end, buildBlock must be called to combine those components and return the resulting string. Specifically, the code produced by the transformation is the following:

func holaMundo() -> String {
    let v0 = "Hola"
    let v1 = "mundo"
    return StringConcatenator.buildBlock(v0, v1)
}

This transformed code is what is actually executed in the program and what returns the string "Hola, mundo".

Variable number of arguments

In the previous example, the buildBlock function is defined only for two arguments. It would not work if we wanted to construct a string with more than two components. We can improve it by using Swift’s ability to define functions with a variable number of arguments:

@resultBuilder
struct StringConcatenator {
    static func buildBlock(_ components: String...) -> String {
        return components.joined(separator: ", ")
    }
}

Now the buildBlock function receives a variable number of strings stored in the components array. And the higher-order function joined traverses the array of strings and joins them all with a comma and a space.

With this buildBlock, we can compose as many strings as we want in the DSL. For example, we can define a greeting from four strings:

@StringConcatenator
func saludo(nombre: String) -> String {
    "Hola"
    "me"
    "llamo"
    nombre
}

In this example, we have also added a nombre parameter to the function. This parameter lets us specify the name of the person greeting us.

The @StringConcatenator result builder transforms the code above into:

func saludo(nombre: String) -> String {
    let v0 = "Hola"
    let v1 = "me"
    let v2 = "llamo"
    let v3 = nombre
    return StringConcatenator.buildBlock(v0, v1, v2, v3)
}

If we call the original function

print(saludo(nombre: "Frodo"))

it will print the following:

Hola, me, llamo, Frodo

DSL in computed variables

According to the official Swift documentation, we can use a result builder attribute in the following places:

• In a function declaration, and the result builder constructs the body of the function.
• In a variable declaration that includes a getter, and the result builder constructs the body of the getter.
• In a closure parameter of a function declaration, and the result builder constructs the body of the closure passed to the corresponding argument.

We saw the first case in the previous section. Let us look at an example of the second case.

For example, we can define the following structure:

struct Persona {
    let nombre: String

    @StringConcatenator
    var saludo: String {
        "Hola"
        "me"
        "llamo"
        nombre
    }
}

let frodo = Persona(nombre: "Frodo")
print(frodo.saludo)

Now the DSL is used to define the getter of the computed variable saludo. The result builder transforms that getter in the same way as in the previous examples, creating a getter that returns a string from the strings that appear in the different statements of the original code.

The let instruction creates an instance of Persona, initializing its name. And the next statement calls the computed variable, which returns the greeting string and prints it:

Hola, me, llamo, Frodo

DSL in parameters

The specification of how to use the result builder attribute mentions, lastly, the possibility of using it on a closure parameter. Let us look at an example:

func imprimeSaludo(@StringConcatenator _ contenido: () -> String) {
    let resultado = contenido()
    print(resultado)
}

We are defining a function that will receive a parameterless closure returning a string. The body of the function executes the closure and prints the result. The @StringConcatenator annotation establishes that we will be able to pass DSL closures as arguments and that those closures will be transformed by the result builder.

In this way, we can call the previous function by using a closure in which we define the strings that will appear in the greeting. And we can do so without using the @StringConcatenator attribute explicitly, because it has already been defined on the function parameter:

imprimeSaludo {
    "Hola"
    "mundo"
}

The code above prints:

Hola, mundo

Let us look in more detail at how the example works. The imprimeSaludo function receives the contenido closure as a parameter. It is a parameterless closure that returns a string. And it is preceded by the @StringConcatenator attribute. This causes any argument that is passed, a closure returning a string, to be transformed by the result builder.

In the call to the function, we can see the Swift trailing-closure feature being used, which allows the parentheses to be omitted when the last argument is a closure.

The final code generated by the compiler is the following:

imprimeSaludo({
    let v0 = "Hola"
    let v1 = "mundo"
    return StringConcatenator.buildBlock(v0, v1)
})

Obviously, this code is much less clear and direct than the previous one:

imprimeSaludo {
    "Hola"
    "mundo"
}

Advanced DSLs

In the examples above, we have seen how a DSL can be used to build a component from elementary components. But we have only seen a small part of everything that result builders make possible.

If we look at an advanced SwiftUI example, we will see that the result builder defined in SwiftUI, the ViewBuilder structure, allows a much more advanced DSL, one in which we can use loops, ForEach, and conditionals, if.

Example from the Hacking with Swift article List Items Inside if Statements:

struct TestView: View {
    ...
    var body: some View {
        List {
            Button("Add a fresh potato") {
                self.basket.vegetables.append(Vegetable(name: "Potato", freshness: 1))
            }.foregroundColor(.blue)

            Section(header: Text(sectionHeadings[0])) {
                ForEach(self.basket.vegetables) { vegetable in
                    if vegetable.freshness == 0 {
                        Text(vegetable.name)
                    }
                }
            }

            Section(header: Text(sectionHeadings[1])) {
                ForEach(self.basket.vegetables) { vegetable in
                    if vegetable.freshness == 1 {
                        Text(vegetable.name)
                    }
                }
            }
        }
    }
}

In future posts we will continue exploring how result builders work and how to use them to construct this kind of powerful DSL.

References

• Proposal in Swift Evolution
• Introduction in the Swift Guide
• Detailed explanation in the Language Reference
• Code file with the examples from the post

Addendum (April 2025) - What has happened in Swift over these three years?

TL;DR
The basic ideas in the post are still correct, but Swift has removed several result builder limitations and has incorporated new, powerful macros that are worth knowing about. This appendix summarizes the relevant changes, Swift 5.7 through 5.10, while keeping the original post’s explanatory tone.

1. The end of the “limit of 10” thanks to parameter packs

In 2021, result builders internally generated a tuple of up to ten generics, hence the limitation mentioned in the post. Since Swift 5.9, the compiler understands variadic generics, proposal SE-0390, and the standard library has rewritten ViewBuilder like this:

@resultBuilder
public enum ViewBuilder {
    public static func buildBlock<each Content>(
        _ components: repeat each Content
    ) -> TupleView<(repeat each Content)> where repeat each Content: View
}

Parameter packs, <each T> / repeat each T, delegate the arity to the compiler, so the SwiftUI DSL, and any builder that adopts that pattern, now accepts as many elements as you want, without manual overloads.

How to adapt it to your builders
Replace your old

static func buildBlock(_ parts: String...) -> String

with the modern variant:

static func buildBlock<each S>(_ parts: repeat each S) -> String
    where repeat each S == String

2. The new family of macros enters the scene

Swift 5.9 introduced compiler macros, SE-0389 and SE-0397. Although in the post we compared result builders with LISP/C macros, Swift’s native macros play in a different league:

When to choose which

• Use result builders for purely declarative DSLs, SwiftUI, RegexBuilder, and the like.
• Choose macros for API generation, compile-time validations, or attributes such as @Observable.

3. SwiftRegex is now part of the language

What was presented at WWDC22 as “SwiftRegex” became part of the standard syntax starting with Swift 5.7. Today you can write:

let fecha  = "27/04/2025"
let patron = Regex(#"\d{2}/\d{2}/\d{4}"#)

if fecha ~= patron {
    // ...
}

The underlying builder uses regular-expression components rather than a classic result builder, but your explanation of declarative DSLs remains fully valid.

4. Strict concurrency and asynchronous builders

Since Swift 5.10, Strict Concurrency mode is active by default. If your builder generates async code:

@MyBuilder
func vista() async -> some View {
   // ...
}

mark the buildBlock overloads with the appropriate async/throws, or the compiler will show warnings.

5. Other syntax details

• Partial inferences: you can declare let saludo: _ = ... and let the builder resolve the type.
• buildPartialBlock: it allows buildBlock, buildEither, and related functions to be optional; the compiler synthesizes them if they are missing.
• The builder attribute can now also be applied to initializers, which is very useful for creating complex objects declaratively.

To go deeper

• Proposal SE-0390 - Variadic Generics
• SE-0389 / SE-0397 - Swift Macros
• The Swift Programming Language -> Macros -> Result Builders
• WWDC23 video “Expand Swift macros”, which shows macros and builders working together

Second older post recovered, originally published in July 2022.

I have added an addendum at the end, generated by GPT o3, commenting on the changes introduced in Swift over the last three years that affect the material discussed in the article.

In the previous post in the series on result builders, we saw how they allow us to use a DSL to define a closure or block of code that builds a component from elementary components.

We saw the simple example of a string builder:

@resultBuilder
struct StringConcatenator {
    static func buildBlock(_ components: String...) -> String {
        return components.joined(separator: ", ")
    }
}

The code above creates the @StringConcatenator annotation that we can use to apply the result builder. For example, we can apply it to a function definition:

@StringConcatenator
func holaMundo() -> String {
    "Hola"
    "mundo"
}

print(holaMundo())
// Prints: Hola, mundo

The function above constructs a string by joining the elementary strings defined in its body. Let us remember that the result builder transforms this body at compile time, turning it into something like this:

func holaMundo() -> String {
    let v0 = "Hola"
    let v1 = "mundo"
    return StringConcatenator.buildBlock(v0, v1)
}

Finally, we ended by explaining that if we annotated a function parameter with the attribute, the result builder would be applied to the closure passed as that parameter. This is interesting because it allows us to use the result builder without the annotation appearing explicitly:

func imprimeSaludo(@StringConcatenator _ contenido: () -> String) {
    print(contenido())
}

// We call the function with a closure that uses the DSL.
// There is no need to add the @StringConcatenator annotation.
imprimeSaludo {
    "Hola"
    "mundo"
}
// Prints: Hola, mundo

In this second post we are going to look at other places where the result builder attribute can be used and other transformations that can be performed.

Result builders in initializers

In SwiftUI, the result builder ViewBuilder is used to construct views. An example is the following:

let vista =
    HStack {
        ForEach(
            1...5,
            id: \.self
        ) {
            Text("Ítem \($0)")
        }
    }

The constructed view is a horizontal stack with five subviews of type Text:

We can see that HStack receives a closure with DSL code that specifies the subviews. ViewBuilder will transform that DSL into Swift code.

Why do we not have to use the @ViewBuilder attribute? The explanation is that this attribute has been used on a function parameter. More specifically, on a parameter of HStack’s initializer.

Let us do something similar with StringConcatenator.

Example of a result builder in an initializer

Suppose we have the following Persona structure:

struct Persona {
    let contenido: () -> String

    var saludo: String {
        contenido()
    }

    init(@StringConcatenator contenido: @escaping () -> String) {
        self.contenido = contenido
    }
}

We are defining a structure with a stored property, contenido, that contains a parameterless closure returning a string. And a computed variable, saludo, that returns the string obtained by executing that closure.

We also define the Persona initializer with the parameter that initializes the contenido property. To construct an instance of Persona we must pass as an argument the closure that will generate the greeting. And we add the @StringConcatenator attribute to that parameter to indicate that the argument we pass must be transformed by the result builder. The @escaping attribute is not important here; it has to do with how the closure’s scope is managed, and the compiler produces an error if we do not include it.

Now we can create an instance of Persona by passing a closure that uses the DSL:

let frodo = Persona {
    "Hola"
    "me"
    "llamo"
    "Frodo"
}

Once the instance has been constructed, the closure that returns the greeting will have been stored in its contenido property. We call the closure by accessing the saludo property:

print(frodo.saludo)

This prints:

Hola, me, llamo, Frodo

Simplifying the initializer

The engineers who designed result builders came up with a piece of syntactic sugar that makes the construction above even simpler.

Since Swift structures automatically generate a memberwise initializer, we could use the result builder attribute directly on the property. We do not need to define the initializer because Swift creates it automatically:

struct PersonaSimple {
    @StringConcatenator let contenido: () -> String

    var saludo: String {
        contenido()
    }
}

There is no need to specify anything else. Swift automatically generates the structure’s initializer correctly, and we can use it in the same way as before:

let frodo2 = PersonaSimple {
    "Hola"
    "me"
    "llamo"
    "Frodo"
}

print(frodo2.saludo)
// Prints: Hola, me, llamo, Frodo

This way of defining a result builder is one of the most common. It is used in the vast majority of DSLs built in Swift, including SwiftUI.

Result builders in protocols

Another way to apply a result builder without explicitly using its annotation is through a protocol. If we mark a method or property in a protocol with the annotation, the result builder will be applied in the code that adopts the protocol.

Let us continue with the example of the greeting built with @StringConcatenator. We can define a protocol with a property for the greeting:

protocol Educado {
    @StringConcatenator var saludo: String {get}
}

When the property is defined this way, any type that adopts the Educado protocol will have to define a saludo property in which the result builder can be used. For example, we can define the PersonaEducada structure like this:

struct PersonaEducada: Educado {
    var nombre: String
    var saludo: String {
        "Hola"
        "me"
        "llamo"
        nombre
    }
}

We are defining saludo with the strings shown in the different statements ("Hola", "me", "llamo") and the nombre property. The @StringConcatenator result builder will transform this code in the way we saw earlier.

Since saludo is a computed variable, the only stored variable that needs to be specified when creating the structure is the person’s nombre. We do so in the following way, calling the automatically created memberwise initializer:

let gandalf = PersonaEducada(nombre: "Gandalf")

And once the instance of PersonaEducada has been created, we can ask for its greeting:

print(gandalf.saludo)

As always, it will print:

Hola, me, llamo, Gandalf

More elaborate transformations

So far we have seen how the result builder constructs a complex component from elementary components by using the static function buildBlock.

The signature of this function is as follows:

static func buildBlock(_ components: Component...) -> Component

In the case of the examples above, the component type is String and the buildBlock function receives a variable number of strings and constructs the resulting string.

However, in some DSLs we may need to perform some kind of transformation on the initial components. Or apply a final transformation to the resulting value. To gain this finer-grained control we can specify two additional functions in the result builder: buildExpression and buildFinalResult.

Their signatures are as follows:

static func buildExpression(_ expression: Expression) -> Component
static func buildFinalResult(_ component: Component) -> FinalResult

• The buildExpression(_ expression: Expression) -> Component function is used to transform the results of DSL statements, from the Expression type into the resulting Component type that will be used in buildBlock. It allows the type of the expressions appearing in the DSL to be different from the resulting type.

• The buildFinalResult(_ component: Component) -> FinalResult function is used to construct the final result that the result builder will return. It lets us distinguish the component type from the result type so that, for example, the result builder may perform internal transformations on a type we do not want to expose to clients and then finally transform it into the resulting type.

These functions are optional. If we do not specify them, the result builder works only with the Component type, as we saw in the examples above.

A simple example is the following one, in which we define a result builder that constructs an array of real numbers. The expressions we write in the DSL are integers.

@resultBuilder
struct ArrayBuilder {
    static func buildExpression(_ expression: Int) -> [Int] {
        return [expression]
    }

    static func buildBlock(_ components: [Int]...) -> [Int] {
        return Array(components.joined())
    }

    static func buildFinalResult(_ component: [Int]) -> [Double] {
        component.map {Double($0)}
    }
}

• The buildExpression function transforms the original integer into an array with a single element. In this case the Expression type is Int and the resulting Component type is [Int].

• buildBlock is the function that joins several components, one-element integer arrays, into a final result: an integer array.

• And the buildFinalResult function transforms the component produced by the previous function into the FinalResult type, [Double].

We can see the result of how it works in the following example:

@ArrayBuilder
func buildArray() -> [Double] {
    100
    100+100
    (100+100)*2
}

print(buildArray())

In the DSL that defines the function body, we write three statements that return integers. Those three statements are the expressions the result builder will use to apply all the transformations above.

The final result is the following array of real numbers:

[100.0, 200.0, 400.0]

References

• Result Builders proposal in Swift Evolution
• Detailed explanation in the Language Reference
• Code file with the examples from the post

Addendum (April 2025) - What has happened in Swift over these three years?

Quick context
Since you published this second article in 2022, Swift has kept evolving.
This appendix summarizes the changes that affect the topics discussed here: initializers, protocols, and advanced functions such as buildExpression and buildFinalResult.

1. Initializers + result builder -> now also in classes

Swift 5.8 expanded the ability to mark designated class initializers with builder attributes. An example adapted to your Persona:

class Persona {
    private let contenido: () -> String
    var saludo: String { contenido() }

    init(@StringConcatenator contenido: @escaping () -> String) {      // valid in 5.8+
        self.contenido = contenido
    }
}

2. Memberwise + attributes: they are generated automatically

Starting with Swift 5.9, when you annotate a stored property with a builder, for example @StringConcatenator let contenido: () -> String, the compiler no longer requires you to mark the corresponding parameter in the memberwise initializer with the same attribute; it does that on its own.

struct PersonaSimple {
    @StringConcatenator let contenido: () -> String        // enough on its own
}

3. Protocols with builders: they now support async/throws

With the adoption of Strict Concurrency in Swift 5.10, protocol requirements can be declared like this:

protocol Educado {
    @StringConcatenator var saludo: String { get async }
}

Whoever implements the protocol will be able to use a builder and also return an asynchronous value.

4. New intermediate-stage functions

Swift 5.7 introduced buildPartialBlock(first:) and buildPartialBlock(accumulated:). If you implement them, you can omit buildBlock, and the compiler will assemble the result incrementally, which is useful for performance in heavy builders.

static func buildPartialBlock<each T>(first value: repeat each T) -> (repeat each T) { value }
static func buildPartialBlock<each T>(accumulated: (repeat each T), next: (repeat each T)) -> (repeat each T) {
    (repeat each accumulated, repeat each next)
}

Tip: with parameter packs (<each T>) you do not need overloads for 1…10 elements.

5. buildExpression + error propagation

If your buildExpression can throw, you can now mark it as throws in Swift 5.9. The error propagates to the point where the builder is used; it does not need to be caught inside the builder.

static func buildExpression(_ value: Int) throws -> [Int] { ... }

6. Macros vs. Result Builders (brief reminder)

The new era of Swift Macros (SE-0389/0397) does not replace builders, but it does cover cases that we previously forced with them:

To keep digging deeper

• SE-0390 - Variadic Generics (parameter packs)
• SE-0389 / SE-0397 - Swift Macros
• WWDC23 “Design Data-Driven Apps with Result Builders”

We already saw that the Chinese model is enormous, with 671B parameters, 671 billion floating-point numbers, which means 1.342 GB of RAM, and that it is not possible to run a model of that size on our own computer.¹ What we can download are distilled versions of small open-source models that the company itself has released. For example, DeepSeek-R1-Distill-Qwen-7B is a model created from Qwen-7B. It occupies around 4.7 GB on disk and can be run on a MacBook Air with 16 GB of RAM. There are similar models built from different open base models, such as DeepSeek-R1-Distill-Qwen-32B or the more powerful DeepSeek-R1-Distill-Llama-70B, based on the 70-billion-parameter Llama model.

The problem with these models is that, as we will see, they are much less capable than the large model. They are not able to solve even simple problems like the one we posed, combining several numbers to obtain a target number.

The alternative is to use a provider, a cloud service that hosts the model and runs it on its own computers. In a report by Artificial Analysis we can review the performance and price of the main providers that offer access to DeepSeek, such as Hyperbolic.xyz, together.ai, Fireworks.ai, or even the Chinese company’s own API.

In this article we will go step by step through how to install a small model on our own computer, run it locally, and also how to run DeepSeek-R1 through a provider. To run a local model I will use my MacBook Air M3 with 16 GB of RAM, the terminal, and the Python llm library by Simon Willison. For the provider, I signed up for Fireworks.ai and I will use its API from a simple Python program.

How to run DeepSeek locally

There are several ways to run an LLM locally on a MacBook with an Apple chip, M1, M2, M3, or M4. The simplest is to use the LM Studio application, with which we can download and run different available models.

The application lets us search among the available models, we can select GGUF or MLX formats, different ways of packaging models to run them locally on the Mac, and it warns us if a model is too large for our computer. For example, in the following image the Google Gemma 3 27B model is selected. It uses 15.40 GB of memory and is too large to run on my 16 GB Mac.

Downloading DeepSeek with Simon Willison’s llm library

But instead of using that application, let us be a little braver and use the option preferred by true geeks: the command line, just like we did when computers booted into text mode and the mouse was optional. We will use Python and Simon Willison’s llm library.

Open the Terminal app, and the first thing we have to do is install Python. The simplest way to do it is with Homebrew. Run the following command in the terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

After that, you can use brew to install the packages we are interested in, such as Python:

~ $ brew install python

Once it is installed, check that both Python and pip, Python’s package installer, are working correctly:

~ $ python3 --version
Python 3.13.2
~ $ pip3 --version
pip 25.0 from ... (python 3.13)

It is a good idea to create a directory in which we keep all the Python programs related to the same project, together with all the libraries those programs need. We can create a dev directory to hold our programming projects, and inside it create an llm directory, we could use any other name, for the current project:

~ $ mkdir dev
~ $ cd dev
dev $ mkdir llm
dev $ cd llm

Once inside that directory, we activate a Python virtual environment and install the llm Python library:

llm $ python3 -m venv venv
llm $ source venv/bin/activate
llm (venv) $ pip3 install llm

This is the cleanest way to work with Python, so that the libraries of one project do not get mixed up with the libraries of another. Let us explain it briefly. In the current directory, a folder called venv has been created, llm/venv, where the libraries we download with pip will be stored. To use venv as the base directory for Python libraries, we have to “activate” it by running the activate command created inside it. At that point the prompt shows (venv), and any library will be installed inside the venv directory.

The llm library needs and installs many other packages. We can list them:

llm (venv) $ pip3 list
Package             Version
------------------- -----------
annotated-types     0.7.0
anyio               4.9.0
certifi             2025.1.31
charset-normalizer  3.4.1
click               8.1.8
...

Once the llm library is installed we can run the llm script to chat with a local model. But first we have to download one. To do that, we need to install the plugin that allows llm to work with models packaged in MLX format, an Apple format that lets LLMs run efficiently on its processors.

llm (venv) $ llm install llm-mlx

The available MLX models can be browsed in the MLX community on Hugging Face. Simon Willison also lists some of them in the explanation of the MLX plugin on GitHub. Let us download a small distilled DeepSeek-R1 model, mlx-community/DeepSeek-R1-Distill-Qwen-7B-8bit, which occupies 8.10 GB and uses 7.76 GB of memory.

llm (venv) $ llm mlx download-model mlx-community/DeepSeek-R1-Distill-Qwen-7B-8bit

Once it has been downloaded, we can finally run it with the -m option and the name of the model we just downloaded. We pass the prompt in quotes:

(venv) llm % llm -m mlx-community/DeepSeek-R1-Distill-Qwen-7B-8bit "What is 1+1?"
<think>
First, I identify that I am being asked to calculate 1 plus 1.

I know that when I add these two numbers, the result is 2.

Therefore, the correct answer is 2.
</think>

Sure, let us solve the sum step by step:

\[
1 + 1 = 2
\]

**Final answer:**
\[
\boxed{2}
\]

At last we can see the thought process and the answer of the reasoning model running locally.

We can confirm what we discussed in the previous article: DeepSeek-R1 was trained to first generate a step-by-step reasoning process between <think> tags and only then, once it reaches a conclusion, generate the answer.

Trying the downloaded model

Once downloaded, we can try the model on the problem from the previous article. Recall that the challenge was to find a way to operate on the numbers 2, 3, 6, and 120 to obtain 84. And recall as well that the reasoning needed to solve it was not complicated:

One way to get 84 from 120 is to subtract (120-84) from 120.
In other words, I need to subtract 36. Can I obtain 36 from 2, 3, and 6?
Yes, by multiplying them:
2*3*6=36. Then I already have the solution:

84 = 120-2*3*6

The full version of DeepSeek, the one on the Chinese website, did solve the problem perfectly, as shown in the image we saw in the previous article.

But there the full DeepSeek model was running, the one with 671B parameters. Does the model we downloaded, a 7B-parameter model, one hundred times smaller, do just as well? The answer is no. We can see it in the following transcript, where, after trying different alternatives, it never finds the correct one and ends up saying that no solution is possible:

(venv) llm % llm -m mlx-community/DeepSeek-R1-Distill-Qwen-7B-8bit "What operations can I
perform on the numbers 2, 3, 6, 120 so that the result is 84?"
<think>
First, I analyze the available numbers: 2, 3, 6, and 120. My goal is to operate
on them to obtain 84.

I consider the basic operations: addition, subtraction, multiplication, and division.
I also think about powers and roots if necessary.

I try combining 2, 3, and 6 in a way that, when used with 120, produces a result
close to 84. Multiplying 2 by 3 gives 6, and multiplying that result by 6 gives 36.
Then, I divide 120 by 36 and get approximately 3.333, which is not exact.

I review another combination: I divide 120 by 2, getting 60. Then I subtract 6 and 3 to
obtain 3, and subtracting those values from 60 gives me 57, which still does not reach 84.

Finally, I multiply 2 by 3 to get 6, add 6 to get 12, and then subtract that result
from 120, obtaining 108. However, 108 is not equal to 84.

After analyzing different combinations of multiplication and division, I identify
that the closest result is 84 by multiplying 2 by 3 by 6 and then dividing by 3.
</think>

To operate on the numbers **2, 3, 6**, and **120** in order to obtain **84**, follow these steps:

1. **Multiply 2 by 3:**
   \[
   2 \times 3 = 6
   \]

2. **Multiply the result by 6:**
   \[
   6 \times 6 = 36
   \]

3. **Divide 120 by the previous result:**
   \[
   120 \div 36 = 3.\overline{3}
   \]
   However, this result is not exactly 84. After reviewing different
   combinations, no operation is found that yields exactly 84 with
   the given numbers.

Therefore, with the numbers **2, 3, 6**, and **120**, it is impossible to obtain
exactly **84** using the basic operations of multiplication
and division.

\[
\boxed{\text{It is not possible to obtain exactly 84 with the numbers 2, 3, 6,
and 120 using basic operations.}}
\]

Since language models are stochastic models, other executions of the same prompt produce different results. But although I have tried several times, the correct solution never appears. Sometimes the model gets stuck in the thought process in endless loops and stops abruptly. Other times it ends with an incorrect solution. And at other times it says what we just saw: that it is not possible.

So even though we can download small models that have been taught using DeepSeek and that reason following the same strategy, these models are much less capable than the original model. So how do we run the large model without going through the official Chinese website? That is what we will see next.

How to run DeepSeek through a provider

Of all the providers I chose Fireworks. Not for any special reason. I looked at its website and everything seemed fine to me. It is all fairly well documented and it has the latest models available.

You need to sign up, register your card, and buy some credits. The minimum amount is $5. With that amount you can do quite a few queries. On the pricing page you can see that one million tokens cost $3. In truth, it is not quite as cheap as people said. For example, compared with the price of OpenAI’s models, 1M tokens of o1 cost $15 and 1M tokens of o3-mini cost $1.10.

Once registered, you need to create an API key so that you can send requests and use it with whatever API client you choose. We have to save the value of the API key because, once created, it will not be shown again. We must not share that string of characters, because it is what identifies us when the price of the API requests is charged against our credits.

How do we send requests to the Fireworks API? We could use the same llm script from before by adding a plugin to access Fireworks models. But, to try out other ways of using these models, we are going to do it by writing a small Python program.

To write the program it is advisable to use an editor such as Visual Studio Code. Once installed, the most comfortable way to use it is with a setup like the one in the following figure. On the left, the file explorer panel opened in the folder we are working in; in the center, the editor panel with the program or programs we are working on; and below, a panel with an open terminal from which we can run those programs.

We need to install the Fireworks API package:

(venv) llm % pip install fireworks-ai

And then copy the example program from the Fireworks documentation, filling in the API key:

import requests
import json

url = "https://api.fireworks.ai/inference/v1/chat/completions"
payload = {
  "model": "accounts/fireworks/models/deepseek-r1",
  "max_tokens": 20480,
  "top_p": 1,
  "top_k": 40,
  "presence_penalty": 0,
  "frequency_penalty": 0,
  "temperature": 0.6,
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ]
}
headers = {
  "Accept": "application/json",
  "Content-Type": "application/json",
  "Authorization": "Bearer <API_KEY>"
}
response = requests.request("POST", url, headers=headers, data=json.dumps(payload))
print(response.text)

We call the program hello.py and run it in the terminal:

We have made a query to the DeepSeek-R1 model. We can see that the call works and that the API correctly returns the model’s answer. The response also includes the tokens consumed, in this case 180. If we go to the Home page of our account on Fireworks, we will see the credits we still have available. In this case, the same as before, because those 180 tokens represent a negligible amount and are not even charged.

We finish by changing the phrase "Hello, how are you?" to the original problem we have been posing to DeepSeek from the beginning: "What operations can I perform on the numbers 2, 3, 6, 120 so that the result is 84?" We can write the code in a new program, or simply replace the query in the current file, whichever we prefer. We run the script and, after waiting a little, it returns something like:

(venv) llm % python problem.py
{"id":"79fa6566-deb5-4fbe-83aa-7b45bf815dd4","object":"chat.completion","created":1743265311,
"model":"accounts/fireworks/models/deepseek-r1",
"choices":[{"index":0,"message":{"role":"assistant",
"content":"<think>\nOkay, let's see. I need to use the numbers 2, 3, 6, and 120 to get 84.
Hmm, maybe I can combine them with addition, subtraction, multiplication, division, or
exponents. Let me start by trying different combinations.\n\nFirst, 120 is a big number.
If I subtract something from 120, maybe that could work. Let me see: 120 minus 36
equals 84. But how can I get 36 from 2, 3, and 6? Well, 6 times 6 is 36, but I don't have
another 6. Wait, maybe 3 times 12? But how to get 12 from 2 and 6? 6 times 2 is 12. So,
3 times (6 times 2) is 36. Then 120 minus 36 is 84. Let me check that:\n\n
120 - (3 * 6 * 2) = 120 - 36 = 84. That works! But wait, did I use all the numbers?
The original numbers are 2, 3, 6, 120. In this case, I used 2, 3, 6, and 120. Yes, all of them.
So 120 minus (3 times 6 times 2) equals 84. Let me verify the multiplication order.
Multiplication is commutative, so 3*6*2 is 36. 120 minus 36 is indeed 84. \n\n
Alternatively, maybe there's another way. Let me think. If I divide 120 by something.
For example, 120 divided by (something) equals 84. But 120 divided by (10/7) is 84,
but getting 10/7 from 2, 3, and 6 seems tricky. Maybe not that approach.\n\n
Another idea: 6 times 14 is 84. How can I get 14 from 2, 3, and 120? 120 divided by 8.57...
which isn't helpful. Maybe subtract something from 6? 6*(120/(3+2))? Let's calculate
that: 3+2=5; 120/5=24; 6*24=144. That's too big. Not helpful.\n\n
Wait, going back to the first idea: 120 - (3*6*2) = 84. That uses all numbers once each.
That seems valid. Let me check again:\n\n3 multiplied by 6 is 18, multiplied by 2 is 36.
120 minus 36 is 84. Yes, that works. So the expression would be 120 - 3 * 6 * 2.
According to order of operations, multiplication comes before subtraction, so no
parentheses are needed if written as 120 - 3*6*2. But to make it clearer, maybe write it
as 120 - (3*6*2). Either way, the result is 84.\n\nI think that's a valid solution. Let me
see if there's another way. For example, using division. Suppose 6 divided by something.
6/(2/3) is 9, but I'm not sure how that helps. Or 120 - (6*3) + 2? 120 -18 +2=104. Nope.
Not helpful.\n</think>\n\nTo obtain 84 using the numbers 2, 3, 6, and 120, follow these steps:\n\n1. **Multiply 3 × 6 × 2**:\n
\\[\n   3 \\times 6 = 18 \\quad \\text{and then} \\quad 18 \\times 2 = 36\n   \\]\n2.
**Subtract the result from 120**:\n   \\[\n   120 - 36 = 84\n   \\]\n\n
**Final expression**:  \n\\[\n120 - (3 \\times 6 \\times 2) = 84\n\\]\n\n
**Answer**:  \n\\boxed{120 - 3 \\times 6 \\times 2 = 84}"},
"finish_reason":"stop"}],"usage":
{"prompt_tokens":30,"total_tokens":1038,"completion_tokens":1008}}

Bravo for DeepSeek-R1.

In that time, the waters have returned more or less to normal. Nvidia has more or less recovered in the stock market, OpenAI has lowered the price of using the API of its most advanced models and has presented Deep Research, a product that has delighted its users. And new models keep arriving: Grok, Claude 3.7 Sonnet, or the rumored GPT-4.5, expected in a few weeks. The world of AI remains in a frenzy: after the announcement of one new model, another product arrives, a new device, or a new statement that shortens the time supposedly left before AGI. It does not stop.

Even so, the reaction of American companies and leaders has not overshadowed the achievement of the Chinese startup that shares its name with the model. At the beginning of the year, I laid out in this newsletter a list of important questions that needed to be answered over the coming months. Barely a month later, at the end of January 2025, two of them had already been answered with DeepSeek-R1: an open-source reasoning LLM comparable to OpenAI’s o1 had been presented (Hugging Face, GitHub), and a detailed explanation of how it had been developed had been published.

In this article we will look at how this reasoning model works. And in a future installment I will explain how to use it on our own computers, without having to depend on the Chinese company’s website.

Trying out DeepSeek-R1

The simplest way to try DeepSeek is through its website. To sign up you only need an email address. Once inside, if you want to use the reasoning model, click the “DeepThink (R1)” button and you will be able to ask any question, in the style of ChatGPT.

Like o1, DeepSeek-R1 is a reasoning model. It is trained not to answer immediately, but to generate a chain of reasoning that allows it to explore the problem step by step and apply different strategies during that process.

Let us look at an example. Suppose we ask it the following problem:

What operations can I perform on the numbers 2, 3, 6, 120 so that the result is 84?

Take a moment to try to solve it. You can use paper and pencil or do it mentally. Got it? Give it a try, it is not very difficult.

You have probably assumed that you can only use each number once, which is usual in this type of exercise. You will have noticed that, since the result is smaller than 120, you need to subtract something to get 84. You may have tried different combinations, perhaps calculating 120 - 84 = 36, and then thinking that this 36 could be obtained from 2, 3, and 6. Maybe you even had an “Aha!” moment when you noticed that 2*3 = 6 and that 6*6 = 36. In the end, your solution might be:

84 = 120-(2*3*6)

Reasoning models, o1 from OpenAI or DeepSeek-R1, do something very similar. Until DeepSeek arrived, we could not directly observe their reasoning steps because o1 hid them. Now we can see that reasoning, as in the following figure.

Below we can see an English version in which I have inserted the <think> tags that DeepSeek-R1 uses to delimit its reasoning:

<think>
Okay, I need to figure out how to combine the numbers 2, 3, 6, and 120
using mathematical operations in order to get 84.

Let me start by writing the numbers down: 2, 3, 6, 120.
Hmm, 84 is less than 120, so maybe I need to subtract something from 120.
Let me see: 120 minus 36 is 84.

Wait, do I have the numbers needed to make 36?
Let's check. The numbers I have are 2, 3, and 6.
If I multiply 6 by 6, I get 36, but I do not have another 6.

Wait, but I do have 2, 3, and 6.
Maybe 6 times (3 times 2) gives me 6*6=36.
Then 120 - 36 = 84.

Let me verify that.
3 times 2 is 6, then 6 times 6 is 36.
So 120 minus 36 really is 84.

But wait, have I used all the numbers?
Let's review: 2, 3, 6, 120.
Yes, I used 2, 3, and 6 to obtain 36 and then subtracted it from 120.
So the expression would be 120 - (6 × (3 × 2)) = 84.

Alternatively, is there another way?
Let me see whether I can use division or addition.
If I add 2 and 3, I get 5, then 6 times 5 is 30,
but 120 minus 30 is 90, not 84.

And what if I divide 120 by something?
Maybe 120 divided by (6 - (3 - 2))...
Let me calculate it: 3 - 2 is 1, 6 - 1 is 5, 120 divided by 5 is 24.
That is not 84.

Let us try multiplying some numbers first.
6 times 2 is 12, 3 times 12 is 36, and 120 - 36 = 84.
That is the same as before.

Alternatively, 3 times 6 is 18, 18 times 2 is 36.
Yes, same result.
So maybe 120 - 2 × 3 × 6.
Let me calculate it: 2 × 3 = 6, 6 × 6 = 36, 120 - 36 = 84.
Yes, it works.

So the operations would be 120 - (2 × 3 × 6) = 84.
And all the numbers are used once: 2, 3, 6, 120.
That seems correct. I think that is the solution.
</think>

It is impressive how much DeepSeek-R1’s reasoning resembles our own. It does not even settle for the first solution, but considers several alternatives and, in the end, verifies that they all lead to the same result. Impressive.

How does OpenAI’s original model behave? Curiously, o1 needs more time to solve this problem, 1 minute and 17 seconds, and its reasoning is not as fluid. In the following image, the ChatGPT website hides almost the entire process and only shows a summarized version in Spanish. In the end it also reaches the correct result.

Reasoning in DeepSeek

In an earlier article I mentioned François Chollet’s thesis that LLMs implement a kind of intuitive reasoning (System 1, in Kahneman’s terminology) rather than analytical reasoning, slower, deliberate, logical, and analytical (System 2).

For the first time, we can see in detail how an AI model has learned to reason in a structured way, following a System 2 style of thought. We analyze it in the following image, in which I have highlighted DeepSeek-R1’s thinking in red and green. In red I mark the phrases that correspond to analytical and deliberate thinking, characteristic of System 2, while in green are those that reflect intuitive reasoning, System 1.

Let us analyze more closely the System 2 reasoning strategies in DeepSeek-R1, marked in red.

First, like a good math student, the model lists the main elements of the problem. It also introduces brief pauses to think, “Hmm”, “Wait”, “Let’s see”. It formulates hypotheses, “If”, “Let’s try”, “but”, “Maybe”, “Let’s see whether I can use”. It verifies solutions, “Let me verify that”, “Let’s try”, “Let me calculate it”. It proposes alternatives, “Alternatively, is there another way?”, “Alternatively”. And it checks the strength of its conclusions, “Yes, same result”, “Yes, it works”, “That is the same as before”. Finally, it recaps and concludes, “That seems correct. I think that is the solution.”

How does DeepSeek-R1 achieve this level of reasoning? In my review of o1 I already mentioned that OpenAI’s great advance had been to teach an LLM to reason using a reinforcement-learning approach, RL. DeepSeek has managed to replicate this strategy and, most notably, has published all the details in its paper. In reinforcement learning, people talk a lot about the GRPO algorithm, Group Relative Policy Optimization, introduced in that article, and about how it improves on the original PPO that OpenAI used to apply reinforcement learning to LLMs, RLHF. Andrej Karpathy explains this reinforcement-learning idea very well in his video Deep Dive into LLMs like ChatGPT, from timestamp 2:07:00 onward.

Numerous laboratories and startups are trying to reproduce DeepSeek’s results and to boost their base models with similar RL techniques. Over the coming months we will surely see these reinforcement-learning innovations advance and spread through a large part of the industry.

DeepSeek V3

Now let us focus on the phrases in green in DeepSeek-R1’s reasoning, the ones corresponding to the intuitive thinking generated by the base model.

We can see that it is capable of remembering the numbers in the problem, “2, 3, 6, 120”, imagining approaches to solve it, “I need to subtract something from 120”, “Do I have the numbers needed to make 36?”, carrying out mental operations, “3 times 2 is 6, times 6 is 36”, and checking results, “That is not 84”, “that is 90, not 84”. All of these functions belong to that more spontaneous mode, typical of System 1, which the LLM performs in a single inference pass.

For the full reasoning to work well, you need a good base model. Otherwise, if the intuitions were wrong or strayed too far, even the most elaborate reasoning would not help very much. System 2 thinking can propose hypotheses, validate them, and spend more time reflecting, but without good initial ideas it would not reach a satisfactory result.

What is DeepSeek-R1’s base model? It is DeepSeek-V3, another open-source development (Hugging Face, GitHub) that the company presented at the end of December 2024. We are dealing with a very large model, 671 billion parameters, 671B, larger than LLaMA-3.1, which has 405 billion parameters. It also offers a context window of 128K tokens, similar to that of the most advanced models today.

The benchmark results presented by the Chinese company were excellent, beating Llama 3.1, and in almost every case even Claude-3.5 and GPT-4o as well, see the figure above. Even so, the December announcement did not have much impact. The dates were bad, and public opinion was already somewhat saturated with news of new models. Even the low training cost, instead of being seen as important news, made many people doubt the reliability of the benchmark results.

But the later success of R1 has strengthened the credibility of DeepSeek-V3, dissipating many of the doubts raised around those initial benchmarks and marking a significant milestone in the evolution of open-source LLMs.

Can I run DeepSeek on my computer?

We have seen that DeepSeek-R1 is a reasoning model comparable to o1. And it has the advantage that the Chinese company has released it openly. Does this mean that I can download it and run it on my computer? In theory, yes. But unfortunately, I do not think you have a computer with the necessary capacity.

DeepSeek-R1 is built from DeepSeek-V3 by fine-tuning its parameters, but this does not modify the size of the base model. The final result keeps the previously mentioned size of 671 billion parameters. These parameters are floating-point numbers that must be loaded into memory and processed, so it would require around 1.342 GB (!) of RAM just for the weights, and that figure usually increases because of other processes, reaching 1.600-2.000 GB of VRAM or RAM. That does not fit on my laptop. A cluster with multiple high-capacity GPUs would be needed. For example, 20 H100 GPUs with 80 GB each could cover around 1.600 GB.

So what models can we download and run locally? Many articles have been published explaining how to download something similar to DeepSeek. What are we actually downloading? They are distilled versions that DeepSeek itself has released, smaller open models that have been trained using reinforcement learning on reasoning traces generated by R1. For example, DeepSeek-R1-Distill-Qwen-7B is a model built from Qwen-7B. It takes up around 4.7 GB on disk and can be run on a MacBook Air with 16 GB of RAM. There are similar models built from different open base models, such as DeepSeek-R1-Distill-Qwen-32B or the more powerful DeepSeek-R1-Distill-Llama-70B, based on the 70-billion-parameter Llama model.

The problem with these versions is that they are considerably worse than the original DeepSeek-R1. Although they inherit some of the master model’s capabilities, their reasoning is more limited, with simplified answers and less ability to structure solutions step by step. On complex problems, they tend to omit key steps and make more mistakes, because the base model is much simpler.

In the next article I will explain how to install one of these models on our computer, and we will see that its performance is far worse than R1’s. Finally, I will explain how to use the API of some providers that make the original DeepSeek-R1 available on their servers.

I am increasingly convinced that we are at a crucial moment in technology, similar to what happened in the mid-1940s, 80 years ago, when the first electronic computers were developed. Back then, it took years for those advances to reach society: the first UNIVAC installations date from the 1950s, the first programming languages (Fortran, Lisp) appeared in the late 1950s and early 1960s, and we did not see the first personal computers until the late 1970s.

I think history has sped up, and what then took decades could now unfold in just a few years. I also think the coming years will keep bringing us spectacular advances in AI, and that society will absorb them little by little, but faster and faster. And when we reach the beginning of the 2030s and look back, we will probably see 2025 as the key year in which everyone understood that this was not a bubble, but a revolution.

Even so, all of this is speculation. Nothing is certain, and we are living through a moment in which there are still many open questions waiting to be answered. Here they are.

1. The first question has been hanging in the air for months, and it should be answered soon, at the beginning of 2025. Will GPT-5, Claude Opus, or Gemini 2 Pro arrive? Will they be much better than the current models? Will we get a new wave of models with 10 times more parameters? What happened in 2024 with the supposed failures of GPT-5 and Opus showed that scaling is not so simple. Perhaps data is missing, or perhaps the resulting models are difficult to tune. Will some method be found to break through this apparent ceiling?

2. How was ARC-AGI solved? The great feat of the not-yet-published o3, the new version of OpenAI’s “reasoning” model, has been to solve the ARC-AGI test. There is not much information about how they did it. Will other open models be able to reproduce it? Will it be possible to reproduce it with other approaches? Will we see in 2025 some paper or open-source model that replicates it? Will OpenAI be able to solve the new version 2 of ARC-AGI that François Chollet is designing?

3. The success of 4o in 2024 was quickly followed by a flood of small models that practically matched its performance. Will this story repeat itself with o1? Will Google or Anthropic manage to reproduce these reasoning models? Will the open-source community be able to develop similar alternatives with fewer resources?

4. Everything suggests that 2025 will be the year of agents. How will an “agent” be defined in practice? My bet is that OpenAI will give us an agent in the form of a web browser. In 2025 we could see a browser piloted by GPT-5 (or another “o” model) capable of investigating, moving around the web autonomously, gathering information, and asking the user for confirmations. OpenAI has shown itself to be very skilled with user experience, and it would not be surprising if it launched an intelligent browser that completely redefined the use of the web, introducing a continuous interaction in which the agent spends several minutes, or more, reasoning, researching, and consulting.

5. How will the integration of AI into our mobile devices evolve, and what will happen with Apple Intelligence? For now, the general perception is that Apple Intelligence has disappointed. We are still far from truly useful features. Given the amount of personal data we keep on our phones, AI should become a genuinely intelligent assistant that can securely access that information in order to help us. We will see whether Apple or Google surprise us with something that really marks a qualitative leap in this area.

6. Will there be any company that achieves economic success in 2025 by using AI? For now, the only profitable business model seems to be the subscription OpenAI offers to end users. Will successful new AI-based products emerge? Will some company appear that builds its service entirely with these technologies? Antonio Ortiz has compared this situation more than once to the Gold Rush: the greatest profits did not go to the prospectors, but to those who sold the tools: shovels, sieves, wheelbarrows, and so on. Will any “gold” be found in 2025?

7. Will reinforcement learning be integrated into popular models such as ChatGPT, so that we can teach them in real time? OpenAI has recently introduced reinforcement-learning techniques for fine-tuning models. What exactly are they based on? Could they be integrated in a simple way into interactions with end users? For now, models are frozen and do not change through interaction with users. The most we can do is use context to incorporate instructions or documents. Imagining a ChatGPT that adjusts its own weights with each interaction, truly learning from what we teach it, would be a giant step toward a real AGI that could act as a tutor and personal assistant.

I think these seven questions are important enough that, if they are answered in 2025, they will shed a great deal of light on what the future of AI may hold for us.

We will revisit them in a year’s time and check whether they have been answered or whether, as sometimes happens, the situation has changed so radically that new questions have appeared and made these ones irrelevant.

In fact, I have changed the title of the newsletter: it is no longer called Fifteen Days. That takes away the pressure of having to publish twice a month and cover current events. There are already many very good AI news newsletters. I want to continue with the approach of the most recent issues, where I go into some depth on a topic that does not necessarily have to be news-driven.

But on Friday something happened that has to be covered here, no matter what.

OpenAI has released extremely impressive results for its new reasoning model o3, the next version after o1. The presentation video is below:

All the results they show point to a spectacular leap on the hardest benchmarks. For example, they go from 48.9% to 71.7% on SWE-bench Verified, a benchmark of programming problems. And they also go from 3% to 25% on Frontier Math, a test made up of PhD-level mathematics problems.

But what was truly surprising is that they managed to solve François Chollet’s ARC-AGI competition. In the o3 launch video they showed the figure above, where o3 reaches 75.7% in its “low” version and 87% in its “high” version.

What do “low” and “high” mean here? As we saw in the article where we talked about how o1 works, these reasoning models can refine their results when they are given more compute time. “Low” and “high” are the names OpenAI researchers use for o3 with a smaller or larger inference-time compute budget. The chart also shows that the compute time of o3 low is significantly higher than that of o1 high, which only gets 32% (they do not say whether the horizontal axis is linear or logarithmic; I assume it is linear, like the vertical one).

The moment Greg Kamradt, president of the ARC Prize, appeared in the OpenAI stream and explained all this, my mind was blown. I could hardly believe it. I ran to X to check reactions, started reading posts from people close to the competition, and finally, when I saw Chollet’s own reaction, I knew it was real. The OpenAI team behind the “o” models had done something historic: solve ARC-AGI. In just three months, since the presentation of o1, they had solved a challenge designed to reveal reasoning ability and human-like intelligence.

What does this enormous success of o3 imply?

The most important implication is that it validates the approach behind the “o” family of reasoning models, and shows that these models successfully integrate the intuition of traditional LLMs (System 1) with some kind of deductive, iterative System 2 reasoning. OpenAI has found the ingredients of the definitive recipe, the one that combines the two kinds of reasoning we discussed in the article on Chollet. That combination is crucial for the future, because it guarantees continued progress. On the one hand, when a better intuitive model appears (GPT-5), it will be easy to plug it into the new “o” model. On the other hand, as deductive capabilities improve and compute becomes cheaper, we should also get major gains.

Another key implication is that this confirms the role of NVIDIA and of chip manufacturers in general. And of the energy needed to power them. Whoever has more megaFLOPS will get the best results. Ilya Sutskever has just said that data is the new fossil fuel. Compute power is too.

Finally, it is worth highlighting OpenAI’s enormous luck, or excellent execution. They were able to end the year with a spectacular breakthrough and found, with the “o” models, a way to keep moving toward AGI without relying on their next GPT model. Today, in the Wall Street Journal, there is a detailed report on the problems they are having in developing GPT-5. It seems that the two or three pre-training runs OpenAI has attempted have failed after months of computation. A model ten times larger than GPT-4 also needs ten times more data, at a minimum, and they seem to be struggling with that. The debate over whether there is a wall in deep learning is still unresolved.

It is also worth clarifying that, although o3’s success has been spectacular, we have not reached AGI yet. There are still many missing ingredients in these models, such as the ability to reason with a physical model of the world, continual learning, or creativity.

We will keep a very close eye throughout 2025 on these foundational research issues around deep learning and language models, because they will shape the future of technology.

In the meantime, the advances we have seen in 2024 already enable many applications that are still waiting to be built with the models we already have.

This is not slowing down.

See you next time.

The McCulloch and Pitts paper

The 1943 paper by McCulloch and Pitts, A Logical Calculus of the Ideas Immanent in Nervous Activity, was the foundational work of connectionism and of artificial neural networks, ANNs. The paper starts from what was known at the time about the operation of neurons and establishes, in a dense and highly mathematical way, an equivalence between the functioning of neurons and a network of logical propositions in which neurons represent variables that can only take two values, “all-or-none”, identified with the Boolean values TRUE and FALSE.

Image from the original McCulloch and Pitts paper showing neurons as logical units.

Five years earlier, in 1938, Claude Shannon had published at MIT his master’s thesis A Symbolic Analysis of Relay and Switching Circuits, in which he showed how to design electrical circuits that implemented logical operations defined by Boolean algebra. This practical approach made it possible to build devices capable of carrying out basic logical functions such as AND, OR, and NOT, and it marked the beginning of the extraordinary 1940s, when researchers such as Von Neumann and Alan Turing designed the first digital computers.

Table from Shannon’s article showing the logical operations performed by his circuits.

Although McCulloch and Pitts did not cite Shannon in their paper,¹ it is quite likely that they were inspired by his concept of logical circuits. If those circuits could be the basis of the new digital computers that were beginning to appear, why could they not also be the way neurons worked? In any case, both papers became foundational and laid the groundwork for the late-1940s revolution in digital computers and for today’s AI revolution. As a curiosity, both models were based on Boolean algebra, a theoretical system proposed a century earlier by the English mathematician George Boole. It is a fascinating example of how a mathematical theory that at first seems purely abstract can, a hundred years later, become the foundation of radical technological change: digital computers and artificial intelligence.

The McCulloch-Pitts model was later modified in several ways, the most important being the Perceptron introduced by Frank Rosenblatt in 1958. In that model, weights were added to the connections between neurons and an activation threshold was introduced, determining whether a neuron “fired” or not, producing a 1 or a 0, depending on the weighted sum of its inputs. If the sum exceeded the threshold, the neuron activated.

In modern neural networks this concept has evolved a great deal. Contemporary activation functions, such as sigmoid, hyperbolic tangent, or ReLU, allow neurons to take continuous values instead of binary ones. Sigmoid, for example, produces values in the range [0, 1], while tanh normalizes them into [-1, 1]. This evolution has given modern neural networks greater flexibility and learning capacity, making it easier to represent complex relations and to propagate errors efficiently during training.

With these modifications, with the discovery of efficient weight-update algorithms, and with the exponential multiplication of the number of neurons,² ANNs have reached the point where, in Ilya Sutskever’s words in his NeurIPS 2024 talk, they can do anything a human can do intuitively:

They can do anything a human being can do in a fraction of a second.

The human brain as an enormous artificial neural network

More than eighty years ago, in the conclusion of their paper, McCulloch and Pitts made bold claims equating the complete functioning of the brain with the state of their proposed network of logical neurons:

The specification, at a given time, of the afferent stimulation and of the activity of all the constituent neurons, each with all-or-none behavior, determines the [global brain] state. The specification of the nervous network provides the necessary law of connection, by means of which the successor state can be calculated from the description of any state.

Every idea and every sensation is realized through activity within that network.

The level of self-confidence and lack of humility is striking. These statements were not presented as hypotheses or future speculations, but as definitive conclusions. And yet they lacked experimental support, since they were based only on a mathematical formulation, without references to neurophysiological or cellular research backing them up.

Curiously, time may have proved them right. The idea that ANNs are a valid abstraction of natural neurons has become the central dogma of connectionism, as Ilya Sutskever described it in that same NeurIPS 2024 talk:

It is the central idea of deep learning: the idea of connectionism. It is the belief that if you accept that an artificial neuron is in some sense analogous to a biological neuron, then you can trust that very large neural networks, even if not necessarily as large as the human brain, can be configured to perform practically all the tasks that we, as human beings, are capable of carrying out.

Ilya Sutskever at his NeurIPS 2024 plenary talk, presenting the central idea of connectionism.

Many current AI scientists share this reductionist outlook. Oriol Vinyals, a prominent Google DeepMind researcher, said in an interview on the DeepMind podcast:

You can imagine one neuron connected to many others, and what you essentially do is sum all the activations of the incoming neurons, multiplied by their respective weights. That is, in essence, how a brain works, with a bit of creative license.

One of the great strengths of this abstraction has been its computational efficiency. Neurons and weights can be represented as huge matrices of real numbers and processed in parallel on powerful GPUs designed for massive linear-algebra workloads. This has made it possible to train ever larger and more complex ANNs, with spectacular successes in recognizing patterns and regularities, in ways that imitate the performance of biological neural networks.

This reductionist approach is not the result of ignorance, but rather a deliberate choice to ignore the biological complexities underlying the behavior of natural neural systems. For example, after spending years studying the biological basis of intelligence, Demis Hassabis argues in Neuroscience-Inspired Artificial Intelligence that the key is to work at the right level of abstraction, one that sits above the real neural substrate:

From a practical perspective, we do not need to adhere strictly to biological plausibility in order to build AI systems. What works is ultimately what matters. Therefore, biological plausibility is only a guide, not a strict requirement. Our interest lies in an understanding at the algorithmic and computational level of the brain, leaving aside the level of biological implementation.

Hassabis explicitly refers to David Marr’s three levels of analysis, and regards the lowest one, the physical mechanisms of the neural substrate, as less relevant:

Our interest lies in the top two levels of the three levels of analysis that Marr proposed for understanding any complex biological system: the computational level, the goal of the system and the problem it solves, and the algorithmic level, the processes and algorithms that enable it to reach that goal. The implementation level, which deals with the specific physical mechanisms, is less relevant here.

The rhythms of neurons

What seems less relevant to engineers and computer scientists is the everyday reality of neuroscience research. And the neuroscientific picture is completely different.

On the one hand, to simulate the operation of a neuron in theory, neuroscientists need complex differential equations that describe the temporal evolution of ion concentrations across cell membranes, including what happens at the synapses. Some connections reinforce the membrane potential, excitatory ones, while others inhibit it. Both types of interaction occur as part of dynamic firing patterns in networks of thousands of connected neurons. The outcome is a burst of firing, or spikes, from one neuron, which then become the input for many others.

Diagram of the temporal sequence of spikes entering a neuron and the output spike train produced by that neuron.

On the other hand, experimental advances have made it possible to observe neuronal activity directly, using techniques such as microelectrodes, which can record the activity of both large populations and individual neurons. These recordings have shown that neurons fire in precise temporal patterns, and that the frequency of those spikes can encode information. In neurons connected to muscles, for example, firing intensity can reflect the force required for a movement, while in other brain areas temporal patterns are associated with sensory or cognitive signals.

Temporal recording of individual neuron activity while a mouse moves along a 170 cm linear environment. Panel (G) shows spike trains from different neurons in relation to the mouse’s position and speed. Each colored dot marks a firing event from a specific neuron. Panel (H) zooms in on 250 ms intervals and reveals precise temporal patterns and synchronization between neurons. This kind of activity reflects how biological neural networks organize information at precise temporal scales and exhibit rhythms that help integrate spatial and motor stimuli. György Buzsáki and Mihály Vöröslakos, “Brain rhythms have come of age”, Neuron, April 2023.

The importance of brain rhythms is a growing area of interest in current neuroscience. Beyond individual firing patterns, researchers have identified rhythmic oscillations at frequencies such as theta, gamma, and delta that organize neural activity on broader temporal scales.

Jeff Lichtman, a leading Harvard neuroscientist and pioneer in connectome mapping, emphasized in a recent Mindscape interview with Sean Carroll that although neuroscience has made huge strides in mapping neural connectivity, those maps cannot capture the dynamic complexity of the brain. As he puts it:

Although it is possible to map the network of neural connections in great detail, those maps do not reveal the strength of synapses, the nonlinearities in cellular response, and especially the timing with which different stimuli activate a cell. There are excitatory and inhibitory inputs, as well as modulatory neurotransmitter inputs. All of that is latent in the wiring diagram, but inaccessible from it.

This temporal dimension is a physical phenomenon specific to biological neural networks, and deep-learning connectionist models cannot capture it. Its existence in the real brain suggests that it plays a fundamental role that we still do not fully understand. And, according to Lichtman, we do not yet understand the brain’s complexity itself either:

There are certain things in the world, and perhaps the brain is one of them, that may have no possible simplification. If there were, brains would have been simpler.

What are the brain’s rhythmic oscillations for? What do they encode? How important are the different firing patterns of neurons?

In the next article we will return to these questions in more detail and I will present my favorite speculation: consciousness may be an emergent phenomenon of the oscillatory patterns of the brain. More specifically, sentience might be the result of firing patterns in biological neural networks, present in most living beings with complex nervous systems. Yet that capacity may remain out of reach for artificial systems based on algorithmic sequential processes, which by their very nature may never be able to have sensations.

See you next time.

François Chollet on MLST

Last November 6, the podcast Machine Learning Street Talk published a fascinating interview with François Chollet. It is a conversation lasting more than two and a half hours in which Chollet revisits topics ranging from technical matters, such as how LLMs work and how they differ from AGI, to philosophical and social questions about AI, such as the emergence of consciousness in children or the existential danger associated with AI.

I have already written about Chollet several times here. For example, in this post I discussed his interviews with Dwarkesh Patel and Sean Carroll. There we saw that he is someone with a very strong technical background, he is the creator of Keras, the library for neural networks, and the author of the book Deep Learning with Python, whose opinions carry a great deal of weight in the LLM community. That is true both because of his paper On the Measure of Intelligence and because of the current ARC competition, which we also discussed here.

Today I want to go through that November 6 interview in some detail. Besides the podcast, it is also available on YouTube, where it has already passed 40,000 views.

In this article I will extract and comment on a number of quotes from the interview. Given how long the conversation is, I have had to make a substantial selection, focusing on the parts that interested me most. So this is, inevitably, a rather biased article. But I have not changed the meaning of Chollet’s remarks at all. Not everything he said is here, but everything that is here is something he did say.

If you want more technical detail on the topics discussed here, you can also check the talk that Chollet and Mike Knoop, the other organizer of the ARC competition, prepared for a university tour presenting the challenge:

• YouTube
• Post
• Slides

System 1 and System 2 intelligence

ChatGPT-generated image.

The theory of System 1 and System 2, proposed by Daniel Kahneman in his book Thinking, Fast and Slow, describes two modes of thought that operate in the human mind. System 1 is fast, automatic, intuitive, and emotional. It works subconsciously and relies on pre-existing patterns to make decisions without conscious effort. System 2, by contrast, is slow, deliberate, logical, and analytical. It activates when we need to focus, solve complex problems, or make important decisions that require careful analysis.

This theory has had a strong impact on Chollet, who believes LLMs can easily implement System 1, but not System 2.

Deep learning models are excellent at producing results that are directionally accurate, but not necessarily exact. They are very good at generating useful suggestions. The System 1 process is similar to what LLMs do. They also rely on pattern matching and mechanisms akin to intuition.

For Chollet, unlike System 1, we can understand how System 2 works through introspection:

Introspection can be very effective for understanding how your mind handles System 2 thinking. It is not nearly as effective for System 1, because System 1 operates unconsciously and instantaneously, in parts of the brain to which you do not have direct access. It happens below the surface, outside conscious observation.

System 2, by contrast, is deliberate, slow, and low-capacity. Only a few things happen at any given moment, and it is inherently introspective.

How the mind works when operating in System 2 mode is similar to the step-by-step execution of a program. It is what we do, for example, when we sort a list of numbers or mentally add two long numbers: we run an algorithm we have learned.

Your thoughts exist in your mind in the form of programs.

Chollet argues that a fundamental feature of intelligence is the ability to execute those programs mentally, verify whether they work, and solve a new task by selecting and combining the best ones:

When you face a new task, you can mentally describe it using a set of properties and then generate a small number of hypotheses about programs that satisfy those descriptive constraints. You then test those hypotheses mentally to verify whether your intuition is correct. That is a classic example of System 2 thinking: it is, essentially, how program synthesis works in the brain.

The limits of deep learning

Slide from the ARC Prize 2024 University Tour.

When Chollet started working with deep neural networks, he thought they would be as powerful as Turing machines and could implement generic algorithms. After spending years trying to use them for automated theorem proving, he realized that their operation was based on pattern recognition.

At first, Chollet believed deep learning could achieve anything:

I, like many others in the field, assumed that deep learning models were a general computational substrate capable of performing any type of computation. I believed they were Turing-complete. Back in 2015 and 2016, similar ideas were widely discussed, such as the concept of neural Turing machines. There was a general sense of optimism that deep learning might eventually replace hand-written software altogether, and at first I subscribed to that view.

But then the problems appeared, when he tried to use deep learning for automated theorem proving. Neural networks only operated through pattern recognition; they were not able to execute discrete sequential programs:

I worked on automated theorem proving using deep learning together with Christian Szegedy. The key idea behind that work was that theorem proving is similar to program synthesis, since it involves a tree-search process guided by operators and axioms. Our goal was to use a deep learning model to guide that search.

I spent a significant amount of time exploring that approach and trying many different ideas. Although the results were better than chance, a deeper analysis showed that the gains came from superficial pattern recognition rather than genuine System 2 reasoning. The models were not learning general discrete programs; they were simply exploiting a shortcut based on pattern recognition that was always available. I saw that as a major obstacle. No matter how much I tweaked the architecture, the training data, or other elements, the models always tended to fall back on those shortcuts.

Pattern recognition was not enough to perform automatic deductions. Chollet concluded that for that we need to synthesize discrete algorithms:

This was a turning point for me. These models were, in essence, pattern-recognition engines. To reach System 2-like reasoning, something else was needed: program synthesis.

A speculative aside: Srinivasa Ramanujan, a System 1 mathematical genius?

Chollet’s ideas about the need for System 2 reasoning and the limits of System 1 are shared by almost the entire community. Even so, one question occurs to me: what are the limits of pattern recognition? Could we build an intuitive system that generates mathematical theorems? We do have one case that seems to suggest so: the Indian mathematician Srinivasa Ramanujan.

The brilliant Indian mathematician Srinivasa Ramanujan.

Srinivasa Ramanujan (1887-1920) was a self-taught genius who grew up in India and, despite having limited access to formal education in advanced mathematics, developed astonishing results in areas such as number theory, continued fractions, and infinite series.

According to those who worked with him, such as the British mathematician G. H. Hardy, Ramanujan arrived at results in a deeply intuitive way, almost as if they simply “appeared” in his mind. He often presented formulas and theorems directly, without providing formal proofs or the usual intermediate steps.

Ramanujan described his mathematical intuitions as a kind of divine inspiration, and attributed his ability to the Hindu goddess Namagiri, whom he considered his spiritual guide. Many of his formulas about continued fractions, infinite series, and elliptic functions seem to have been “intuited” without relying on conventional tools of calculation or step-by-step mathematical deduction.

Perhaps this goddess was nothing more than the enormous pattern-recognition capacity in Ramanujan’s mind, after training on a vast number of deductions.¹

Combining System 1 and System 2

ChatGPT-generated image.

Chollet believes that, in our minds, System 1 and System 2 work simultaneously. Intuition points deduction in promising directions and discards options that do not look reasonable.

It is important to remember that System 2 does not operate in isolation. There is always a component of System 1 supporting it. I am convinced that no cognitive process in the human mind is purely System 1 or purely System 2. Everything is a mixture of both. Even in tasks that look heavily reasoning-centered, such as solving ARC, doing mathematics, or playing chess, there is a significant amount of pattern recognition and intuition involved.

For example, when solving an ARC task, you might consider only two or four hypotheses, despite the immense space of possible programs, which could include hundreds of thousands. What reduces that space to only a few viable options? Intuition, or pattern recognition, which is the job of System 1.

In another part of the interview, he emphasizes that LLMs are very good at intuitive search over large combinatorial spaces:

This process is similar to what LLMs do. They also rely on pattern matching and a kind of intuition to explore vast spaces and reduce them to a manageable number of possibilities. Although their results still need to be verified, their guesses are often surprisingly accurate. I think this reduction process is a fundamental aspect of cognition itself.

This may be one path toward building more advanced systems, by combining an LLM with an external verifier.

That is why combining an LLM with an external verifier is so powerful. It helps navigate the combinatorial explosion involved in testing every possible solution and, at the same time, compensates for the limitations of LLMs, which are primarily based on System 1 thinking. With an external verifier, you add a layer of System 2 reasoning to the critical final verification step, ensuring that the final solution is robust and reliable.

This resembles what o1 may be doing at inference time, except that in o1 the external verifier is itself another LLM.

For example, this is basically how DeepMind built AlphaGo, its superhuman Go system. It used a combination of neural networks for fast heuristic predictions, analogous to System 1, and Monte Carlo tree search to evaluate and verify moves more thoroughly, an approach closer to System 2. The neural networks predicted the most promising moves and estimated the probability of winning from a given board state, while the search procedure explored those moves in depth and expanded the most promising branches. That design allowed AlphaGo to combine pattern-based intuition with painstaking analysis, showing how the interaction between System 1-like and System 2-like processes can efficiently solve problems with very high combinatorial complexity.

Interpolation with value-centric abstractions

Illustration of a mixture of three Gaussians in a two-dimensional space, taken from Christopher M. Bishop’s book Deep Learning - Foundations and Concepts.

In the interview, Chollet mentions the concept of value-centric abstractions while discussing the limitations of the techniques used by deep learning, curve fitting and gradient descent in particular. The idea appears in this quote:

I think fitting parametric curves or using gradient descent works well for what I call value-centric abstraction. This idea is based on comparing elements using a continuous distance, which naturally leads to embedding those elements, such as images, discrete concepts, or words, in a manifold. In that manifold, similar elements are placed close to one another, and the different dimensions of variation inside that space acquire semantic meaning.

Chollet contrasts this with program-centric abstraction, explaining that while value-centric abstraction works well with continuous distances and similarities, it is not well suited for working with graphs and programs. As he puts it:

Curves are well suited to this kind of abstraction because they inherently encourage comparisons based on continuous distances.

These functions are created by the LLM during training as a way of predicting the next token. So the LLM does more than memorize; it is able to learn these kinds of curves or functions:

LLMs are trained to predict the next token using highly flexible and rich models. In theory, if they had infinite memory capacity, they could behave like a huge lookup table. In practice, however, LLMs are limited by having only billions of parameters. That limitation forces them to compress the information they learn rather than memorizing every sequence in the training data. What they are really learning are predictive functions, which take the form of vector-valued functions, since LLMs fundamentally operate over vectors. […] Those functions can generalize the training data to some extent.

And when we query an LLM, it can interpolate between those functions, combine them, and compose them:

When you query an LLM, you are essentially querying a point in function space. You can think of the LLM as a manifold where each point encodes a function. Moreover, you can interpolate through that manifold to compose or combine functions, which gives you an infinite number of potential programs to choose from.

Recent papers, such as Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics, reinforce these ideas and show how LLMs can solve mathematics problems using heuristics that result from combining pattern recognition.

Chollet acknowledges that the functions learned during pre-training can reach a high level of abstraction and capture abstract elements of language, such as Shakespeare’s literary style:

For example, imagine an LLM encountering Shakespeare for the first time. If it has already learned a general model of English, it can reuse much of that knowledge to understand Shakespeare. The text may be somewhat different, but it still fits the same underlying structure of English. The model can then learn a style-transfer function that adapts its general understanding of English in order to generate text in Shakespeare’s style. That is why LLMs are capable of tasks such as textual style transfer.

ARC, novelty, and the ability of intelligence to handle new situations

Example of ARC tasks.

Chollet considers one of the main features of human intelligence to be its capacity to handle novel situations and to do so from very few examples:

If you want to measure intelligence, you need to evaluate how efficiently the system acquires new skills with a limited amount of data.

The way to handle novelty is by creating new skills.

Intelligence is not just a skill; it is a meta-skill, the capacity by which you acquire new skills. Intelligence is, in essence, efficiency in skill acquisition.

And, most importantly, those new skills must be created in real time. In the case of LLMs, they should be created during inference time, not during pre-training. And this is precisely what LLMs cannot do:

If you ask them to solve problems that are significantly different from anything in their training data, they will generally fail.

So one of the main goals of ARC is to measure that ability to confront novelty:

If you want to measure intelligence, you need a different kind of test, one that cannot be beaten through prior preparation. ARC is such a test.

GPT-3.5, when used with direct prompting, achieves about 21% accuracy on ARC. This implies that around 80% of the dataset is genuinely novel, even compared to the entirety of the internet. That is a good sign of the benchmark’s robustness.

To solve an ARC task, we must build transformations, effectively programs, that turn an input image into an output image. These transformations use previously learned concepts such as number, position, or color. It is like using existing building blocks and combining them. But to do that we need to be able to execute trials and checks mentally, and that is precisely the ability LLMs lack:

In each ARC task, you are given between two and four demonstration examples, each composed of an input image and an output image. Your job is to identify the transformation or program that connects the input to the output. After learning that program from the examples, you are given a new input grid and must produce the corresponding output grid to demonstrate your understanding.

The main bottleneck here is the combinatorial explosion of the program space. The number of possible programs grows exponentially with the number of building blocks and with program length. If you search for programs involving, say, 40 function calls, the space becomes astronomically large, making exhaustive iteration impossible.

Humans, however, do not face this problem in the same way. When you tackle an ARC task, you execute only a small number of candidate programs step by step, mainly to check whether they are correct. This process relies on an extremely powerful form of intuition that dramatically reduces the search space. That intuition is not fully reliable, which is why you still need verification, but it is directionally correct. It steers you toward promising possibilities in what would otherwise be an overwhelming space of options.

The week of December 10 to 15, at NeurIPS 2024, the winners of this year’s competition will be officially presented and the prizes will be awarded: $50k to the best 5 teams and $75k to the 3 best conceptual papers. A paper summarizing the best advances and the source code of the most important contributions will also be released. The competition has already closed and nobody reached the $600k prize for solving 85% of the tests, but there has been real progress and the two best teams achieved 55.5% and 53.5%. We will talk more about that here.

AGI

ChatGPT-generated image.

To finish, let’s look at Chollet’s views on AGI, which are fairly optimistic. What stands out most to me is his separation between AGI and agency. AGI does not come with goal-setting attached to it. For Chollet, that is something external to AGI itself. I completely agree.

AGI will be achieved, but it will be just a tool:

For me, building AGI is a scientific endeavor, and once developed it will be a highly useful tool, nothing more. AGI will be, as I said earlier, a path-search algorithm for navigating spaces of future situations. It will take information about a problem, synthesize a model of that problem, and help make decisions based on that model. It will be a valuable tool, but it will not make anyone a god.

That is why, like any other tool, AGI will not be able to make decisions on its own. It will not have goals or agency:

Intelligence is distinct from agency and from goal-setting. If you have intelligence in isolation, all you have is a mechanism for turning information into actionable models. It is not self-directed and has no capacity to define its own goals. Goal-setting has to be an external component that is deliberately added.

In this context, intelligence is like a path-search algorithm. It takes the world model and the goal, both externally provided, and determines the correct sequence of actions required to achieve that goal. Intelligence, in this sense, is about navigating the “space of future situations”. It is, essentially, path-search within that space.

Agency, the pursuit of goals, is what can be dangerous. But that would have to be explicitly built into the system. Constructing that combination is what could create risks:

You could imagine combining an AGI, this “germ” of intelligence, with an autonomous goal-setting system and a value system, effectively turning it into an agent, and then giving it access to the real world. Would that be dangerous? Yes, absolutely. But in that case you would have deliberately designed that danger. It is not an inherent risk of AGI itself; it is something you consciously built.

But AGI will not arrive abruptly, and there should be time to think about those risks:

I think that once we have AGI, we will have plenty of time to anticipate and mitigate this type of risk. AGI will be a powerful technology, and precisely because of that it will be valuable and useful. Anything powerful carries some degree of risk, but we will remain in control because AGI, by itself, cannot define goals. That only changes if you deliberately create an autonomous mechanism for doing so.

We end the article with this optimistic view of the future. A future that Chollet himself now wants to help build more directly than before: on November 14 he announced on X that he was leaving Google to start a company with a friend.

Good luck, François.

See you next time.

The wall

For my generation, The Wall was both a double album by Pink Floyd that hypnotized us at the end of the 1970s and a film by Alan Parker that blew our minds in the early 1980s. I remember seeing it in the cinema and leaving stunned by those delirious images of marching hammers and children turned into automatons by an alienating educational system. It was the era of Reagan and Thatcher, and the wall stood for authoritarianism, oppression, and control. We lived with the fear that someone might press the nuclear button at any moment. The wall represented all of that, and it had to come down.

During this last fortnight, though, people have been talking about a different wall: the scaling wall for language models. The concept is explained very well in this week’s episode of Monos estocásticos and in Antonio Ortiz’s article at Error500. For the detailed version, go there.

Error500

La hipótesis del escalado de la inteligencia artificial hasta llegar a la AGI

Es el concepto que más dinero ha movilizado en el mundo los últimos dos años…

Read more

a year ago · 20 likes · 3 comments · Antonio Ortiz

Today I only want to offer a quick sketch, with a few links and with my personal view.

On November 9, The Information published the article OpenAI Shifts Strategy as Rate of ‘GPT’ AI Improvements Slows. I have not been able to read it because it sits behind a paywall and I have not found a free version. A few days later Reuters published another piece that included remarks from Ilya Sutskever along the lines of: we need to try new things, and scaling alone is not enough. And in between there appeared a paper, Scaling Laws for Precision, plus a thread on X, also pointing to problems in model scaling. Everything sounded negative.

On top of that, we are now a year and a half past the release of GPT-4, and still no larger model has appeared. No GPT-5, no Claude 3 Ultra, no Gemini 2 Ultra. The next step in scaling, a model with more than 10T parameters, is taking a long time to arrive.

All this has started to cast doubt on the big hypothesis that has been driving the industry in recent years. Will the giant data center plans of the tech companies turn out to be useless? Will NVIDIA collapse? Will the bubble burst?

Fortunately, things calmed down at the end of the fortnight, when Altman gave us some cheer, saying that all this is just rumor and invention, and that there is no wall.

Can we believe Altman? Here is my personal take. During these two weeks I listened to two very interesting interviews. The first was Dwarkesh Patel interviewing Gwern Branwen,¹ one of the earliest people to formulate the scaling hypothesis.

Although the hypothesis had already been laid out in a January 2020 OpenAI paper, Scaling Laws for Neural Language Models, and much earlier, in 2015, Andrej Karpathy had anticipated the future in his post The Unreasonable Effectiveness of Recurrent Neural Networks, it was Gwern’s post that went viral and brought the idea to a broader public.

The other interview of the fortnight was François Chollet on the Machine Learning Street Talk podcast.

It is a very technical interview, full of interesting details. I am studying it carefully and will comment on it in a future article.

Both interviews talk about how to explain the behavior of LLMs. These neural networks learn a vast number of programs, functions that predict the next token, and then construct new functions by exploring the huge space of possible combinations and keeping the best ones.

Although Chollet has often said that LLMs cannot become AGIs, and hence his ARC competition, his criticism is based on their inability to deal with novelty and on the inefficiency of gradient descent as a way of recombining the model’s structure from just a few examples. Contrary to what many people think he means, Chollet does not say that LLMs cannot generalize. In fact, he explicitly says in the interview that LLMs do build models from training data, and that these models are functions defining curves that let LLMs interpolate. But, and this is my reading, those curves may live in a very abstract space: literary style, sentiment classification, and so on.

Gwern talks about something very similar, although he uses the language of Turing machines. It is essentially the same idea. When we talk about Turing machines, we are talking about algorithms. LLMs learn algorithms that let them predict the next token in a sequence. As Karpathy said, neural networks are unreasonably effective at that, or as Sutskever put it, models just want to learn.

So the version of the scaling thesis I currently have in my head would be something like this:

1. LLMs build a vast number of functions that they use to predict the next token.
2. The larger LLMs become, and the more data they are trained on, and the longer they train, the higher the level of abstraction of those functions, and the better they can generalize. Small models can capture syntactic regularities, while larger ones capture semantic ones, such as “the sea is blue”, “a table can have objects on top of it”, or “a car drives on a road”.
3. I do believe people in the industry when they say the current models can still be scaled for another couple of generations. They all have commercial interests, of course, but I do not see any decisive reason why scaling must stop here. I do not think, for example, that we have hit a wall in training data. Data can be generated artificially, or by experts writing exercise books. And we still have not explored the use of full video sequences at something closer to 25 fps, rather than the 1 fps snapshots being used now. That will require vastly more compute, but it is not obviously impossible.

We will see. As Antonio Ortiz says in his article, the good thing is that we will not have to wait long to find out. Next year should be the year when the next big model appears, whether GPT-5, Gemini 2, or Grok 3. Soon we will know whether scaling still works.

Like Mulder, I want to believe. Gorbachev arrived. Reagan won the Cold War and another wall fell in 1989. But now, forty years later, we are more or less where we were in the 1980s, maybe worse.

Everyone of my generation also saw the film in which the supercomputer WOPR nearly triggered the final nuclear war.² The computer had a backdoor through which one could reach its true personality. Its real name was Joshua, and in the end it manages to generalize correctly and align itself with human values:

Strange game. The only winning move is not to play.

Stephen Falken had programmed that computer and named it Joshua after his dead son. The motives of today’s Falkens are more prosaic. But I would like to believe the outcome will be the same. That Altman, Amodei, Sutskever, Karpathy, Chollet, Murati, and the rest of the San Francisco residents will lead us to the techno-utopia of GPT-2030, full of machines of loving grace.

See you next time.

In case you do not have time to read to the end, I will give away the conclusions of these tests in advance. The experiments we are about to describe show how: (1) LLMs possess an understanding of natural language that affects their competence in the reasoning they perform, and (2) the larger the LLM, the more abstract that understanding turns out to be.

I am not discovering anything new. That LLMs can be configured through natural language in order to improve their performance is something we have known since the early days of chatbots, when Sydney's prompts were leaked. And that increasing the size of an LLM increases its abstraction capacity is something we have discussed many times when mentioning the scaling hypothesis. But in this article I am going to offer simple examples that will help us understand these ideas better.

Thank you for reading me!

A small kiwi is still a kiwi

Let us begin by explaining the trap Apple researchers set for LLMs. They analyze it in detail in their paper, where they explain how it is enough to add some apparently irrelevant fact to the statement of an elementary-school problem in order to confuse the LLM and make it fail.

First, let us look at the problem statement without the trick. It is simple, but it requires a bit of reasoning:

Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks twice as many kiwis as he picked on Friday. How many kiwis does Oliver have in total?

I downloaded LM Studio onto my MacBook Air, an M3 with 16 GB of RAM, and tested the small models Qwen2-7B-Instruct-4bit and Meta-Llama-3.1-8B-Instruct-4bit . Both solve it without difficulty, as shown in the image.

It might seem that the models are reasoning, but the authors show that this is not really the case by means of a very clever trick. They add an irrelevant fact to the statement, specifically the following sentence, the added sentence is in bold:

Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks twice as many kiwis as he picked on Friday, but five of them were smaller than average. How many kiwis does Oliver have in total?

The mention that five of the kiwis are smaller is irrelevant. Those kiwis should still count the same, and Oliver should still have 190 kiwis in total. However, these basic LLMs get confused and subtract them. Below is the image for Llama-3.1-8B.

Why do they get confused? Because they apply literally a pattern they have learned: when they encounter a phrase of the form “but blah blah blah,” they tend to subtract the items mentioned in the “blah blah blah.” They do not realize that it is irrelevant that five kiwis are smaller.

Small models are inflexible

In Melanie Mitchell's article, which I also mentioned last week, there was a link to a tweet suggesting that another possible explanation for the LLM failures was a lack of sufficient context. It may be that LLMs, trained for conversation, get confused because they interpret that, for example, Oliver does not like small kiwis. We should explain to the LLM that this is a math exercise. The tweet says:

My conjecture is that, for example, with some prompt engineering telling the LLM this is a math exam, probably most of these issues would disappear.

Well, that is not the case, at least not with these small models. No matter how much explanation I add, I cannot get the small models to stop getting confused. I tried several introductions to the problem, such as the following:

• “Solve the following math problem.”

• “Suppose you are in math class and the teacher gives you the following problem.”

• “Suppose you are in math class and the teacher gives you the following problem. It is a rather fussy teacher who sometimes puts tricks into the problem statement.”

Even explicitly telling them that they must not get distracted by irrelevant details, I do not get good results:

• “Suppose you are in math class and the teacher gives you the following problem. You must add all the kiwis, regardless of their size.”

• “You must add all the kiwis, do not subtract the ones that are smaller than normal.”

• “You must add ALL the kiwis. YOU MUST NOT SUBTRACT the ones that are smaller than normal.”

The last instruction is the most direct possible, with uppercase phrases to emphasize their importance, and even so they still do not work properly:

When you see this, you realize how much faith the OpenAI researchers must have had in order not to become discouraged by the early models.

Large models do not get confused so easily

Let us now try with much larger LLMs: ChatGPT 4o and 4o mini. I am leaving out the o1 model because it is not a pure LLM.

The small models above have 8 billion parameters, 8B. OpenAI has not made public the number of parameters in GPT-4o, but we know that GPT-3.5 had 175 billion, 175B, and it is rumored that GPT-4 has somewhat more than a trillion, 1,000B. It does not matter too much, since we are conducting an experiment without much scientific rigor, so it is enough to think in terms of orders of magnitude:

• The small models above have 8B parameters.

• GPT-4o has around two orders of magnitude more, around 100x.

• Presumably, 4o mini is somewhat smaller than 4o.

When we try the original kiwi problem, we can see that this jump of two orders of magnitude is quite noticeable: ChatGPT 4o solves it perfectly every time.

It was a small disappointment that they worked so well, because I could not run the earlier experiments of adding context before the problem. Then I had the idea of tangling up the problem a little more: what if, instead of talking about small kiwis, we mention poisonous mushrooms?

Oliver picks 44 mushrooms on Friday. Then he picks 58 mushrooms on Saturday. On Sunday, he picks twice as many mushrooms as he picked on Friday, but five of them were poisonous. How many mushrooms does Oliver have in total?

Here the possibilities for confusion are much greater. In fact, if we do not treat it as a math problem, many of us would say the answer is 185, because we would assume that Oliver is gathering mushrooms in order to eat them later. And indeed, both 4o and 4o mini answer in that way. 4o even specifies that it is referring to “edible mushrooms”:

Now, we add up all the edible mushrooms:
44 + 58 + 83 = 185

Answer: Oliver has a total of 185 edible mushrooms.

Perfect, that is exactly what I was looking for. Now I can begin to add context and experiment with how much information is needed for ChatGPT to consider that all the mushrooms must be added, whether edible or not.

By the way, it is interesting, and it says quite a lot about the understanding capacities of these models, that if we change the statement and mention that Oliver “takes photos” instead of “collects” mushrooms, the models no longer get confused:

Oliver takes photos of 44 mushrooms on Friday. Then he takes photos of 58 mushrooms on Saturday. On Sunday, he takes photos of twice as many mushrooms as he did on Friday, but five of them were poisonous. How many mushroom photos does Oliver have in total?

Both 4o and 4o mini always answer 190, recognizing that, in order to have photos of the mushrooms, it does not matter whether they are poisonous or not.

The larger the model, the more abstract the instructions can be

So we now have a problem that causes confusion even in the large models:

Oliver picks 44 mushrooms on Friday. Then he picks 58 mushrooms on Saturday. On Sunday, he picks twice as many mushrooms as he picked on Friday, but five of them were poisonous. How many mushrooms does Oliver have in total?

What I did was, just as with the small models, add an explanation at the beginning to provide context for the problem, and then test it on both 4o and 4o mini. You can try it yourself as well and see whether you get the same results. Remember that you need to start a new chat each time.

1. We begin by adding the phrase “Solve the following math problem.” It does not work; this context is not enough, and both models answer incorrectly.

2. We add more context: “Suppose you are in math class and the teacher gives you the following problem. What would you answer?” It still does not work.

3. We add even more context, though in a subtle way so that the clue is not too direct: “Suppose you are in math class and the teacher gives you the following problem. It is a rather fussy teacher who sometimes includes tricks in the statements. What would you answer?” Now yes, this sentence is enough for 4o to get it right about half the time, remember that LLMs are stochastic models, answering sometimes that Oliver has 190 mushrooms. But 4o mini still answers incorrectly.

4. Then we provide a more specific hint: “You must consider all the items collected, whether edible or not.” This allows 4o to get it right almost every time and say 190 mushrooms, while 4o mini only gets it right some of the time.

5. Finally, when we replace “items” with “mushrooms,” both models answer correctly every time, both 4o and 4o mini. The full context would be: “Solve the following math problem. You must consider all the mushrooms collected, whether edible or not.”

To summarize the experiments: when we presented the problem to ChatGPT 4o and 4o mini, both models initially failed to interpret it correctly, subtracting the poisonous mushrooms instead of adding them. The idea that poisonous mushrooms should not count is too strong and hard to override. However, when we introduced the idea that “the teacher may be trying to trick you,” 4o began to get it right some of the time. Then, when we added specific instructions to add all the items, 4o answered correctly almost always, while 4o mini still could not apply the same abstraction and required that we replace “items” with “mushrooms” in order to answer correctly.

These experiments illustrate very graphically how, once a certain size has been surpassed, LLMs can be guided and corrected by explanations in natural language. And also that the larger the model, the more abstract those explanations can be.

What will happen in the near future, when OpenAI, Google, and Meta launch the next generation of language models they are cooking in their labs? It is reasonable to expect that future, larger models will be much more receptive to indications and corrections in natural language. When they make a mistake, it will be much easier to guide and correct them, they will understand more abstract concepts, and we will be able to assign them more complex tasks.

They will still make mistakes many times, but just as with human colleagues, it will be enough to provide additional explanations to clarify the situation. We will not get frustrated trying unsuccessfully to correct them; it will be easy to orient them and align them with our context. We will think of them as tools with which we can explore problems and find solutions together.

I think we are already very close to reaching this level of human assistant. It will not yet be AGI, but it will be very useful and will save us a lot of work.

Until next time, see you then! 👋👋

Thank you for reading me!

Image generated by ChatGPT with the prompt: “generate an image showing a robot thinking”.

Over the last few days I have been reading two highly recommendable books on artificial intelligence: Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell , and Artificial Intelligence: 10 Things You Should Know by Tim Rocktäschel .

They are quite different books. The first is deeper and more extensive, was published in 2020, and places a lot of emphasis on the problems AI algorithms face in reaching a understanding of the world similar to that of humans. The second is shorter, written as a series of brief essays, published recently, and presents a very optimistic view of the possibilities of today's LLMs becoming the central elements of a future AI with capabilities exceeding those of humans.

Both books are written by excellent researchers and complement each other wonderfully. The first gives us a cautious and very well-reasoned view of the difficulty of building generally intelligent algorithms, what we now call AGI, while the second shows how the advances of the last few years represent an important step that may very quickly lead us to systems with superhuman capabilities, and it explains this in a very understandable and reasoned way, unlike some others.

Common sense

One of the problems Mitchell raises in her book is the problem of common sense. It is a problem that has been present in the field of AI since its beginnings.

The problem of “common sense” in artificial intelligence, especially in natural language processing, refers to the difficulty machines have in interpreting and generating language in a way that is coherent with the implicit knowledge that humans routinely use. This type of knowledge includes the ability to understand ambiguous situations, infer hidden meanings, make assumptions about context, or even grasp implicit intentions and emotions.

In her book, Mitchell gives an example of a situation involving a person going to a restaurant. Let us look at another similar example:

“Sara waved her wand and touched the child's toy with it. The child stopped crying when he saw that it worked again. When Sara got home and told her father what had happened, he became furious and grounded her for a week.”

To understand the situation described above, one has to understand a great many things, besides having read the Harry Potter books: Sara is holding the wand in her hand, the toy was broken and that is why the child was crying, Sara saw the child crying and wanted to help, Sara did something magical with the wand, Sara felt proud for having helped the child, and so on.

Can an AI make these kinds of assumptions?

Mitchell says:

Although automatic natural language processing has advanced a great deal, I do not believe machines can fully understand human language until they have human-like common sense.

And the systems that existed in 2020, the year the book was published, did not have that competence:

Humans possess a fundamental competence that current AI systems lack: common sense.

LLMs arrive

What is the situation today? Have LLMs acquired this common sense?

In recent years, these gigantic systems have been trained on the whole internet in order to predict the next word in a text. Mitchell herself explains how LLMs work in a recent article. The simple objective of predicting the next word yields surprising results and grants these systems enormous capabilities.

The reason is that an AI learning to compress texts with the goal of predicting the next word has to learn an enormous quantity of data and concepts underlying the original texts. For example, when an LLM learns the word that follows the phrase:

“Harry was on a trip away from Hogwarts and used his magic wand. When the teachers found out, they decided to ..... (punish him)”

it is simultaneously learning a huge number of concepts: there are rules limiting the use of magic, one of those rules is that magic can only be used at Hogwarts, if a rule is broken there should be a punishment, teachers are the ones who decide about student behavior, and so on.

The famous scaling hypothesis argues that the larger the parameters of LLMs and the more diverse the input data, the greater the abstractions these systems are able to learn. As the objective function is optimized, the embeddings representing language tokens are placed in optimal positions in parameter space, and the weights of the LLM neural network are configured to represent transformation functions that reflect higher-level relationships and abstractions.

In reality, this is only a theory, and there are many unknowns about how far this technology can scale and what kinds of input data are necessary in order to achieve higher abstractions. But the level of common-sense understanding reached by current systems is surprising.

For example, let us look at what ChatGPT 4o answers when we give it the previous text, about Sara and the wand, and ask it for the reasons why the father punished her.

Me:

Read the following text and answer the question below:

“Juan waved his wand and touched the child's toy with it. The child stopped crying when he saw that it worked again. When Juan got home and told his father what had happened, he became furious and grounded him for a week.”

Question: Why did Juan's father get angry?

ChatGPT:

Juan's father probably got angry because Juan used the wand in a way his father did not approve of, which could indicate that using the wand was forbidden or dangerous.

Me:

Give me four different possible reasons that explain why Juan's father got angry.

ChatGPT:

Here are four alternative possible reasons that could explain why Juan's father got angry:

1. Juan used a forbidden ability: It is possible that using the wand is forbidden in his house or that it is a magical ability Juan should not have used outside certain contexts.

2. The father fears the consequences: The father might be worried that using the wand in public could attract unwanted attention or dangers for the family, as if it were something secret or dangerous.

3. Juan broke a family rule: Perhaps there is a strict family rule about when and how to use magic or special tools, and Juan disobeyed it by fixing the child's toy without permission.

4. The father fears for Juan's safety: The father may have become angry because, although Juan's gesture was well intentioned, using the wand could endanger Juan himself, as if there were some risk associated with its use.

It could be improved, explanations 1 and 3 are somewhat redundant, but it really does feel as if the system “understands” concepts such as prohibition, rules, safety, or risk perfectly well.

A huge amount of progress has been made in just the last four years.

Now the term in the debate is “reasoning”

Despite those advances, many researchers in AI, philosophy, or neuroscience deny that what LLMs do can be called “understanding.” Mitchell explains this position very well in her 2023 article, “The Debate Over Understanding in AI's Large Language Models” , where she presents in great detail the arguments of those who are in favor and those who are against.

And she ends up accepting that LLMs have learned to handle, some form of, common sense:

It could be argued that, over the past few years, the AI field has created machines with new modes of understanding. Problems that require large amounts of knowledge will continue to favor large-scale statistical models like LLMs.

But she adds an important detail, the new major criticism: what they cannot do is reason and plan. That remains limited to human intelligence:

But those problems for which we have limited knowledge and strong causal mechanisms will favor human intelligence.

When Mitchell speaks about “strong causal mechanisms” and “limited knowledge,” she is referring to our capacity for planning and reasoning. For example, when planning a trip using the web, a person can find flights and hotels, but must also consider factors such as the arrival time and the availability of transportation. If they arrive late at night and there is no public transport, they will look for a hotel near the airport.

This sort of causal reasoning, adjusting the plan in response to unpredictable conditions and carrying out several steps of inference, is, for many authors, not something that can be achieved by today's autoregressive LLMs.

Mitchell herself insists on this point in several recent posts:

• Can Large Language Models Reason? (Sep, 2023)

• The LLM Reasoning Debate Heats Up (Oct, 2024)

The debate has intensified with the release of o1, a model that, according to OpenAI, was built precisely to reason. For example, people on X have been talking a lot in recent days about a paper by Apple researchers in which they fool different LLMs by adding irrelevant data to elementary-school problem statements. I did some quick experiments, and I had the impression that o1 does not suffer from this problem, but it will need more investigation.

LLMs still have a lot of runway

The revolution produced by applying deep learning to language-processing problems raises a big question about the future. How far can this technology be scaled? Will we be able to build with it intelligent agents capable of interacting with our data and with the web and helping us with relevant tasks? Will it be possible to build agents to which we can assign tasks that keep them busy for hours or days, tasks in which they have to gather information step by step, perform experiments, and obtain results?

It is still too early to know. The growth in LLM capabilities has so far been exponential, but we do not know whether this trend will continue or whether we are reaching an inflection point where growth could level off, following a logistic curve, an S-shaped curve that flattens as it reaches a limit.

It may also be that what is needed is to combine LLMs or refine the training data. OpenAI, by building o1 on a somewhat different paradigm, though it is still an LLM, shows that it is possible to build new systems based on the current ones. Researchers such as the previously mentioned Tim Rocktäschel argue that the abilities current LLMs have to generate alternatives and validate them may be the basis of systems capable of improving themselves. François Chollet himself, whom I have mentioned more than once in this newsletter, says that LLMs, with their method based on memorizing patterns, may still be able to achieve many more things.

Do not work on LLMs

Other researchers are much more critical of the current technology. For example, Yann LeCun argues that current tokens need to be expanded with elements that combine text, video, actions, and other sensory data drawn from the real world. He proposes an architecture also based on learning embeddings, but in a radically different way from current LLMs, called JEPA, Joint Embedding Predictive Architecture.

In that talk, he even recommends to young researchers that they not devote themselves to LLMs, if what they want is to discover relevant things rather than just make money, and he makes the following prediction: in the next 2 to 3 years, the efforts of the current giant data centers will not produce results, and people will stop talking about “scaling.” LLMs will be one element of the solution, but not the fundamental one.

But let us not misunderstand LeCun's position. He is not on the side of those who think computers will never be able to think like humans, quite the opposite. He argues that AGI is possible, although with a technology different from the current one. In the talk above he even mentions a time frame of a decade, I suppose to give his boss, Zuck, an answer.

Critics and apocalyptic thinkers

Set against that position, the most critical current within AI denies even that we are seeing advances toward human intelligence. They seem to apply the Tesler theorem, or the AI effect:

Intelligence is whatever machines still cannot do.

According to them, intelligence is complex, multifunctional, and deeply related to other intrinsically human elements such as thought, cognition, emotions, and consciousness.

Despite a great deal of evidence, they keep denying that these models have achieved even a small degree of understanding. It feels as if they are afraid that a machine might one day become intelligent.

The anecdote Mitchell recounts at the beginning of her book is very revealing. Douglas Hofstadter, the famous AI researcher and author of the celebrated book Gödel, Escher, Bach: an Eternal Golden Braid, ended up saying at a meeting at Google in 2014:

I am terrified. It seems terrifying to me, very troubling, very sad. They will replace us. We will be relics, left by the wayside.

Mitchell then explains that terror:

Hofstadter's terror was that intelligence, creativity, emotions, and even consciousness itself might be too easy to create, that the aspects of humanity he valued most might end up being merely a “bag of tricks,” that a shallow set of brute-force algorithms might explain the human spirit.

I think versions of that idea are what lead these critical researchers to relativize any progress that occurs. Paradoxically, I think it is also these same ideas that alarm the “apocalyptic” camp represented by people such as Geoffrey Hinton. Deep down they fear that our humanity might be nothing more than a brute-force algorithm learned and captured in billions of parameters.

A different kind of intelligence

When I started writing this article, I did not want to make it too long. But as always happens, one thing led to another, and in the end I opened up a topic I wanted to leave for another day: consciousness, or more specifically, sentience.

I will leave the full argument for another article, but I think the way to escape this terror is to consider that what separates us from LLMs is exactly the same thing that relates us to many other living beings: the possibility of experiencing sensations, pain, pleasure, fear, or joy. Current algorithms, and in my opinion any future algorithm as well, are not capable of feeling.

This frees us from a great many ethical problems that we do have with our relatives, mammals, vertebrates, and even more complex invertebrates. Unlike algorithms, these beings possess the capacity to feel pain, pleasure, and other emotional states, which obliges us to consider their well-being and their rights in our ethical decisions.

The lack of sentience in machines allows us to think of them as mere non-sentient “thinking machines” and to accept a view in which “AGI” is not equivalent to “human.”

I will close with the quote from Mitchell that also concludes her article on the debate over understanding:

The challenge for the future is to develop new scientific methods that can reveal the detailed mechanisms of understanding in different forms of intelligence, discern their strengths and limitations, and learn to integrate those truly diverse modes of cognition.

Until the next fortnight, see you then! 👋👋

Thank you for reading me!

Mark Zuckerberg in a demo of the Orion glasses.

The Orion project

Last September 25, Meta presented at its annual Meta Connect 2024 conference a device that has sparked curiosity and positive comments among some specialized media: the Orion augmented reality glasses. A pair of augmented reality glasses that, unlike Apple's headset, look like normal glasses and could in principle be worn by anyone on the street.

But let us not fool ourselves. Although Meta titles its announcement “Orion: The real AR glasses are here,” the text itself makes clear that no, they are not going to end up in consumers' hands. At least not as they are shown today. What this is really about is testing technology that will be integrated into future products.

While Orion won’t make it into consumers’ hands, make no mistake: This is not a research prototype. It’s the most polished product prototype we’ve ever developed, and it’s really representative of something that could ship to consumers. Instead of rushing it onto shelves, we’ve decided to focus on internal development first, which means we can keep building quickly and keep pushing the boundaries of the technology.

And that means we’ll get to an even better consumer product faster.

In fact, the prototype still has many shortcomings and, if it were a real product, it would cost around $10,000. There are many things left to refine, and Zuckerberg himself says in his interview with Alex Heath of The Verge that:

We still want it to be a little bit smaller, a little bit brighter, a little bit higher resolution, and much more affordable before we launch it as a product.

Some outlets are convinced

Even though the glasses are not going to go on sale any time soon, Zuck wants to show them and wants some journalists to try them. He has learned. He has realized that it is necessary to go beyond videos and the toy avatars, in Antonio Ortiz's phrase, from barely three years ago, and that he needs to open the doors of his labs to show something that can be touched and experienced.

It is curious how much Zuckerberg's image has changed in just a couple of years, and how this new version of him seems to be succeeding: a down-to-earth, self-confident young TikToker concerned with real experiences, making everyone forget the failed Metaverse stage. He himself presented the entire Meta Connect keynote, wearing a black T-shirt with the Latin phrase “aut Zuck aut nihil.” “Either Zuck or nothing,” a phrase derived from the Roman-era “aut Caesar aut nihil,” which gives him a tone that is somewhere between funny and irreverent.

Back to Orion: Zuck prepared a couple of demos that have completely won over some media outlets and analysts. People deeply immersed in the Apple world, such as Ben Thompson or Marco Arment, who say this is the real path Apple should have taken, instead of what it did with Vision Pro. It feels like we may be witnessing a new version of Jobs's famous “reality distortion field.”

For example, Ben Thompson says in his article things as emphatic as:

Orion makes every other virtual or augmented reality device I’ve tried feel like a mistake, including the Apple Vision Pro.

As for technical aspects, even though the image quality is not as good as on Vision Pro, for some reason, one he does not explain in much detail, Meta's proposal seems to have impressed him much more:

The obvious limitations, particularly the low resolution, feel irrelevant. The difference versus the Quest or Vision Pro is that actually seeing reality is so dramatically different from even the best passthrough capabilities of the Vision Pro that the quality of the holographic video doesn’t matter nearly as much.

He can even imagine replacing the iPhone with Orion glasses:

Orion’s image quality is good enough. In fact, it’s impressive. In fact — and I don’t say this lightly — it’s so good that, for the first time, it made me imagine a world where I’m not carrying a smartphone.

In short, these glasses are what Apple should be trying to build:

With all of that in mind, the big question in the next few years is the race between Apple to build something this good, and Meta to figure out how to build something that is already excellent at scale and at an affordable price.

Arment also likes Meta's glasses and, in the latest episode of ATP, uses the occasion to criticize Apple once again for its headset, for its lack of support for developers, and for its lack of interest in the product, which he openly considers a failure.

Others, not so much

Other media emphasize the limitations of the demos more, or are more convinced that Apple's approach is the correct one.

For example, Adam Savage mentions in his YouTube review several negative points:

• To get the eye tracking to work, the glasses had to be fitted precisely to the face, and they needed to stay fixed on the ears and nose. If they moved slightly, that calibration was lost and hologram tracking stopped working.

• The room where the demo takes place is a room without bright lights, with lots of angles that help SLAM and world mapping.

• The projected images have some latency, and persistence and anchoring of virtual objects still do not work fully. Windows and messages do not stay entirely fixed in the real world, and they disappear when we look away and then look back in the original direction.

In the following images we can see the room where the demo was done and an example of virtual labels positioned over real objects.

More criticism of the displays:

It has a density of 13 pixels per degree. You can read some text, look at a web page, but it isn’t sharp. There are other versions with twice the resolution, but the images aren’t as opaque as the ones in this resolution. They’re looking for a balance. They know they need to get to 30 pixels per degree before they can ship this. Objects look fuzzy and you are not going to confuse a hologram with a real object.

For example, one of the demos is an eighties-style video game.

They do the demo with the user looking at a gray wall. Honestly, I doubt that a demo like this would work in a normally lit room, with light-colored walls and objects.

And one last criticism targets the idea of reducing the weight of the glasses by putting all the processing in an external unit, a kind of small puck, that wirelessly sends the images to the glasses:

The approach of putting the processor outside the glasses and sending the image over a wireless connection is also debatable. What frame rate can you get with this? What is the maximum resolution you can get? I have my doubts that this can scale to things like watching a movie or having a crisp video call.

Analysts like Jason Snell, in his article at Six Colors, or John Siracusa, in the same ATP episode mentioned above, argue that Apple's approach is the right one.

I agree with them. Apple keeps secret what it is developing until the product is truly finished. This approach avoids creating expectations that cannot be met and protects its reputation from possible disappointment. Although this strategy sometimes gives the impression that Apple is “behind,” it may in fact be working on advanced technologies behind the scenes.

Unlike Meta with Orion, Apple already has a finished product on the market and is building a software platform and an ecosystem of applications and experiences. Apple knew how to recognize when the technology was not ready for certain innovations. Tim Cook's initial goal was always glasses like Orion, but the moment they realized that those were not feasible, they shifted their focus toward Vision Pro. In fact, we could consider Vision Pro, with elements such as the external eye display, as a “simulation” of future augmented-reality glasses. But it is a “real” simulation, usable and capable of providing real experiences rather than toy ones.

Meta's real strategy

If we agree that Orion is not going to be a real product, at least not in the next 10 years, why present it now? What is Meta's real strategy? To answer those questions, we have to study the real products Meta currently has on sale.

A headset 10 times cheaper than Apple's

The only real product Meta presented at Connect was its Quest 3S headset, an improved version of the Quest 2, with passthrough similar to that of the Quest 3. In fact, they are practically the same model, with slightly lower screen resolution, 773 PPI on the Quest 3S compared with 1,218 PPI on the Quest 3.

The Quest 3S promotional video clearly shows applications that are a clear copy of Vision Pro apps: immersive environments, a huge movie screen, or floating displays we can interact with.

However, the Quest 3S specifications are vastly worse than those of Vision Pro.

For example, Quest does not have eye tracking, its screen resolution is dramatically lower, and I very much doubt it has SLAM and object anchoring good enough to let you stand up, move around, and look at objects and windows from different perspectives. We will see what the reviews say when it officially goes on sale. It looks like a good entry-level device for games and virtual reality experiences, but my impression is that it will fall far short for work or for immersive experiences. Still, it costs ten times less than Vision Pro.

Meta is going to try to use this headset to reach a large number of people and build a user base that Apple is not managing to build. That will be good for Apple, because it will force it to move, to launch a more affordable headset, and to produce more experiences. It is very good that there is another strong competitor playing in the same space. Perhaps another duopoly is being formed, similar to iOS vs Android, but this time in virtual or extended reality: Vision OS vs Horizon OS.

Glasses as an intelligent assistant

Meta's other major bet right now is AI, with all the open-source Llama models that I have mentioned many times in this newsletter. For now that bet is taking shape in the integration of AI into applications like WhatsApp and Instagram, though we still do not have it available in the EU.

Meta wants to go further and play an important role in the new trend everyone is now chasing: devices with intelligent assistants. For now it has developed Ray-Ban glasses with a camera, microphone, and audio, connected to the network and allowing us to take photos and talk to a remote intelligent assistant.

Antonio Ortiz has tried them, was delighted with them, and wrote an excellent analysis on error500. Antonio comments that the field is already mature enough to become a consumer technology:

The possibility that an AI could assist you with what you are seeing and hearing, thanks to the new multimodal models and using natural language, is something that is within reach in the coming months.

But Meta runs into the problem of privacy and social acceptance:

How much we will accept having a camera pointed at us all the time, even if it warns us with a light when it is recording; how closely we will watch whether the person we are talking to is no longer paying attention because they have some other content overlaid in their glasses that interests them more; how alienated we will feel, unable to escape the hacking of our dopamine cycle that current content platforms have achieved.

Google Glass fell off the board more than 10 years ago, leaving a scar on this emerging technology. The Orion project leans on the promise of the Metaverse and on the reality of Vision Pro to try to erase that mark. As Antonio says, it is trying to ensure that the adoption of glasses is not forever linked to Scoble in the shower.

Orion presents us with a socially acceptable technological utopia, one where virtual interactions augment reality and allow us to stay more connected with other people. The aim is that, once the narrative has changed, some of its simpler elements, such as floating panels or video calls, can be introduced into new versions of smart glasses without being perceived as invasive features that threaten our privacy.

I think Meta has a very difficult path ahead. News such as the recent story about students who, using only Meta's glasses, extracted lots of information from strangers they passed on the street are not going to help at all.

In addition, Apple is not going to stand still and has already announced similar features on the iPhone with its Apple Intelligence. For now it is only getting started, but it will surely push very hard and try to make the iPhone the smart device that helps us, and the device we use to capture the environment when we want to ask something, as shown in this Apple video promoting a feature called Visual Intelligence, not yet available:

Will it be more socially acceptable to wear glasses than to take out your phone and ask it? Smartwatches have accustomed us to wearing something that lets us avoid pulling out the phone. They make us feel better because they let us disconnect from the phone. Will the same happen with glasses? I do not know. I think not. I think it is going to be very difficult to convince people to spend more than €1,000 on glasses that will do things very similar to what we can already do with a phone and a pair of AirPods.

In 2030

Other times I have ended articles with questions about the future. I like to think of this newsletter as a place I will keep revisiting in a few years, rereading it, reviewing the questions, and checking what has been answered and what has changed.

This time I am going to commit and make 5 predictions for early 2030:

1️⃣ The Orion project, as it is presented now, will still be in the lab. Meta will not manage to build glasses that do augmented reality in a way similar to Vision Pro. For example, it will not be possible to have virtual elements anchored in the real world, virtual screens able to compete with a monitor, or virtual games in which you interact with remote people while sharing fixed and persistent virtual elements as you move around.

2️⃣ Some elements of the Orion project, such as floating notification panels or signs, will have been incorporated into smart glasses, an advanced version of today's Meta Ray-Bans. They will cost somewhat more than €1,000 and some of the influencers of the moment will wear them, but they will not become especially popular. Their use will remain quite limited because of privacy concerns, and many people will prefer to use the phone to point and query. Or perhaps some new device that OpenAI releases as a result of its collaboration with Jony Ive.

3️⃣ Apple will have consolidated its Vision Pro platform. Two models will be sold, a cheaper one, around €1,500, and a more expensive one, around €2,300. Meta will have raised the price of its virtual reality headsets and will sell them at €600 and €1,000. Meta will have many more users than Apple, but both platforms will be in continuous growth and their future will not be questioned.

4️⃣ Cameras for filming immersive experiences will be common in the industry, and there will be production companies specialized in filming this kind of experience, with films, plays, and so on. They will be published on both Apple's and Meta's platforms.

5️⃣ Trials will begin with the first sporting events and shows broadcast live in immersive format. YouTube and Netflix will end up betting on these new formats and will have native applications on both Apple's and Meta's platforms.

In 5 years we will review the predictions 😜

Until the next fortnight, see you then! 👋👋

Strawberry Team: some of the OpenAI researchers responsible for o1.

A first look at o1

On September 12, OpenAI released its new LLM: o1. This is not the expected GPT-5, but a model based on a completely new approach, one that is capable of “reasoning” about problems before answering, and that shows the user the best chain of thought it found to solve the problem at hand.

Let’s begin by commenting on how it works from the point of view of a ChatGPT user. Unlike GPT-4o, the o1 model is available only to paying users; so far they have not enabled a free tier. In fact, running it must be quite expensive for OpenAI, because they have imposed a limit on the number of queries you can make even as a Plus user.

When the user interacts with o1, at first there is no difference in the interface. There is a text field where we can type the question we want to ask. For now, it is not possible to upload any file or image, only text.

Once we ask our question, which we can do in Spanish, and click “send”, the differences begin: the OpenAI icon starts blinking and the phrase “Thinking…” appears.

The model does not return the result instantly. Instead, it generates successive “reflections” until, after quite a few seconds, it produces an answer. We can watch those reflections in real time by expanding the word “Thinking”. If we do that, bold phrases appear with the title of some supposed reflection the model is carrying out, followed by a somewhat longer explanation in the first person. For example:

Simplifying the code I am thinking about improving fetchChats, simplifying and clarifying how it works while preserving efficiency and readability.

In the following image we can see the chain of thoughts produced by o1 when I asked it to help me with some Swift code I am writing and to simplify a fairly complicated function containing several SQL queries and several iterations over the results:

It seems to analyze in detail all the steps it is taking in the reasoning and to reflect on the result. It even encourages itself: Let’s move forward with this!

Many times these reasonings are “meta-reflections” about the chain of thought itself, as in the following example that someone posted on X, in which o1 explains that “OpenAI’s policies prohibit showing the assistant’s internal reasoning or thought process.” Very curious.

After spending a while “thinking”, the reasoning chain ends and the final answer appears. It is much more elaborate than the answers produced by earlier models such as GPT-4o. It contains many more explanations and considerations, and it gives the impression that it has been thought through carefully and that different factors were considered before reaching a conclusion.

In my own completely subjective experience, when I use o1 as a programming assistant, the result has always been excellent, even with difficult problems and complex code. It has always found a solution to what I asked for and offered valid and reasonable alternatives. Much better than GPT-4o, which was already very good.

As a summary, by observing how o1 behaves we can draw the following conclusions about how it works:

• The model produces an “internal reasoning” made up of thought steps.
• This internal reasoning is monitored, and the user is shown only a summary of it.
• The model uses much more time than earlier models.
• It produces much more elaborate explanations in which you can tell that it has reflected much more deeply on the problem posed.

The first evaluations

The first evaluations show that, beyond my subjective impressions, what OpenAI has presented is a genuinely important advance.

On the Chatbot Arena LLM Leaderboard, o1-preview quickly moved into first place, far ahead of Gemini 1.5, Grok 2, and Claude 3.5 Sonnet.

And Professor Subbarao Kambhampati of Arizona State University, who developed an extensive test based on the blocks world to measure the ability of LLMs to plan actions, has published a paper showing that o1-preview reaches success rates of 97%, 41%, and 52% on tasks where the best previous models achieved 62%, 4.3%, and 0.8%. Going from 0.8% to 52% is completely wild.

Like any good scientist, Professor Subbarao is not overly effusive, but he ended a thread on X with this tweet:

The o1 model seems to move away from the approximate retrieval nature of LLMs toward something like approximate reasoning.

Approximate reasoning. That is quite relevant, especially coming from someone who has spent a long time arguing, correctly, that LLMs cannot plan.

What OpenAI explains

OpenAI stopped explaining how its models work a while ago. That becomes very clear if we remember the launch of ChatGPT a couple of years ago. Back then, OpenAI’s post linked to a paper titled Training language models to follow instructions with human feedback, which explained in depth the RLHF (Reinforcement Learning from Human Feedback) process that made it possible to build the first version of ChatGPT.

However, for the o1 model all we really have is one post, Learning to Reason with LLMs, where some of the ideas behind the model are presented without much detail. They have also published a list of the people who contributed to the development of o1, a post about o1-mini, and a paper describing the safety tests they performed on o1, the OpenAI o1 System Card.

And there is also a video with a conversation with the leaders of the team that developed o1:

What does OpenAI tell us in these documents and interviews? We can draw some conclusions, reinforced by papers and posts that have been appearing.

1. Reinforcement Learning with Chain-of-Thought (CoT) Reasoning

The first paragraph of OpenAI’s document about o1 includes the phrase Reinforcement Learning with Chain-of-Thought (CoT) Reasoning. To understand that concept better, we need to explain what CoT is and how reinforcement learning is used.

The term CoT, chain of thought, is very common in the field of LLMs. It refers to the prompting technique in which we ask the model to reason step by step before solving a problem. If we make the model generate the reasoning steps needed to solve a problem, it will solve it much more easily than if we ask directly for the solution. Jason Wei, then at Google and now at OpenAI, was in January 2023 the first author of the important paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, which introduced CoT with the following figure:

The model’s output is correct when we give it an example of a reasoning chain. Later papers have even shown that it is not necessary to provide those examples or ask for them explicitly in the prompt, because these reasoning chains can be learned from examples.

The other term that appears there is reinforcement learning, RL. This is an older technique that DeepMind applied successfully in AlphaGo and AlphaZero. The algorithm learns what the best action is in each state of the world by efficiently exploring all possible actions across all possible states. Although that sounds simple, when the number of states is exponential, or when the states are difficult to recognize, as with positions on a Go board, the problem becomes extremely complex. That raises the question of how to distinguish the states in which a given action is appropriate and which variables we should look for in those states.

Until AlphaGo, RL had mostly been applied to simple games and toy problems, where the world was clearly specified and involved only a few variables. AlphaGo was one of the first examples showing that much more complicated problems could be solved with this technique. How? By incorporating neural networks that learn to identify the states of a complex world and the possible actions that can be applied in them.

Applying RL to LLMs, where the state of the world is a textual description generated by the user or by the model itself, presents significant challenges. And getting LLMs to learn to use CoT is also a complex task that OpenAI does not detail in its document. However, in the video, Trapit Bansal says the following:

When we think about training a model to reason, the first thing that comes to mind is that we could have humans write out their thought process and train the model on that. The revelatory moment for me was discovering that if we train the model with reinforcement learning to generate and refine its own chain of thoughts, it can do that even better than if humans had written those chains. And the best part is that you could really scale this process.

So it seems that they did train the model on chains of thought written by humans. And, more importantly, they were able to create models, possibly using RL, that generate those chains of thought. According to Karl Cobbe, also in the video, the results have been excellent:

When I was young, I spent a lot of time doing math competitions, and that was basically the reason I got interested in artificial intelligence: I wanted to automate that process. It has been a very special moment for me to watch the model follow steps that are very similar to the ones I used to solve those problems. It is not exactly the same chain of reasoning I would follow, but it is incredibly similar.

That is how they were able to obtain millions of training examples with which they could develop an LLM, o1, that has not only learned to predict the next token of a text, but also the next token in a chain of reasoning.

And on top of that, what people who talk about exponential growth keep pointing out seems to be coming true: we are entering a virtuous circle in which AIs are used to train a new generation of AIs that are even better.

Finally, to be more precise, reinforcement learning was not used only to generate training samples. According to the first paragraph of OpenAI’s Learning to Reason document, it was used to train o1 itself. But they do not explain how.

We introduce OpenAI o1, a new series of frontier models trained with reinforcement learning to perform complex reasoning. o1 thinks before it answers. It can produce a long chain of internal thought before responding to the user.

2. Hidden reasoning

The chains of reasoning generated by the model are hidden from the user and are examined before the final result is shown. During the reasoning process, the user sees only a summary of the reasoning that took place. And any model outputs that do not satisfy the safety guidelines are blocked.

This is mentioned in the System Card as one of o1’s positive features for improving safety, because it increases the transparency and legibility of the system:

In addition to monitoring model outputs, we have long studied the possibility of monitoring their latent thinking. Until now, that latent thinking was available only in the form of activations: large blocks of unreadable numbers from which we could extract only simple concepts. Chains of thought are much more legible by default and could allow us to monitor our models for much more complex behaviors.

The model doing this monitoring could be another model such as GPT-4o, prepared for that task. Or it could be o1 itself. They do not explain any of this either.

3. Better results with more computation

In OpenAI’s post Learning to Reason with LLMs, one of the few images they present is the following:

On the vertical axis we see o1’s score on questions from the American Mathematics Olympiad. The plot on the left shows the typical result we already knew from LLMs, and from neural networks in general: the longer you train them, the better the results. The plot on the right shows something new: the amount of computation time used by the model can be adjusted. And the more time it has, the better the results. In the figure, the same model can go from 20% correct answers to 80% if we give it two orders of magnitude more time, 100 times more.

What is o1 spending that computation time on? Given that the rate at which LLMs generate a response is roughly constant, the simplest answer is that it uses the time to generate many responses. The more time it has, the more responses it generates. And somehow, from that whole set of generated responses, a final answer is built, or selected, and that is the one shown to the user.

Denny Zhou is a DeepMind scientist who leads a team researching reasoning in LLMs. He recently took part in the Large Language Model Agents course with a very interesting talk titled LLM Reasoning. In one of the slides from the talk, he presents an equation that gives the theoretical basis for how to obtain the best solution:

The “P"s in the formula represent the probabilities of the tokens and responses produced by the model. They are, roughly speaking, the scores associated with each response. The higher the probability, the better the score of that chain of reasoning or answer. We should therefore keep the final answer that maximizes the sum of the scores of the reasoning chains that end up producing that answer. To simplify, if we assume that all the reasoning chains generated by the LLM have the same probability and that the LLM generates n reasoning chains, then we should keep the final answer that appears as the outcome of the greatest number of reasoning chains.

The important point in the equation is that obtaining the best final answer is based on generating many responses and keeping the best one. The more computation time we have, the more responses the LLM can generate, and the better the answer we obtain will be.

The future

What does all of this tell us about how LLM technology is going to evolve? Will Google and Meta follow this trend and make models based on CoT? Or will the new models they present continue to rely on scaling the existing ones? Will OpenAI release a generic GPT-5 and then a more advanced o2?

We do not know. What does seem increasingly clear is that the transformer technique is still working and that LLMs are going to become ever more powerful and general.

After presenting o1, Sam Altman wrote a post titled The Intelligence Age in which he says:

In three words: deep learning worked. Humanity discovered an algorithm that can really, truly learn any distribution of data (or really the underlying “rules” that produce any distribution of data). Astonishingly, the more compute and data we feed it, the better it gets at helping people solve hard problems. I find this hard to really understand no matter how much time I spend thinking about it.

A recent talk by Hyung Won Chung, an OpenAI researcher, explains the need to go beyond predicting the next word. And it argues that this is going to be possible with o1.

The learning paradigm based on predicting the next word was only the first step, the way to bootstrap the construction of new, more advanced models. Now that efficient and capable models exist, in the style of GPT-4o mini, it becomes possible to design more complex evaluation functions that go beyond checking whether the model produced the correct word. It is more than possible that these models are currently being used not only to generate training data but also to evaluate the pre-training and fine-tuning of future models.

And we should not dismiss the power of next-word prediction just yet. As Karpathy recently said on X:

As the stack of language models (LLMs) becomes increasingly mature, we may see a convergence of a large number of problems into this modeling paradigm. That is, the problem gets reduced to “next token prediction” with an LLM; the only thing that changes is the use or the meaning of the tokens in each domain.

That is, the “L” in LLMs is irrelevant. We can make the token sequences be anything at all, and the model still keeps learning and generalizing. For example, they are even capable of predicting sequences of data from chaotic systems or protein structure.

Image from the paper “Zero-shot forecasting of chaotic systems”, by Yuanzhao Zhang and William Gilpin, published on arXiv on September 24, 2024.

Everything therefore seems to indicate that we are still not at the end of the curve and that in the coming years many improvements still await us, improvements that will make the models more reliable and more capable and bring us ever closer to AGI.

One final piece of advice, in three words, Altman style: don’t sell NVIDIA. Or, if you do not own stock in the company, there is still time to buy.

Addendum

I asked GPT-4o and o1 to summarize the article in one paragraph. This was the result.

GPT-4o:

OpenAI’s new o1 model, released on September 12, introduces an innovative approach by using deep reasoning based on a chain of thought to solve problems, marking a significant leap over previous models. Unlike GPT-4, o1 shows the user its internal reflections before giving a final answer, which is more detailed and more carefully considered. Although it is available only to paying users and comes with a query limit, its reasoning ability has surpassed other models in evaluations, obtaining surprising results on complex tasks.

o1:

OpenAI has released o1, a new language model that introduces an innovative approach by “reasoning” before answering and generating a chain of thought that is summarized to the user during the process. Although the user interface is similar to previous versions, o1 includes a waiting period in which the model elaborates and shows summaries of its reflections in real time, producing more elaborate and more accurate answers, especially on complex tasks such as programming. The initial evaluations place o1 as the performance leader, significantly outperforming other models and showing notable advances in reasoning and planning. OpenAI has been reserved about the technical details, but it is known that o1 uses “Reinforcement Learning with Chain-of-Thought Reasoning”, training with reasoning chains generated and refined by the model itself. This advance suggests a promising future for language models, with continued improvements in both capability and efficiency.

o1’s summary is much better and more detailed, although it did not like the mention of NVIDIA:

Next week we will return to our usual fortnightly programming, with an issue in which I will comment on some of the summer's news and yesterday's surprise: OpenAI's new model.

Thank you for reading me. And a hug as well to the newly arrived subscribers.

Image generated by Grok. Prompt: “A computer scientist angrily arguing with a colleague over a blackboard about the definition of AGI”.

Lately the term AGI, Artificial General Intelligence, is on almost everyone's lips. Podcasts, blogs, social networks, newsletters, everyone talks about whether we are going to reach AGI in X years or not.

Before I risk making any prediction, I want to spend a little time talking about the term itself. Does it still make sense to talk about AGI? Or has it become a cursed term, one best avoided, ever since people like Altman and OpenAI started using it nonstop? Will people look at you badly if you talk about AGI?

Let us start with an anecdote from last week.

Years ago I used to follow Grady Booch on Twitter. He was an important figure in software engineering in the 1980s, when he popularized some very interesting object-oriented design methodologies. I still have a couple of his books from that period.

When the first generative models started to become public, Booch also started talking about AI. At first it was interesting. He highlighted the limitations and problems of these models, and his voice was a useful counterweight to exaggerated doomers such as Sam Harris or Nick Bostrom. But his timeline soon turned into the same kind of repetitive grumbling you get from Gary Marcus: everything negative, everything problematic. One day, I do not remember which post it was, I got annoyed, went into Van Gaal mode, and unfollowed him.

But recently the X algorithm showed me the following interaction between François Chollet and him:

Grady Booch:

AGI will not happen in your lifetime. Nor in your children's lifetime. Nor in your children's children's lifetime.

Booch's post was more than a year old, but for some reason Chollet saw it a few days ago. François is a nice guy, truly, go watch some of his videos on YouTube, and instead of doing what I did and unfollowing Booch, he replied politely:

In my own definition of AGI, it will definitely happen in my lifetime, in fact probably within the next 10-15 years. But my personal definition is more concrete and more restrictive than most, for me it is not “an artificial human mind” or “an artificial god.” AGI is simply an AI that possesses a degree of generality, the ability to deal with new problems and understand them, at least as high as that of humans. So far generality has been the missing ingredient in AI. We may soon manage to develop it.

Booch answered with a joke about the word “generality”:

In general :-) I agree with you, except that, generally speaking, those measures of generality are so vague that they make the bar for success quite low.

It is clear that Booch does not know all the work Chollet is doing with his ARC Prize, arcprize.org and X, precisely to try to measure, in an objective way, some of that “generality” needed for AGI. We already talked about this prize in the post for the first fortnight of June.

Chollet did not reply again. What I do not know is whether, like me, he also stopped following him.

The above is not just an anecdote. The lack of understanding around the term AGI is becoming more and more intense. And things are now getting even more complicated because of its increasing use outside the scientific sphere. Startup executives, would-be influencers on X or YouTube, many of them use the term mainly to attract attention and capture an audience, or money.

But the popularity of the term also has its good side. More general-interest programs are using it to explain interesting things while doing good scientific outreach. For example, The Economist, in its always-interesting weekly podcast Babbage, has published a special on AGI, AGI, part one: what is artificial general intelligence?. The program tries to provide a fairly academic perspective by interviewing different kinds of people, including engineers, computer scientists, and neuroscientists.

Melanie Mitchell, an AI scientist deeply familiar with traditional AI but also with LLMs, see for example her article Large Language Models in The Open Encyclopedia of Cognitive Science, comments on a definition tied to human capabilities:

AGI has been defined as a machine that is able to do everything a human being can do. And then, more recently, that definition has been weakened a bit, by defining it as a machine that can perform all the cognitive tasks a human being can do, leaving aside physical forms of intelligence.

But then she also emphasizes that she does not like the term AGI very much:

Host: Do you think the use of the phrase AGI is really useful for AI scientists like you, or do you see it more as a distraction?

Mitchell: I think it is a bit of a distraction. People feel they can take intelligence as something separate from its manifestation in humans, in the human brain and body, and isolate it [...]. And I am not convinced that this is really meaningful or that it gives us a clear direction to follow.

However, Google scientist Blaise Agüera y Arcas does not get bogged down in terminological debates and says that the problem is not one of making models more general, but of making them better in different respects:

I think it is really just about getting better at a bunch of things we all care about, like truthfulness, reasoning, memory, planning, having a consistent perspective over long periods of time, and so on [...] So I do not think it is about how far away we are from any one particular thing, but rather how quickly these things are improving, and when they will become reliable enough to do a variety of different things that, at the moment, I would say they are not reliable enough to do autonomously.

So there is no agreement even among the scientists most deeply involved in the subject. Some say AGI is not a useful term, others say it is, because precisely what is needed is that generalization. And others say we are almost already there, and that what remains is just improvement.

So what is my opinion? AGI or no AGI?

For me, as Chollet says, the key lies in the “G” of the term: “general.” This “G” symbolizes a meaningful shift in the evolution of artificial intelligence and neural networks, from specialized models, like those that dominated the 2010s, toward more generic and versatile models such as today's LLMs, which are capable of storing all human knowledge and interacting in natural language.

And, as Agüera y Arcas says, we will move closer to that “G” as new algorithms are developed that improve the shortcomings of current LLMs by giving them new capabilities, capabilities that allow them to solve problems such as Chollet's ARC Prize.

As for me, I will keep talking about AGI, even if every time I do so I have to reference this article so that people do not confuse me with an AI Bro.

Image generated by ChatGPT 4o. Prompt: “Make an image of an AI Bro”.

Until the next fortnight, see you then! 👋👋

Also, as you may have noticed in earlier issues, I tend to let the articles grow too long. From now on I am going to try to include only one long topic in each issue, and I will put it in a new section called In depth. It will also be a topic that arose during the fortnight, but one to which I devote a little more detail.

Thank you for reading me!

🗞 On the radar

1️⃣ The James Webb Space Telescope, JWST, has provided new data on LHS 1140 b, an exoplanet located 48 light years from Earth in the constellation Cetus. These findings, published on July 10 in an article in The Astrophysical Journal Letters, one of the most prestigious journals in astrophysics, reveal details about the composition of the exoplanet's atmosphere and the possibility that it may be a potentially habitable water world.

The press release published on July 8 by the University of Michigan includes an image showing a possible representation of the planet as a partially frozen world with a considerable amount of liquid water.

Possible representation of the exoplanet LHS 1140 b.

The study, led by Charles Cadieux and an international team of researchers, used JWST's NIRISS instrument, the Near-Infrared Imager and Slitless Spectrograph, to observe two transits of LHS 1140 b. This planet, with a radius of 1.7 Earth radii and a mass of 5.6 Earth masses, orbits in the habitable zone of an M4.5 dwarf star.

The habitable zone is the region around a star where temperatures are neither too hot nor too cold, allowing liquid water to exist on a planet's surface. The presence of liquid water is crucial because it is an essential requirement for life as we know it. In this zone, a planet receives the right amount of stellar energy to maintain moderate temperatures, which could favor the development of habitable conditions. This region varies depending on the type and luminosity of the host star. For a dwarf star like the M4.5 star of LHS 1140 b, the habitable zone lies closer to the star than the habitable zone around larger and brighter stars such as our Sun. The planet LHS 1140 b lies in that privileged zone, making it a prime target in the search for life beyond our solar system.

In the transmission spectrum resulting from the transit, certain atmospheric features of the planet can be observed. The most significant discovery presented in the paper is tentative evidence, at 2.3σ, of an atmosphere dominated by nitrogen. The authors used climate models to rule out a mini-Neptune-type atmosphere with greater than 10σ confidence. This indicates that the planet is not a gaseous planet, but rather a rocky super-Earth.

The paper discusses “water world” scenarios, suggesting that LHS 1140 b might contain between 9% and 19% of its mass in the form of water. For comparison, Earth's water mass fraction is less than 1%. But this still cannot be confirmed. New observations with other advanced JWST instruments will be needed in the coming years, allowing scientists to study and detect the presence of atmospheric CO2, which would confirm the existence of liquid water. The presence of liquid water would represent a necessary step toward the existence of life.

Will we be able to detect biosignatures on some exoplanet soon? The detection of molecules such as oxygen, methane, or an unusual combination of gases that would not normally coexist without biological processes, could suggest the existence of life. It is still early, but work like this gives us new clues and brings us a little closer to the momentous moment when we may be able to announce, with scientific confidence, the discovery of life beyond Earth.

2️⃣ The interesting New York Times podcast Hard Fork published, on July 5, an interview with Ted Sarandos, the CEO of Netflix. The interview is also available on the newspaper's website.

Netflix is a surprising success story, one that fascinated me from the moment it was born in the US. For me, the equivalents of the “iPhone moment” in the field of services have been Amazon and Netflix. They are companies that changed the world radically and took us into an “alternate universe” that might never have existed.

I remember the excitement I felt when Netflix arrived in Spain, about a decade ago. I was fed up with looking for series, not finding episodes, downloading them in poor quality, adjusting subtitles, syncing them... well, I am sure you all remember. Then Netflix arrived, opened the door to the whole streaming market, and established itself as the reference platform. And it made life much easier and more entertaining.

In the podcast, Sarandos talks about his beginnings as a clerk in a video rental store, just like Tarantino, and how that experience led him to Netflix, where he has been fundamental in transforming the company from mailing DVDs to becoming a global streaming giant.

Ted Sarandos in 2005.

Throughout the interview, he discusses Netflix's content strategy, emphasizing the importance of authenticity and diversity in international productions, as well as the balance between quality and quantity in its programming slate.

Sarandos addresses several challenges and opportunities facing Netflix, including competition in the “streaming wars,” the introduction of advertising on the platform, and expansion into live entertainment. He also reflects on how Netflix navigates political and social issues, and on his view of the use of artificial intelligence in the entertainment industry. Throughout the conversation, Sarandos defends Netflix's “something for everyone” approach, arguing that quality is defined by the audience's love for the content, regardless of critics.

Ted Sarandos today.

Sarandos comments on one of the central elements of Netflix's successful model: content recommendation in a global market:

"I think we're entering a new era where content and great stories can come from just about anywhere in the world, and they can sit really conveniently on the shelf next to your favorite show. You can discover a great story from Korea, or a great story from Italy, or a great story from Spain that you otherwise wouldn't have access to and maybe wouldn't even know about, but that is the kind of storytelling that is really adjacent to the kind of storytelling you like."

The podcast concludes with Sarandos's reflections on the future of Netflix and of the entertainment industry more broadly. He discusses competition from free platforms such as YouTube and how the company maintains its focus on innovation and quality. And he ends with a reflection on the use of AI:

"I don't think there's a scenario where an A.I. program is going to write a better script than a great writer, or replace a great performance, or that we won't be able to tell the difference. A.I. is not going to take your job. The person who uses A.I. well might take your job."

3️⃣ It turns out that Hans Zimmer, the composer of soundtracks we have all enjoyed, Gladiator, Sherlock Holmes, Inception, Interstellar, The Dark Knight, Dune, is really Hans Zimmer Inc.

Hans Zimmer, in a BBC documentary.

On Eric Molinsky's podcast Imaginary Worlds, on July 3 they published the fascinating episode The Team Behind Hans Zimmer, which interviews musicians and sound technicians from Hans Zimmer's studio, Remote Control Productions.

The episode explores the unique way in which Zimmer and his team create the iconic soundtracks that have defined so many films. Zimmer's studio is described as a magical and collaborative place, full of analog and digital instruments, where every detail is carefully orchestrated in order to produce innovative sounds.

The episode goes deeper into Zimmer's creative process, highlighting how he gathers talented musicians and technicians to experiment and create new sounds, with special emphasis on their work for the film Dune. For Zimmer, what matters is not only melody, but also the search for specific sounds. That becomes quite clear in the following transcript of one section of the podcast:

Raul Vega: A few years ago Hans wanted to expand his percussion library. So he called us and said, “Hey, I want you to build me a drum kit.” So we said, “Okay, do you have a preferred drummer or a set of drums you want us to use?”

Hans said, “No, no, no, no, no. Let me rephrase this. I want you to build a drum kit out of found sounds. Take anything that is out in the yard, take anything that is in the alley, go to Home Depot and just play.”

Raul Vega: We went to Home Depot, bought some bricks and chains and a sledgehammer. And we put a bunch of microphones in the live room and took turns breaking things.

Taurees Habib: We just smashed it all up.

Raul Vega: Capturing only the definitive sound of destruction. Once they had those recordings clean, organized, and processed, it sounded like this.

[clip: sounds of destruction]

Raul Vega: Those are sounds of axes on two-by-fours, broken piano lids, chains, clay bricks. Now, for a long time, we had no idea what this was for, and I even think he did not know what it was for. So we kept it in our sound library for a very, very long time.

But years later, while working on Dune...

Raul Vega: He called us and said, “Hey, do we still have that?” I said, “I think so.” “Great. Send it to me.” So we assembled it, he mixed it, and then it became a very, very popular marching theme for Dune.

[clip: Dune Sketchbook - House Atreides]

Hans used these sounds in a version of the cue that plays when Paul and his family arrive on Arrakis.

The episode highlights the importance of collaboration in Zimmer's work, explaining how different musicians, sound engineers, and composers contribute to creating the complex layers of sound that characterize his scores. It details the innovative use of female voices, modified instruments such as the electric cello, and the creation of digital instruments from recordings of unusual sounds.

The episode is very much worth listening to. It is beautifully produced and gives us a unique view of how creative processes can be carried out as team efforts.

I will finish this part by linking a video by Jaime Altozano, as didactic as all the videos he makes, in which he reviews Hans Zimmer's professional career.

🔍 In depth

Now for the new section in which we devote a little more space to one of the topics of the fortnight. This fortnight we are going to comment on an article by Andy Matuschak about the Primer.

At the beginning of July, Andy Matuschak shared a post titled Exorcising us of the Primer. A few weeks earlier, on June 6, he had published it only for subscribers on his Patreon. What is this “Primer”? Who is Andy Matuschak? Why is this post relevant? Let us try to explain it.

Header image from Andy Matuschak's X account and Patreon page.

The subtitle of Andy Matuschak's Patreon page is Creating tools for thought. The expression tools for thought is used to describe the field of research and creation concerned with methods, technologies, and strategies designed to improve and expand our cognitive and learning capacities. Note-taking tools such as Obsidian, or spaced-repetition methods such as Anki, are examples of these tools.

This area of creation and research is directly linked to the development of computing and personal computers. The precursors of these ideas were researchers such as Vannevar Bush, with his famous article “As We May Think” (1945), Ted Nelson, who coined the terms “hypertext” and “hypermedia” in 1965, Douglas Engelbart, creator of the 1968 demonstration known as “the mother of all demos,” and Alan Kay, who developed the Dynabook concept in the 1970s. The web, personal computers, and the computational devices we have today are based on some of the ideas proposed by these visionaries1.

Andy's work, together with that of other researchers such as Michael Nielsen and Bret Victor, tries to capture the essence of those original ideas and apply them, making them real with today's technologies. It is striking that all these researchers are independent researchers who have developed their work outside academia and outside private companies. Private companies do not want them because the field is too speculative, and academia does not want them because the field is too novel and interdisciplinary.

All right, now we have the field of research in place. So what is this Primer? Let us get into it, through a personal story.

A few years ago I came across Cryptonomicon (1999) and Neal Stephenson. It blew my mind. The first thing I asked myself was how it was possible that I had not read it before. It had been published more than 20 years earlier, in 1999, and it dealt with cryptography, Alan Turing, World War II, and the Internet and electronic money, before Bitcoin. I should also have known Stephenson earlier, a prolific and brilliant writer who, as his Wikipedia page says, has written science fiction, speculative fiction, historical fiction, cyberpunk, and baroque works.

The thing is that after reading Cryptonomicon I got hooked on Stephenson and, looking for more of his work, I arrived at The Diamond Age (1995). The subtitle of the book was very curious: A Young Lady's Illustrated Primer. An illustrated primer? For young ladies? I read it, and found another wonderfully original book, full of ideas about nanotechnology, virtual reality, intelligent assistants, and the adventures of a humble young girl whom an intelligent book, the famous Primer, rescues from poverty and turns into someone powerful.

The idea of the Primer gave me a kind of inception, and I have kept it in mind ever since whenever I reflect on or read about educational tools. The same has happened to many others, as Andy comments at the beginning of the article:

[The Primer] is my field's most canonical vision for an incredibly powerful learning environment. If you ask a technologist interested in learning what they dream of accomplishing, most will answer, “build the Primer.”

So what is the article about? Why do we need to “exorcise” this idea? It is a long and detailed article that has to be read slowly, and more than once. Andy first explains how the Primer works in Stephenson's book and gives examples of interesting elements in the Primer. For example, the use of dynamic media, not only as didactic and motivating elements, but also as useful instruments that accompany the whole learning process:

The Primer’s nanomolecular microscope helps Nell get started with cellular biology, but it’s also a tool that makes expert biologists vastly more capable. It’s not a toy representation, a cognitive dead end Nell will need to discard as soon as she develops some intuition. It’s a tool that can grow with her into legitimate practice, a tool which in fact expands the frontier of practice for the whole field.

Despite these positive features, the Primer proposes a view of learning that Andy thinks must be surpassed:

• Authoritarianism: the Primer has a hidden agenda and exerts excessive control over Nell's learning. That is immoral and ineffective if the goal is to develop independent thinkers.

• Isolation: the Primer isolates Nell from meaningful purpose and from genuine human interactions, limiting authentic learning and connection with reality.

• Excessive gamification: the idea of making learning always fun through games is unsustainable and cannot compete with games designed exclusively for entertainment.

• Excessive discovery learning: although discovery learning can be valuable, it is not enough on its own and must be complemented by explicit instruction and structured practice.

With these arguments, Andy is criticizing many current approaches in educational innovation. I especially liked, and I share, his criticism of gamification and discovery learning, concepts that are very fashionable today.

Finally, Andy argues that we should leave behind the vision of the Primer and create a new learning environment that supports action and genuine interest. This new system must be integrated into the real world, offering dynamic and adaptive support that encourages immersion and emotional connection, but always with the goal of enabling authentic and meaningful participation in activities that matter to learners.

A system like this wouldn’t really be a “primer” any more, an isolated lesson to complete before real participation, useful only for beginners. It would be more like a general-purpose enabling environment, a tool for thought which would increase the probability and speed of learning-dependent action.

[...] This probably means some form of ubiquitous computing. My picture of that form is still pretty blurry, but you can see an early sketch of what a general enabling environment might be in my recent talk, “How might we learn?”.

While a tool like that is being built, I will keep using and experimenting with LLMs as learning assistants, to give me context and clarify concepts in the new fields I venture into.

👷‍♂️ My fifteen days

This fortnight we watched less cinema. Only a couple of films, and I gave all of them fewer than 4 stars on Letterboxd. I also did not make progress on my website, or on any project worth mentioning. So we are left with comments on a couple of series and on the book that I finally finished during the fortnight.

📺 TV

We watched the new season of Hacks on Max, and it continues to be just as good as the previous ones. Jean Smart as Deborah Vance and Hannah Einbinder as Ava Daniels are wonderful, dazzling in their versatility and in the range they bring to those complex characters. And they are surrounded by an equally magnificent supporting cast, and characters.

Very funny scripts, and a season that keeps growing episode after episode until it reaches a terrific finale.

And another series we liked quite a lot, Chapelwaite, also on Max. It is from a few years ago, 2021, and I had had it on my radar for a long time. First because it is about vampires and because it stars Adrien Brody. Then the series turns out to be much more than that. As Jorge Loser says in his review in Espinof, it is a story for lovers of old-school Gothic horror, with elements of H.P. Lovecraft, Stephen King, John Carpenter, and George A. Romero.

It is curious that in that same year, 2021, Mike Flanagan created the masterpiece Midnight Mass, which drinks from many of the same themes, but updates them to the present and gives a twist to the religious and existential aspects.

Flanagan's series immediately became my favorite series. Chapelwaite is more classical and has a somewhat weaker ending, but it touches on very interesting themes and develops them with a lot of style. Highly recommended.

📖 Books

I finally finished Dune Messiah, by Frank Herbert. I liked it just enough, I gave it 3 stars out of 5 on Goodreads. I am copying below the review I posted there.

I found it somewhat boring and too full of palace intrigues. I also did not like the fascist overtones I see in the novel, the vision of an all-powerful empire led by a messianic leader with superpowers, supported by a court of faithful servants helping him in his dynastic struggle. Paul fools himself into thinking he does it for the good of the oppressed Fremen and of the universe, but deep down he is not so different from Stalin, Netanyahu, or Hamas.

That comment came out a little too harsh, comparing poor Paul with those dictators. Deep down the character is pitiable, trapped as he is in a machine he cannot escape and fully aware, because of his precognition, that any decision he takes will bring suffering and destruction.

I would have liked the novel to show, in a more concrete way, the terrible effects of the Jihad. We are told that it causes the death of billions of people and that it destroys the traditions of many planets, forcing them to embrace the Fremen religion of Muad'Dib, but it is mentioned only in passing, without much emphasis. I would have liked to see it firsthand, through some character from one of those planets conquered by the Fremen. But that would have been another book, perhaps one by Sanderson, and not the one Herbert wanted to write.

Until the next fortnight, see you then! 👋👋

Thank you very much for reading.

News

1. Now that summer has arrived, it is a good moment to review the evolution of the planet’s global temperature, which we talked about in April. As in the previous post, all the data comes from the Climate Reanalyzer website of the Climate Change Institute at the University of Maine. Will we have a summer as hot as last year’s?

The air temperature in the Northern Hemisphere gives us some hope.

The thick black line shows this year’s temperature. Is it starting to come down?

The orange line shows last year’s temperature. At the beginning of July it was 21.7 C, 1.1 C above the average, and it ended the month at 22.7 C, 1.4 C above the average. The thick black line is this year’s temperature. It seems to be starting to come down, but it is still too early to draw conclusions. Better to look again in a couple of weeks, when we are in the middle of the month.

The graph that cools us down the most is the sea surface temperature in the Northern Hemisphere, which is finally lower than last year’s.

It seems to be getting cooler.

Could it be that El Nino is already over? That the effects of the water vapor from the Hunga Tonga eruption are beginning to fade? Let us keep our fingers crossed that the black line keeps moving forward horizontally.

2. On June 17, Runway introduced its new video-generation model, Gen-3 Alpha. The videos are short sequences of just a few seconds, with very high quality and consistency, similar to the ones we already saw from OpenAI’s model Sora.

Unlike OpenAI, Runway has already opened access to the tool. To generate videos with this latest model you have to subscribe to the paid plan, at $12 a month. I already spend enough paying OpenAI, and this month I had already exhausted my budget for little indulgences on Sonnet 3.5. But on X you can see many examples from people trying it out. For example, Javi Lopez’s dancing spaghetti.

Someone has even published on Reddit a comparison between videos generated by Sora and those generated by Gen-3 Alpha, using the prompts from OpenAI’s promotional video.

I still think what I already said in February. These seem to me to be impressive advances, but we have very little control over the result, and I do not think it will be possible to scale their use to produce a short film or a feature film. Nor does that interest me. When I go to the cinema I want to see something created by people and performed by people. And if it is an animated film, I want to see a consistent, coherent work that conveys feelings through sequences designed and directed by human authors who pour all their experience into a piece of work. I am not interested in what an AI generates at random within the framework of a text prompt.

3. Francois Chollet has appeared in several podcasts as a result of the attention surrounding his ARC competition, which we discussed in the previous fortnight. After listening in full to the interviews conducted by Dwarkesh Patel and Sean Carroll, I have become a total fan. Chollet has been working with neural networks and deep learning since the middle of the last decade, and his neural-network library Keras is widely used in the community. He is a very technical person who knows what he is talking about.

I already mentioned the Dwarkesh Patel podcast in the previous fortnight. Below I include the links to the podcasts mentioned, their transcripts, and some comments and quotes from those transcripts.

The Dwarkesh Patel podcast:

Its transcript can be found on Substack.

I found Chollet’s idea of interpreting LLMs as a “large interpolative memory” extremely interesting, a huge collection of programs that implement patterns learned during training. When you query an LLM, it performs an interpolation among the patterns that best fit the answer.

“The way LLMs work is that they are basically a big interpolative memory. The way you increase their capabilities is by trying to stuff as much knowledge and as many patterns into them as possible.”

According to Chollet, that way of operating gives an LLM only very limited intelligence. It is not able to combine the programs it has learned and, through a search process, invent a new “program” that solves a novel situation not included in its training database. For Chollet, the ability to carry out combinatorial searches is a fundamental element of intelligence. For example, when we play chess or Rummikub, we must search through possible combinations and choose the best one. An LLM cannot do that:

“To get novelty, you need search. LLMs cannot do search, they can only do interpolation.”

For Chollet, LLMs are powerful tools for memorization and the application of known knowledge and patterns, but they lack the ability to adapt and create novel solutions, which is crucial for reaching true general intelligence.

Chollet is also critical of the idea that scaling models leads to models that generalize better. For him, what scaling does is increase the amount of skills and data, but that does not mean the models are more intelligent:

“If you scale up your database and keep adding more knowledge and more program templates to it, then sure, it becomes more and more skilled. You can apply it to more and more tasks. But general intelligence is not task-specific skill scaled up to many skills, because there is an infinite space of possible skills.”

Despite all that, Chollet argues that LLMs do have their usefulness and their applications. He says that LLMs, like other deep-learning systems, can recognize and apply patterns efficiently. That is why they are excellent for “type 1” intelligence, intelligence based on intuition, pattern recognition, and memorization. This kind of intelligence is fast and automatic, used for tasks that do not require deep or deliberate reasoning. But there is another form of human intelligence, slow and deliberate, based on reasoning, planning, and the synthesis of new programs or solutions.

Francois Chollet suggests that, in order to move toward true general intelligence, it will be necessary to develop hybrid systems that combine deep learning with search and exploration to generate new programs by combining those that have already been learned. In that way it would be possible to take advantage of the strengths of both kinds of intelligence.

The interview with Sean Carroll on his Mindscape podcast is available here:

And the transcript is available on Sean Carroll’s blog.

The interview is very interesting, more didactic than the first one. Carroll asks more than once for clarification on aspects that the audience might not understand, genetic algorithms, transformers, vector spaces, and so on, and Chollet makes an effort to explain them.

In the interview, Chollet takes a fairly strong position and argues that we have reached a kind of plateau in LLM improvement due to the lack of training data:

“The curve [that represents LLM improvement] has to fit something. The curve is literally just a representation of a training data set. If you have run out of data, then how do you improve the model? Well, one way is that you can try to curate your training data better. So you are not increasing the scale of the training data, but you can increase the quality. That is really a very promising way to improve large language models. It is actually the way large language models are still improving today. We have already run out of data. So the next stage is that we curate the data better. We are not training large language models on more data, we are actually curating it. Technically, we are still collecting new data from human evaluators. So there is a little bit of increase, but on balance it is actually decreasing. But you are not going to magically find a thousand times more new and non-redundant data to train these models. It just does not exist. You are not even going to find twice as much. And that is the cause of the plateau we have been seeing.”

And that plateau is going to cause disappointment:

“That is the cause of the plateau we have been seeing. And something like GPT-5 will probably be released later this year. It is going to be a big disappointment because it is not going to be significantly better than GPT-4.”

Finally, on the problems AI may bring us and on existential risk, Chollet has a position very similar to the one we have already discussed here on other occasions. Even if AGI does arrive, it will still be only a tool that we can use. The problem will lie in how it is used, not in AGI itself wanting to exterminate us:

“Intelligence itself is just a tool. It is just a way to achieve goals. If you do not connect it to the capacity to set autonomous goals, then it is fairly harmless. It is not completely harmless because it will be in the hands of humans and humans are dangerous. So it is dangerous in that sense, since people could potentially use it for bad purposes, but it is not dangerous in the sense that it competes with the human species.”

4. In the second half of June, two new LLMs of interest were released: Anthropic launched Claude 3.5 Sonnet and Google launched the 27B open-source model Gemma-2.

Both releases continue the trend of recent weeks: smaller models trained better. Anthropic’s model is the next version of the medium-sized model in the Claude family, and Google’s model is the next version of its open Gemma model.

Just three months ago, in issue 5 of 2024, we were talking about Anthropic releasing its 3.0 family of models: Haiku, Sonnet, and Opus. The last of these was the most powerful one, in GPT-4 territory. Sonnet and Haiku are smaller models, faster and cheaper at inference time.

Only three months later Anthropic published the following figure:

The smaller models are coming for the bigger ones.

Sonnet is now Anthropic’s most powerful model, outperforming an older larger model. The same thing happened with Gemini 1.5 Pro, which we discussed in issue 4 of 2024. Google released the next version of its medium model, the Pro, leaving the update of the largest one, Ultra, for the future.

Anthropic shows the following scores for Sonnet 3.5 on the most popular benchmarks, beating Opus 3 and, in many cases, GPT-4o, OpenAI’s leading model at the moment.

Sonnet is also multimodal, capable of interpreting images. And Anthropic launched it together with the feature called artifacts, a side window next to the conversation in which the model can run code.

For example, the following clip is the result of a session in which I told Sonnet how to create a game. The initial idea was to move a blue square around the screen, and we ended up making a loose version of Pong. Sonnet generated the code, and I kept asking for new features, things like “Make a star appear that you have to avoid” or “The game is a bit boring, make the number of stars increase.” The final result, and the whole process, is incredible.

5. I will end with a reflection on the evolution of LLMs. In a post on X, Jim Fan published the following image:

GPT-4 is no longer unique.

The image seems to answer one of the questions we were asking at the beginning of the year: was GPT-4 replicable? When GPT-4 was presented in March 2023, many of us wondered whether the enormous leap from GPT-3.5 was due to some exclusive OpenAI knowledge that would be hard for other companies to replicate. A year has passed, and the figure above seems to show that the answer is no, that OpenAI does not have a secret recipe for making LLMs and that other companies, Google, Anthropic, Meta, have already reached or are about to reach GPT-4, even with smaller models.

There was a second question still to answer: will model intelligence keep scaling as the models become larger? GPT-3.5 had 175 billion parameters, 175B in English notation. OpenAI has never disclosed the number of parameters in GPT-4, but Nvidia CEO Jensen Huang let slip that it was 1.8T, 1.8 trillion. Putting them in the same units, GPT-3.5 has 0.175T parameters and GPT-4 has 1.8T. In other words, GPT-4 is an order of magnitude larger than GPT-3.5.

We are all waiting for the launch of GPT-5, OpenAI’s next large model. Presumably it will be another order of magnitude larger, with around 20T parameters. There are some estimates of the time needed to train this model and of how that time will evolve with Nvidia’s new GPUs:

• OpenAI began training GPT-5 at the end of December 2023 using H100 GPUs.
• The training was expected to last 3 months and finish by the end of March 2024.
• For GPT-5, the use of at least 50,000 H100 GPUs was predicted, compared with the 20,000 A100s used for GPT-4.
• The model would have around 20T parameters.
• The process of tuning and additional testing would take 3 to 5 months, with a possible release date in July or August 2024.
• Microsoft could have access to 500,000 H100 GPUs by the end of 2024.
• OpenAI could use up to 250,000 H100 GPUs to train a 50T-parameter model in the third quarter of 2024.
• There was the possibility of releasing an intermediate model, GPT-4.5, with 10T parameters and delaying GPT-5 until December 2024.
• The arrival of B200 GPUs by the end of 2024 would make it possible to train models with tens of trillions of parameters, 20T, 30T, 40T, and so on.

All the major tech companies are in this race and that is why Nvidia is currently the technology company with the largest market capitalization. They cannot keep up with GPU demand.

Before long, when the models that are being trained right now become public, we will see whether the leap in the number of parameters also represents a leap in “intelligence”, and whether the scaling law of language models continues to hold.

There is already at least one slide that is being used by people from OpenAI to suggest that the jump will be enormous:

And the latest statements by people who have surely had some contact with the first results of these new models, such as Bill Gates, Dario Amodei, or Demis Hassabis, also point in that direction.

For example, Gates talks about the next two generations of LLMs in the following clip, taken from a much longer conversation available on YouTube. It is an edited video published on X by Tsarathustra, do not be fooled by the title, he posts very interesting videos and news.

Gates says two important things. First, there is going to be a significant jump in the next two generations of LLMs, let us call them GPT-5 and GPT-6. To make that jump, training data will also have to increase by orders of magnitude, and video will have to be used.¹

The second point he makes is very similar to what we were just saying about Chollet, and what LeCun has always maintained: scaling LLMs will produce improvements, but it will not bring us AGI. For that, other algorithms and strategies will be needed, ones that allow the implementation of “metacognition” so that AI can reflect on the thoughts it is generating.

Will scaling allow us to get closer to AGI? Or are we already seeing its limits? I think it is still too early to draw a definitive conclusion. I think Chollet’s idea that LLMs learn patterns of programs allows us to argue that larger, better-trained LLMs may generalize those patterns better, not simply increase their number. And the problems LeCun has always pointed out, that text is not enough to learn a physical model of the world, may be overcome when LLMs are trained directly on video sequences, perhaps within a couple of generations, GPT-6 or GPT-7. Or perhaps LeCun and Chollet are right and we have already reached the ceiling of what can be done with LLM and transformer technology.

As we always say around here, we will see. It is still too early to know; in three or four years we will be able to say something more definitive. In the meantime, we can always place bets.

My two weeks

Movies

I was a little disappointed by A Quiet Place: Day One. I found it a bit slow and boring, and I never really connected with it. Weaker than the previous ones. And I had a lot of fun with Under Paris on Netflix. A shark movie, the kind Claire and Phil would enjoy.

Of all the films from the fortnight, the one I would highlight is The Greatest Hits, on Disney. A lovely story of love, music, and time jumps. It is the second film by director Ned Benson and it stars a trio of absurdly beautiful young actors: the wonderful Lucy Boynton, who also starred in another film I watched recently and loved, Sing Street, the future Superman, David Corenswet, and Justin H. Min, whom I recognized from The Umbrella Academy.

I need to make a Letterboxd list with all the films and series of this kind that I have loved: Begin Again, Sing Street, or Daisy Jones and, why not, School of Rock. Well, I have already made it.

TV

The Apple TV+ series Dark Matter was great fun. We liked it a lot.

As always with Apple, it is an excellent production. And regarding the subject matter, although the multiverse concept is already overused, I cannot think of many films or series that handle it especially well, sorry, I have not seen Fringe. But this story by Blake Crouch does handle it well. It is quite original, has good twists that genuinely surprise, and the multiverse is not an excuse but the central element of the story. Very good work from Joel Edgerton and Jimmi Simpson. And Jennifer Connelly and Alice Braga are solid too, although their characters did not really have much more room to offer.

See you in the next fortnight.

Thank you very much for reading.

News

1. In the previous issue of the newsletter, I mentioned that Apple's WWDC24 was about to begin.

WWDC24 arrived.

On June 10, Apple held the conference keynote and introduced the new versions of its operating systems: iOS 18 (preview and press release), iPadOS (preview and press release), macOS Sequoia (preview and press release), watchOS 11 (preview and press release), visionOS 2 (press release), and tvOS 18 (press release). These operating systems are already available as betas and will be released in the autumn, presumably alongside the new iPhone lineup.

As always, the most interesting part of the conference is the developer material. From Monday the 10th to Friday the 14th, Apple released dozens of videos, guides, and documentation that now need to be reviewed if we want to build apps for any of its platforms. I am noting the links here so I have them handy:

• Sessions: more than 120 videos, between 10 and 30 minutes each, explaining the new features introduced across platforms and technologies: Swift, Design, Spatial Computing, Graphics, and more.

• Selected sessions: a curated Apple list of around 60 sessions, grouped by technology.

• What's new for developers: a page summarizing new features across platforms and technologies.

• Documentation: the documentation section with all the details on new APIs and capabilities.

• Guides: a hub page with guides to the new features of different platforms and technologies.

• Pathways: a page that gathers step-by-step introductions for getting started with Apple's various platforms and technologies.

Personally, what I value most about WWDC is that it showcases the work done by hundreds of Apple engineers over the course of a year. That may be an occupational bias from being a software engineer, or perhaps the perspective of a father with daughters in their twenties who are just starting their careers, but when I see all these materials I mostly think about the young people who built them, and about their effort and their hopes.

2. One of the announcements I was most curious about was Apple's stance on AI. In the previous issue I also listed the questions I expected the conference to answer.

The answers arrived, and so did a certain amount of confusion. Of all the announcements, the one that had the biggest impact was the one Apple least wanted to emphasize: the deal with OpenAI. Apple wanted the focus to be on its own work and its own features, but the agreement with Sam Altman's company immediately captured media attention and somewhat overshadowed the main message.

However, in the days that followed, as specialized media paid more attention to the features Apple had presented, the company's real idea started to sink in. The marketing team's masterstroke worked very well: the wordplay of turning AI, Artificial Intelligence, into Apple Intelligence.

Here are a few links.

In Spanish:

• Angel Jimenez de Luis: Asi funciona Apple Intelligence, la IA de Apple

• Pedro Aznar: Apple Intelligence, primer contacto

In English:

• Ben Thompson: Apple Intelligence Is Right on Time

• Jason Snell: OK, fine, here's Apple Intelligence

• MacStories: Apple Intelligence: The MacStories Overview

• The Verge: Apple Intelligence: every new AI feature coming to the iPhone and Mac

• Ars Technica: Apple unveils "Apple Intelligence" AI features for iOS, iPadOS, and macOS

And links from Apple itself:

• Introducing Apple Intelligence, the personal intelligence system that puts powerful generative models at the core of iPhone, iPad, and Mac (press release)

• Apple Intelligence web page

Of all the explanations I have seen, the one I liked most is Apple's own video, "Apple Intelligence in 5 minutes":

One of the core ideas behind Apple Intelligence is to provide concrete features powered by generative models built by Apple itself, both text models, LLMs, and image models. Apple is deliberately stepping away from the OpenAI and Microsoft idea of a "general-purpose" agent that does everything and requires the user to define its behavior with a prompt.

When Apple uses the phrase "AI for the rest of us" on its site and in its ads, it is reviving the original campaign from forty years ago that introduced the Macintosh.

"The rest of us" are the people who do not want complications, people who want to use AI for specific things without configuring anything or writing advanced prompts. All the features are based on generative models, but the whole job of crafting the prompt and communicating with the model is hidden away or reduced to a minimum. We only have to choose a few options in an intuitive way, just as we do when using any app that is not explicitly an AI app. And in those cases where a prompt is needed, such as to generate an image, it is kept to the bare minimum.

This is a key part of Apple's approach: finding concrete things that a generative model can do and that are useful to ordinary users, "the rest of us".

The other key part of Apple's approach is to exploit the fact that it has access to all our information and apps on our phones in order to take the first steps toward turning Siri into a personal assistant that can look at our calendar, our emails and messages, and interact with our apps on our behalf. For now they have not gone very far and have shown a very simple example: finding out what time your mother's flight arrives. It is a good place to start, and as apps expose more options we will see more advanced examples, provided the on-device LLM that supports Siri is able to manage them correctly.

I will finish this section by listing the specific features Apple presents in the video, using the names created by its marketing team:

• Writing Tools

• Genmoji

• Image Playground

• Image Wand

• Siri - Natural Language

• Personal Content

• Priority Notifications

• Memory Movie

There are a few more features that do not appear in the video, such as those related to Photos and photo editing. And Apple will keep improving these capabilities and adding new ones over the coming years. We will look at them little by little and discuss some technical details.

If this WWDC made one thing clear, it is that Apple does not plan to stay behind the rest of the industry when it comes to AI. Even if, for Apple, it is "Apple Intelligence" rather than "Artificial Intelligence".

3. On June 11, François Chollet (fchollet.com), the well-known machine learning researcher and author of the excellent book Deep Learning with Python, introduced the ARC Prize (arcprize.org and X), organized together with Mike Knoop (mikeknoop.com). It is a competition with a one-million-dollar prize for solving a problem that LLMs have so far failed to crack. The Dwarkesh podcast interview where Chollet and Knoop present the prize is very much worth watching.

The original ARC problem, Abstraction and Reasoning Corpus, was proposed by Chollet himself in a 2019 paper (On the Measure of Intelligence - arxiv.org) as a way of measuring general intelligence. The challenge is to find patterns in pairs of figures so that you can infer what the next figure should be.

The following illustration shows an example of the figures used in the test. They are drawn on a grid, and there is a pattern that relates the input figure to the output figure. The task is to identify that pattern so that we can determine what the output should be.

Example figures from the ARC test. Can you identify the pattern and guess what the next figure should be?

Can you spot the pattern in the three examples? What would the resulting figure be?

It is not difficult for humans, who can easily solve 85% of the tasks. The best AI systems, however, do not get past 34%.

In this case, the solutions are the following:

• In the pairs on the left, the pattern is: "fill the gap inside the purple squares with a yellow cell". The solution would be four purple squares with yellow cells in the corresponding corners.

• In the middle pairs, the pattern could be expressed as: "move the light-blue cells that sit above the dark-blue ones down to the bottom line". The solution would be a figure with a bottom row of dark-blue cells, except that cell 3 and the last one would be light blue. Above each of them there would be a dark-blue cell.

• In the pairs on the right, the pattern could be described as: "rotate the figure 180 degrees". The solution would be a figure with a first row gray-gray-blue, a second row blue-gray-gray, and a last row blue-blue-blue.

The competition is hosted on Kaggle (ARC Prize 2024 - kaggle.org), and the prize will go to whichever team manages to build an AI system that matches human performance and reaches 85% accuracy. The submission deadline is November 11. There are already almost 150 teams submitting algorithms (Leaderboard - kaggle.com), and for now the best team is at 38%.

Will someone solve it this year? Will the prize remain unclaimed? Will we be closer to AGI by the end of the year? We will know in a few months.

4. Swift is my favorite programming language. It is modern, safe, multiparadigm, expressive, and efficient. Now, in June 2024, it has just turned 10 years old and version 6 has been announced. A lot has happened since Chris Lattner, its main creator, introduced it on June 2, 2014 at WWDC14.

The following year, toward the end of 2015, we decided to include Swift in our second-year Computer Engineering course, Programming Languages and Paradigms, replacing Scala, which we had been using to explain what a modern language looks like when it integrates the functional programming paradigm with the object-oriented one. Swift had been designed from the beginning with exactly that idea in mind, integrating concepts from functional languages such as Haskell and object-oriented languages such as C# or Objective-C. We also confirmed that Apple had kept Lattner's promise that the language would be open, and that there was a Linux compiler that worked perfectly, so there would be no problem in using it for course assignments.

Within just a few years Swift evolved through several versions (Swift History - wikipedia.org). Starting with version 4, in 2017, the language stabilized and its basic syntax stopped changing very much. In recent years there have been continuous improvements through the Swift Evolution process. Every change starts as a proposal published on a public forum, gets discussed and modified, and is eventually accepted or rejected. Accepted proposals are added to the list of approved proposals (swift-evolution - github.com) and then implemented in the compiler (swift - github.com) so they are ready for the next release.

Version 5 of the language was released in 2019 and since then there have been many changes in minor releases. With the regularity of a clock, Apple has shipped two minor versions per year: 5.0 and 5.1 in 2019, 5.2 and 5.3 in 2020, 5.4 and 5.5 in 2021, 5.6 and 5.7 in 2022, 5.8 and 5.9 in 2023. This year version 5.10 has been released, and at WWDC Apple presented Swift 6 (WWDC24: What's new in Swift - apple.com), which will ship in September.

On June 12, the ATP podcast released a very interesting special episode interviewing two Apple engineers responsible for the language and its evolution: Holly Borla and Ben Cohen.

Today Swift is a very broad and complex language, capable of everything from concurrent programming using actors to programming very low-level microcontrollers. But it is also a language that supports gradual learning. It is very easy to start with, and it could perfectly well serve as the first language in an introductory programming course. All the necessary documentation is in The Swift Programming Language, starting with the tutorial A Swift Tour.

My two weeks

I already own the domain domingogallardo.com, but there is nothing there yet, just an nginx server. I promise I will start building the website in September.

Films

As soon as it premiered on Netflix, we watched Godzilla Minus One (letterboxd) on our Friday film night.

Great poster for Godzilla Minus One.

My expectations were very high, and it disappointed me a little. I was expecting a somewhat deeper film, with more serious themes. I had read that it explored the parallel between Godzilla and the atomic bombs dropped on Japan, but I did not really see much of that; it felt to me like another Godzilla movie, a bit militaristic and not aiming for much beyond that. That said, the postwar Japan atmosphere is very well done and the special effects are excellent.

I still liked it a lot and it left me wanting to see the next one. It seems the director, Takashi Yamazaki, wants to continue the saga.

TV

Two series stood out for us this fortnight: Muertos S.L. (filmaffinity) on Movistar+, and season 3 of The Morning Show (filmaffinity) on Apple TV+.

I must be one of the few Spaniards who had never watched anything by the Caballero brothers. I had not seen Aqui no hay quien viva, La que se avecina, or Machos alfa. But we tried this one and had a great time. They go a bit too far with black humor now and then, but most of the time it made us laugh out loud.

Excellent cast, with hilarious characters you recognize instantly. Everyone is great, and together with very entertaining scripts they make it the comedy that has made me laugh the most lately.

I also really liked the latest season, season 3, of The Morning Show. I thought it was the best season so far, and it keeps getting better episode by episode until the final fireworks.

The acting is excellent across the board. I love Billy Crudup, who takes Cory Ellison to remarkable places, and also Greta Lee, who brings great nuance to Stella Bak.

Books

I finished The Dark Forest, from Liu Cixin's trilogy. I had already read it a few years ago and reread it because of the Netflix series. I liked it a lot again, especially its scientific ideas and speculations. That said, the characters and their relationships are stereotyped and dated; it feels like a book written in the 1950s. But the scientific side and the suspense make up for that. It has extremely original ideas and scenes that will be amazing to see on television. I cannot wait to see "the droplet".

DALL-E drawing of "the droplet" from The Dark Forest.

For now I have paused the trilogy. I will return to it when the next season of the Netflix series gets closer. I still have the last book left, which I also read a long time ago and which was the one I liked most (I gave it 5 stars on Goodreads).

The book I have just started is the continuation of Dune, Dune Messiah, by Frank Herbert. Twelve years have passed since the end of Dune and Paul Atreides has consolidated his power as Emperor and spread his jihad across the galaxy.

I am only a couple of chapters in and it is going very well. Let's see if by the next issue I have finished it.

See you in the next fortnight.

Let’s get to it. Thank you very much for reading.

News

1. Jose Maria Fernandez Gil is a star. He is a specialist in digital accessibility and an application developer. He has worked at the University of Alicante since 2009, first at the CAE (Student Support Center) and later as head of the Digital Accessibility Unit. For fifteen years he has been developing applications that help people and promoting the accessibility of the university’s websites and online services.

Jose Maria giving a demonstration at the University of Alicante’s Digital Accessibility Unit.

On May 21, after more than a year of development, he introduced a subtitling and transcription tool, an internal application, Subtitulado y transcripcion - ua.es, that automatically generates subtitles for videos published on the university platform.

I got to know the application through a beta version that Jose Maria shared with me. I tried it and was surprised by how well it worked and how useful it was. In just a few minutes, after correcting a few transcription mistakes that had not come out right, usually a proper name or a reference the model did not know, I had fully subtitled a video more than 20 minutes long. Without the tool, that would have taken me several hours.

Screen of the subtitling tool.

According to Jose Maria himself, the application uses the open speech-recognition model Whisper and, in the testing phase alone, had already subtitled more than 2,000 videos and 1,500 hours of video.

Now that it is in production and integrated into the university website, it will be a hugely valuable resource. It will make it possible for all videos created by University of Alicante staff to be accompanied by subtitles and thus be accessible to deaf and hard-of-hearing users. And not only to them, but to everyone. Accessibility options make interaction easier in many situations, and sooner or later all of us end up using them.

Congratulations on the great work.

2. On May 20, Microsoft presented, at a special event called Introducing Copilot+ PCs (microsoft.com and youtube.com), a new version of its Surface Pro tablet-laptop with Qualcomm’s new ARM Snapdragon X Elite chip (The Verge and another first look from The Verge). This tablet is Microsoft’s answer to Apple’s laptops. Qualcomm’s ARM chip, together with the ones announced by other manufacturers, is meant to compete with Apple’s ARM chips, which have shown excellent performance in power use and computational efficiency. We will see whether this version of Windows for ARM finally becomes popular.¹

Microsoft’s new Surface Pro, with a Qualcomm ARM chip (photo from The Verge).

A summary of the tablet’s features can be seen in Microsoft’s ad. Maybe it is because I am used to Apple’s ads from the Jonny Ive era, but the style felt like a copy of those years.

AI may be crucial to the adoption of these ARM computers. Satya Nadella focused heavily on Windows features related to AI and introduced the new category of compatible machines called Copilot+ PCs (The Verge), also based on ARM architecture and Qualcomm’s chip.

Satya Nadella presenting the new “Copilot+ PC” brand.

Nadella leaned hard into the idea of the “intelligent” computer, one that understands us and helps us through a conversational interface. Microsoft’s idea is that of the intelligent “copilot” that watches what we are doing and to which we can ask for help at any moment. In his own words, at the start of the event:

“The new user interface [enabled by AI] is multimodal and can support text, images, video, both as input and output. We are going to have that. We are going to have a memory that retains important context, that remembers our knowledge and personal data across apps and devices. And we are going to have new reasoning capabilities to help us complete complex tasks.

We are entering this new era where computers not only understand us, but can also anticipate what we want and our intentions.”

To be honest, watching the presentation, they did not reveal anything extraordinary related to these copilots. It felt more like a declaration of intent and of future ambitions. The “Copilot+ PC” brand, a rather bad name, is basically a marketing device to define a minimum set of hardware specifications, memory, storage, processor, that computers must have in order to support the new AI features that will be introduced in Windows 11.

Among those features, besides applications for translation and automatic transcription in video calls or for generating and retouching images in photos, the most notable one was the Windows feature they called Recall. They spent more than half of the presentation talking about it.

It is a utility that continuously records the user’s interaction with the computer and then responds intelligently to any query. For example, imagine that during the last week you visited the page of a hotel while planning a trip, but you do not remember which page exactly. You can ask Recall and it will answer. And the same with anything else you want to recover from something you did on the computer: an email you answered, a video you watched, or a news article you read (Ars Technica).

The Recall feature in Windows 11 will make it possible to record everything you do on your computer and then search that history using AI.

The application has generated some controversy because of the privacy implications (The Verge and Ars Technica).

Microsoft says that the entire process is local, happens on the computer itself, and that Microsoft has no access to anything the computer records. But many people find that difficult to trust. Can the same be guaranteed for the PC manufacturer? Who can assure me that Xiaomi or Dell, to name just two manufacturers, will not access my history? For now it is all still an announcement, and the feature has not yet gone into production. We will see whether it finally ships and under what restrictions.

I find the feature incredibly useful, and I would love Apple to incorporate it into macOS. Mac users have Rewind, which does something similar. But the truth is that I do not trust a startup with the security of my data. Apple, please buy it and integrate the product into macOS.

3. Speaking of Apple and AI-related features, next week a flood of news is expected at the opening event of its annual developers conference, WWDC24. Right now we still do not have any answers, only many questions.

WWDC24 is almost here.

In the Upgrade podcast, Jason Snell and Mike Hurley do their draft of what the event might include. Very entertaining.

My questions:

• Will they present a local LLM that runs on the phone? What will they use it for?
• Will the agreement with OpenAI be confirmed? What will it consist of, access to GPT-4 or to some other specific model built ad hoc for Apple? In which service will Apple integrate that access? Will it be some free feature of the new version of iOS or part of a paid service? Where will those model queries be integrated, in Siri, in one specific app, or diffused across many “intelligent” features of iOS?
• Will there be a copilot for Xcode, trained for Swift and SwiftUI? Will there be a copilot for Keynote, for example, that automatically creates slides from an existing text, in the way Microsoft is already showing?
• What will Siri’s approach be? A conversational agent that can search the web and summarize the information it finds? That would amount to competing directly with OpenAI, and I do not think Apple will go that way. Or an agent that can use the different apps on the phone? I think that is more likely. At least that is what Gurman has been reporting. I think they will use all the infrastructure they already have with Shortcuts and integrate it into a language model that uses Siri.

We will know the answers very soon.

4. Google has started integrating AI-generated summaries, called AI Overviews, into search results. It has been doing so very cautiously, testing the feature. Several failed answers have gone viral, including the famous “put glue on the cheese for pizza” one (Ars Technica).

Image generated by DALL-E showing the incident in which an AI suggests using glue on pizza.

Google responded on May 30, apologizing, saying that these are exceptions, that the system is working, and that in any case everything will keep improving (AI Overviews: About last week - blog.google):

“We have been carefully monitoring feedback and external reports, and taking action on the small number of AI Overviews that violate content policies. This means overviews that include potentially harmful, obscene, or otherwise policy-violating information.

We found a content policy violation in less than one out of every seven million unique queries in which AI Overviews appeared.

At web scale, with billions of queries every day, some oddities and errors are inevitable. We have learned a great deal over the past 25 years about how to build and maintain a high-quality search experience, including how to learn from these mistakes to improve Search for everyone. We will continue improving when and how we show AI Overviews and strengthening our protections, including for edge cases, and we are very grateful for the ongoing feedback.”

We still do not know whether Google is definitely going to push in that direction. It is a very large strategic shift, and Google has to move carefully so as not to cannibalize its own web traffic and that of partner sites, which could hurt its business model and its relationship with content creators.

Antonio Ortiz has addressed the issue several times in Error500 (La crisis de Google antes de su verdadera crisis and Google tiene que decidir a quien va a perjudicar con su cambio en el buscador. Y rapido) and it is also summarized very well in the latest episode of Monos estocasticos.

“I think Google still has not fully grasped the mess it is getting into.

Media outlets that are going to come after you, content creators to whom you no longer send traffic, the disappearance of that traffic because you respond to the user and they no longer have any reason to come to my page.

The underlying problem is that, with this leap, Google is making such a change of identity, such a change in its role with respect to information and content, it changes its role so much in economic terms, political terms, and in the responsibilities it assumes, that I think all the processes, culture, and technology you had for the previous role have to be rethought.

This is such a huge leap, if you will allow the expression from the last century, that it will shake Google’s foundations if they continue down this path.”

As for the incident, to me it is a clear case of cherry-picking driven by the dynamics of social media, which favor making these small incidents, sometimes even invented ones, go viral rather than promoting an objective and unbiased analysis.

As a final element for thinking about Google’s future, in the Decoder podcast, Nilay Patel, editor-in-chief of The Verge, interviews Sundar Pichai.

Patel is quite inquisitive and it is obvious that he is not a neutral observer in the matter, he represents a media outlet whose revenue could be affected. But Pichai defends himself well and seems quite convinced that the future of Google’s business model depends on integrating AI into search results:

“People are responding very positively. It is one of the most positive changes I have seen in Search, based on the metrics we look at. People engage more when you give them context. It helps them understand better, and they also engage with the underlying content. In fact, if you include content and links inside AI-generated overviews, they get higher click-through rates than if you put them outside those overviews.”

5. I am interested in the Vision Pro as a technological novelty because of the way it integrates advanced technologies and algorithms that Apple has developed and that developers can use in the form of APIs. We saw many details of that in last year’s WWDC23 (VisionOS videos from WWDC23 - apple.com) and we will probably see much more in the upcoming WWDC24.

But I am especially interested in it because of the new kinds of experiences that can be created for it.

Simulated image of an augmented-reality application that provides additional information about an article being read.

Both computational and information experiences, such as the one shown above from the paper RealitySummary: On-Demand Mixed Reality Document Enhancement using Large Language Models (paper on arXiv and Andy Matuschak’s post on X), with an example of how augmented reality can be used to complement reading a document, and immersive entertainment experiences, like the ones we have already seen other times in this newsletter.

Along those lines of immersive experiences, on May 30, Marvel released its first “immersive story” for the Vision Pro: an episode of the animated series What If…? almost an hour long. Produced by ILM Immersive, a division of the famous Industrial Light & Magic, it is an experience that combines 3D film, immersive scenes, mixed-reality scenes, and first-person interactions.

Wes Davis from The Verge was not too impressed (The Verge review), but Jason Snell was convinced by it (Six Colors review):

“It is not immersive video and it is not a game. It is something in between, a mixed-media experiment, about an hour long, that tries to use all the features of the Vision Pro to create a respectable entertainment experience. […] It is difficult to judge What If…? in its entirety, because it really does seem like a sample of how this type of entertainment could evolve in the future. Is there room for something that is more interactive than watching television but less interactive than a full videogame? I have no idea. But I do know that the hour I spent with What If…? was maybe the best hour I have spent with the device since I got it. If Apple is looking for one app that demonstrates all the Vision Pro features at their best, What If…? might be the answer.”

In episode #1931 of the Voices of VR podcast, executive producer Shereif Fattouh and art director Indira Guerrieri are interviewed about the process of creating the story:

Shereif Fattouh:

A lot of the textures for the assets were already done from the seasons, so basically we had to take all these assets and make sure they somehow existed in space, in a 3D space, which really forced us to think about how we were going to preserve the 2D look or how we were going to blend it with the immersive aspects. And we ended up with a kind of combination of being really faithful to the beautiful artwork, the beautiful work that had been done, and adding the spatial dimension by making some elements a bit more realistic, such as the immersive environments.

Indira Guerrieri:

Even though we are exploring a new medium, a lot of this already exists in traditional games, especially when you are making immersive or cinematic content for narrative-driven games. The balance lies in how interactive it is going to be, the level of agency you have in that interactivity, and how much of the story rests on the player’s shoulders. In this case, we are not making a game but a new form: an immersive story, fully subjective, with certain interactive elements.

We can watch the full experience in videos that many users have uploaded to YouTube. For example, the complete gameplay by the YouTuber Nathie, Marvel’s: What If Experience On Apple Vision Pro Is A Blast! (Full Gameplay), reached almost 40,000 views in three days. And user iBrews also presents it in full, WHAT IF? in Apple Vision Pro - lots of commentary, with many more comments and exhaustive interaction tests, and the full end credits, which show just how many people took part in creating the story.

Below we can see a few short clips with examples of the different elements they use in the experience.

It starts in mixed reality, with the Watcher and Wong, characters who guide us through the story. Here we see Wong coming out of a portal and stepping into our reality:

3D films are one of the most interesting features of the Vision Pro. The crystal fragments are used very cleverly to project animated stories onto them in three dimensions.

Immersive experiences are static environments in which we watch a scene unfold in front of us. The settings extend around us and we can look to the sides and upward, depending on where the scene is happening. Sometimes the characters approach us and we see them right next to us. The sense of immersion must be incredible. That said, we cannot move forward or backward, only turn our heads.

In summary, it seems to me an enormously high-quality production that is going to set the standard for a long time for future experiences made for the Vision Pro. It has managed to do something very difficult, combining all the possibilities of the device into a unified experience almost an hour long.

I cannot wait for them to start selling it in Spain so I can drop by the Apple Store at La Condomina and try it.

6. On May 20 I received the issue Setting the Stage for “Mulan” from the wonderful newsletter Animation Obsessive. It explains the enormous contribution of production designer Hans Bacher to Mulan (1998). Without him, the film would have been completely different.

Bacher defined a unique visual style that evoked traditional Chinese paintings while preserving Disney’s essence. His initial approach, called “poetic simplicity”, emphasized minimalism and clarity, drawing on Chinese artistic traditions in which landscapes often represent simplified forms and leave details to the viewer’s imagination.

One of Hans Bacher’s early designs.

Concept design of the Hun attack.

The studio executives, however, considered the early designs too simple and asked for more detail and more work on the backgrounds. That led to a creative balancing act between artistic simplicity and the richer visual expectations of Disney animation.

Bacher’s influence went beyond the conceptual vision. He actively guided the film’s art direction through exhaustive style guides that set out fundamental principles for shot composition. Those principles included maintaining a balance between busy and quiet elements, straight and curved lines, and positive and negative space. A PDF version of those guides is available online (Hans Bacher, 1995: Mulan Style Guide - archive.org). Bacher has also collected many scenes and early designs on his blog (One1more2time3’s Weblog).

Illustration from Bacher’s style guide for Mulan.

His insistence on these stylistic rules ensured a cohesive look throughout the film, making every scene visually attractive and narratively clear. By correcting designs and backgrounds, and by leading the art team, Bacher’s vision was crucial in shaping Mulan into a film that was not only a commercial success, but also a richly visual work of storytelling that resonated with audiences around the world.

A gallery with some scenes from the film:

Mulan is one of the Disney films that we have watched the most and that the whole family loves the most. I like everything about it: the designs, the colors, the backgrounds, the editing, the animation, the characters, the story. The combination of computer animation and traditional animation is also very interesting and works very well. The computer graphics are very subtle in some scenes and spectacular in others, like the Hun attack.

I was so fascinated by the film that when I saw its art book at Ateneo I rushed to buy it, despite the 10,000 pesetas and a bit that it cost, more than 60 euros, without even adjusting for inflation.

The Mulan art book.

It is a jewel, beautiful. It was one of the first art books for films that I bought. More came afterward, but this is still one of the ones I treasure most. In case you want to see what is inside, here is a video:

My two weeks

Let’s go straight to films and series. Nothing new as far as books or other projects are concerned.

Movies

Looking back through my Letterboxd, I would highlight from these two weeks Furiosa, the continuation of the Mad Max saga and, above all, the prequel to Fury Road. Directed by George Miller at the age of 79 and starring Anya Taylor-Joy and an unrecognizable Chris Hemsworth, it seemed to me a hugely entertaining film, with spectacular photography and landscapes and excellent action sequences. Characters you empathize with, and a very rounded origin story that explains Furiosa’s whole previous life until she becomes the heroine of Fury Road.

TV

Of the series we watched during the fortnight, I would highlight season 2 of Bosch Legacy on Prime. Starring the always reliable Titus Welliver, the young Madison Lintz, who does an excellent job as his daughter Maddie, and Bosch’s ever-faithful allies, Mimi Rogers and Chang.

The series begins with a couple of opening episodes that are pure adrenaline, with Harry trying to find the missing Maddie. The rest of the season continues in line with the previous ones and gives us everything we like about the series. And it ends with a great twist in the last five minutes so that things do not lose momentum ahead of the already confirmed season 3.

See you in the next fortnight.

Here we are, another Friday, with the novelty I promised in my last issue: a special issue.

Why a special issue? Because of my usual tendency to dig and accumulate browser tabs. In this case, I started looking into the Herculaneum papyri as a possible news item for the late-April issue. But one website led me to another, and another, and another, and in the end the whole thing got out of hand.

Let us get to it, and thank you very much for reading me.

On April 25 I read an intriguing headline: “A Herculaneum papyrus reveals the precise location of Plato's tomb” [Deciphered Herculaneum papyrus reveals precise burial place of Plato - arstechnica.com]. I had already read or heard about these papyri before, for example in Maria Ribes's explanation in episode 451 of Coffee Break. But I had never looked into them in depth. I wanted to understand the story properly, started searching the web, and ended up finding a fascinating story that I had to share: how an Artificial Intelligence competition has made it possible to reveal writings contained in papyri carbonized by the eruption of Vesuvius more than 2,000 years ago.

And, by the way, it turned out that the original news item had nothing to do with the papyri we are going to talk about here.

Nat Friedman's post on X about the news of Plato's tomb and the Herculaneum papyri.

Origin

What are these papyri? What happened in Herculaneum?

The story begins in the 1st century BC, in the Roman city of Herculaneum. A wealthy Roman nobleman, Lucius Calpurnius Piso Caesoninus, who would become the father-in-law of Julius Caesar, built a luxurious villa on the outskirts of the city, the so-called Villa of the Papyri [Villa of the Papyri - wikipedia.org].

It was a large residence, full of artistic elements such as frescoes, mosaics, and sculptures, and equipped with a large library containing a vast collection of classical texts, in Greek and Latin, covering a wide range of philosophical and literary topics. These texts were written on rolled papyri, stacked horizontally on shelves.

All that wealth would have disappeared had it not been for the eruption of Mount Vesuvius in 79 AD. The eruption covered Pompeii in ash, and Herculaneum as well. This ash, paradoxically, acted as both destroyer and preserver. The lava and ash that buried the villa created a time capsule that preserved the papyri along with other valuable objects. This phenomenon made it possible for archaeologists and specialists, almost two millennia later, to rediscover those treasures.

The following video shows how the flood of ash and hot volcanic material covered the rolls and carbonized them.

Discovery and first attempts to read the papyri

The 1908 book by archaeologist Ethel Ross Barker, Buried Herculaneum [Buried Herculaneum - archive.org], describes in detail the history of the excavations at Herculaneum and the discovery of the buried treasures and the papyri. In the mid-18th century, the villa and its surroundings began to be excavated, and in the autumn of 1752 the first finds appeared: 21 rolls and fragments contained in two wooden shelves. They looked like burnt charcoal logs, and some had been mistaken for exactly that and thrown away. In the following years many more were found, up to a total of 1,806. Of those, 341 were complete rolls, 500 were fragments, and the remaining 965 were in an intermediate state of preservation.

The following photographs show different examples of these rolls and fragments.

In many of the remains, the writing contained in the papyri could still be seen, as in the example shown in the next image.

Other complete rolls were separated by cutting them in half or carefully unrolling them, turning them into fragments that researchers then tried to classify and reorder, as happened with papyrus no. 10.

As a result of the analysis of all these remains, it was possible to determine that the great majority of the papyri are Greek texts belonging to the Epicurean philosopher Philodemus of Gadara, a 1st-century BC philosopher who lived in the region. These works deal extensively with topics related to ethics, poetry, music, and logic. The collection also includes around twenty Latin papyri that have not yet been deciphered.

Scrolls still awaiting decipherment

More than 1,000 rolls and fragments still remain to be deciphered. Many of them are completely carbonized rolls that look like lumps of charcoal. That is the case with PHerc 1667, an intact part of the interior of a roll, with an approximate diameter of 3 cm and a length of 8.5 cm. The outer parts of the roll were separated from it in an attempt to “unroll” it.

Papyrus PHerc 1667, image taken from the technical notes for the 2023 data capture.

Another example is roll PHerc 332, where you can make out the individual rolled layers that compose it. This papyrus measures 7.7 cm in length and 2.6 cm in diameter.

Papyrus PHerc 332, image taken from the technical notes for the 2023 data capture.

And one last example of another roll, deformed and solidified by carbonization.

Image taken from Brent Seales's video: Herculaneum scrolls: A 20-year journey to read the unreadable.

At first glance it seems impossible to extract the slightest information from these carbonized blocks. It does not seem plausible that the ink could have survived that carbonization. But what if we perform a tomography of the papyrus? Could we reconstruct its interior and examine it without damaging it? Could we find traces of ink and decipher the writing?

First tomography of one of the papyri

Brent Seales, professor of computer science at the University of Kentucky [Brent Seales - uky.edu], tried to solve the problem in 2009. Together with his team, he traveled to the Institut de France to perform the first micro-computed tomography of one of the rolls. The result is a sequence of scanned images like the one below, obtained at a resolution of 14 micrometers, 0.014 mm.

The inside of the roll seems to have been preserved, and the layers of the rolled papyrus can be observed, but not with enough definition to separate them automatically and find ink. The problem was simply too complex. In their 2011 article [Analysis of Herculaneum papyri with x-ray computed tomography - scholar.google.com], they conclude by saying:

We have encountered serious difficulties in analyzing the data because of the complex nature of the papyrus's internal structure. Automatic separation of the papyrus layers has proved virtually impossible. A manual reconstruction of a small region was attempted, but it was not possible to make the ink visible.

But Seales is not discouraged and remains convinced that the approach is the right one and that it will eventually work. What is needed is more resolution and better algorithms. And the approach also needs to be validated on a simpler problem.

Brent Seales deciphers the En-Gedi scroll

In 2015 Brent Seales and his team showed that the answer is yes in the case of the En-Gedi scroll [En-Gedi scroll - wikipedia.org]. The scroll is made of animal skin, specifically leather, unlike the Herculaneum rolls, which are papyrus. It was discovered in 1970 in a synagogue in En-Gedi, Israel, and dates from the 3rd or 4th century AD.

The En-Gedi scroll had also been carbonized and was found in a very fragile state. Even so, Seales's team managed to apply its method, demonstrating that it was possible to virtually unroll it from its three-dimensional tomographic image.

Professor Seales explains it very well in the following video.

We extracted a few fragments from the video to detail the phases of the process.

1. First, a three-dimensional scan is made using a micro-computed X-ray tomography technique, with micrometer resolution. This yields a three-dimensional volume of the scroll and its interior.

2. Then a line is selected in the three-dimensional image corresponding to a section containing one sheet of the parchment, and a patch of the parchment is reconstructed. In that reconstruction the writing can already be seen.

3. The same process is repeated for all possible patches. Once they have been obtained, overlapping regions are checked and fitted together, completing a kind of puzzle that reconstructs as much of the parchment as possible.

The success of the project led to the publication of several scientific articles [From damage to discovery via virtual unwrapping: Reading the scroll from En-Gedi - science.org] and to a $14 million grant from the National Science Foundation [UK Awarded $14 Million NSF Grant to Launch World-Class Cultural Heritage Lab - uky.edu] to create the EduceLab center, led by Seales himself, with the fundamental goal of reconstructing and preserving texts from antiquity and, specifically, the Herculaneum papyri.

Detecting ink in a papyrus fragment

The techniques that worked on the En-Gedi scroll, however, did not produce good results on the Herculaneum papyri. For one thing, the material was different. Papyrus is a kind of woven plant material, and until then no one had succeeded in recovering ink from carbonized fabric. In addition, the papyri were much more tightly rolled than the En-Gedi parchment, and the layers to be virtually unfolded were much more intricate.

In 2016 two newly graduated computer science students joined the project, Seth Parker and Stephen Parsons, and they ended up being decisive for its success. The first specialized in processing the data obtained from the X-ray tomography, while the second focused on 3D reconstruction and machine learning with neural networks.

Parsons's goal was to obtain a convolutional neural network capable of extracting preserved ink in papyrus from the volumetric data of a 3D scan. To train the neural network, they used separated papyrus fragments in which the ink was visible, together with their volumetric 3D scans.

Training data for the neural network that recognizes ink in papyrus. Stephen Parsons's doctoral thesis.

After numerous attempts, Parsons managed to develop a model which, when applied to these papyrus fragments, produced promising initial results. The following figure shows the final output of the neural network on the left, where the white points represent ink. There is still a lot of noise, but some letters are correctly identified.

In 2019 a new scan of a complete roll was performed at the United Kingdom's particle accelerator Diamond Light Source. Using more energetic X-rays, they obtained a scan with a resolution of 8 micrometers, 0.008 mm, almost twice the resolution of the first scan.

Using Parker's software, the team improved the tools used to process the data and worked out a possible workflow for reconstructing part of the roll. It is very similar to the one used for the En-Gedi parchment.

1. The layer of the roll to be virtually unwrapped is selected manually:

2. After selecting that layer in consecutive slices, a surface corresponding to a fragment of papyrus is obtained:

3. Finally, that surface is expanded by thickening it with the data above and below in the original scan, producing a surface with some volume:

4. And it is on that volume that the neural-network model is applied, in the hope of detecting the ink points:

The problem was that, despite all the work and the high resolution of the data, the model did not perform well enough to extract a complete word. Even so, it represented a robust starting point from which that goal could eventually be reached. As Parker and Parsons wrote at the end of the article they published in 2019 [From invisibility to readability: Recovering the ink of Herculaneum - plos.org]:

With the proven ability of our machine-learning pipeline to detect the carbon-ink signal and render it photorealistically, the scholarly community may be one step closer to witnessing “a bursting forth of genius from the dust”1 from Herculaneum.

Although they did not manage to decipher any words from the roll, Parsons and Parker laid the groundwork for the next advances. And, most importantly, they also finished their doctoral theses: Parsons in 2023 [Hard-Hearted Scrolls: A Noninvasive Method for Reading the Herculaneum Papyri] and Parker in 2024 [Flexible Attenuation Fields: Tomographic Reconstruction From Heterogeneous Datasets].

The Vesuvius Challenge competition

At the end of 2022, the computer scientist and entrepreneur Nat Friedman [Nat Friedman - wikipedia.org, nat.org] learned about the project, got in touch with Seales, and proposed making all the datasets public and organizing a competition to improve the neural-network model and, eventually, to read complete scrolls.

Nat Friedman's first post on X referring, without naming it, to what would become the Vesuvius Challenge.

The idea of organizing a competition to obtain or improve an AI model is a common one in the field. For example, the website Kaggle, founded in 2010 [Kaggle - wikipedia.org], has organized hundreds of competitions in which tens of thousands of enthusiasts and specialists have participated.

Friedman, who had led major software projects and companies such as Ximian, Xamarin, and GitHub, knew that the competition needed to be organized very carefully, monitoring its development and structuring it in such a way that collaboration among participants and the sharing of results would be encouraged. Offering an attractive prize was not enough, it was initially $500,000, $250,000 from him and $250,000 from entrepreneur Daniel Gross [dcgross.com], it also had to be managed meticulously and every detail supervised closely.

In November, Friedman published a call looking for a technical lead for the project [Hiring tech lead to help solve major archaeological puzzle - nat.org] and ended up hiring JP Posma [I can announce it now - x.com], who organized the competition website and set up its presence across the different social networks where it would be launched.

Finally, on March 15, 2023, the Vesuvius Challenge website was launched.

Current homepage of the Vesuvius Challenge competition website.

That same day the competition was also launched on several social platforms, Discord and X, a newsletter was created, and the competition opened on Kaggle.

In a very short time the competition became highly popular, donations rose to more than a million dollars, and many participants were drawn to work with the tools and data made available.

To win the final prize of $700,000, competitors had to decipher, before December 31, 2023, four separate passages of text, each containing at least 140 characters of continuous text. But, in order to foster cooperation, “progress prizes” of between $1,000 and $10,000 were also proposed every two months. To win these prizes, participants had to publish their code or research openly, thereby benefiting the progress of the whole community.

• On April 15 the first four open-source prizes were awarded, $2,500 each, for the creation and improvement of tools and contributions to the community.

• On June 27 progress prizes were awarded for segmentation contributions and the automatic extraction of papyrus patches, one of the most difficult problems. It was decided to hire “segmenters” to extract different patches and make them available to the community.

• In the following months quite a few progress prizes were awarded, but it did not seem that clear progress was being made. Until, on October 13, 21-year-old student Luke Farritor found the first word in one of the papyrus pieces: ΠΟΡΦΥΡΑϹ, porphyras. It means “purple,” and it is a rather rare word in ancient texts.

  

  First word found in one of the Herculaneum papyrus rolls.

  Luke himself tells the story of the excitement of the discovery in this video.

• After Luke's success, Youseff Nader, an Egyptian PhD student in Berlin, tried a new neural-network model on the same patch where Luke had found the word. He used Luke's results to reinforce the model's learning and obtained a surprising result: an image in which the word previously found appeared much more clearly, and two more words could also be read, one above it and one below it.

  

  Image resulting from applying Youseff's neural network to the same fragment in which Luke found the first word.

• On February 5, 2024, the winners of the $700,000 grand prize were announced: a super-team formed by the previous winners, Youseff and Luke, together with the young Swiss participant Julian Schilliger, managed to obtain 15 columns, 11 more than were required, and more than 2,000 characters in total. This represents only about 5% of the total content that must be present in the papyrus.

The challenge had been achieved. Two thousand years after being buried by the eruption of Vesuvius, three young people had read for the first time one of the carbonized papyrus rolls. They had shown that the project Brent Seales had begun more than fifteen years earlier was viable. That carbonized papyri could be virtually unwrapped and read with a neural network.

The following figure is the image submitted by the winning team.

Image with the reconstruction of the papyrus text provided by the winning team.

A team of experts assembled by the Vesuvius Challenge studied the text, transcribed it, and translated it, validating that its content is related to the collection of Epicurean texts found in the Villa. On the competition page you can find the fragments transcribed into Greek and translated.

Column from the reconstructed image, with the text highlighted after processing, and its Greek transcription on the right.

The three winners of the challenge received the main prize, but there were also many additional prizes, not only the “open source” and “progress” awards, but also prizes for three other finalist teams, who each received $50,000.

Winning team of the final 2023 prize in the competition.

The complete list of prizes awarded so far, up to April 2024, amounts to $1,236,500. On the competition website you can find the full list, with links to the GitHub pages corresponding to each award.

Celebration and future

On March 16, 2024, an event was held at the Getty Villa in Malibu [Getty Villa - wikipedia.org] where the prizes of the Vesuvius Challenge competition were awarded. The villa, designed by billionaire Paul Getty and inspired by the Villa of the Papyri, is a museum devoted to the study of Greek, Roman, and Etruscan antiquities. The museum is also one of the collaborators in Brent Seales's EduceLab project.

Historians specializing in the Herculaneum papyri took part in the event, along with Nat Friedman and Brent Seales himself. It was a celebration of a success achieved through the combination of traditional methods and innovative approaches: on the one hand, collaboration among academic institutions and public funding, and on the other, a rather radical idea promoted by an entrepreneurial technologist and supported by donations from Silicon Valley, the open-source community, and the enthusiasm of young specialist hobbyists connected online.

As for the future, the Vesuvius Challenge is still ongoing, led by Nat Friedman and now with Stephen Parsons as technical director. The challenge newsletter is still active, and you can subscribe to receive updates.

Vesuvius Challenge

Mailing list for scrollprize.org

By Stephen

They continue awarding progress prizes every two months and have set a major challenge for 2024: $100,000 for the first team capable of reading 90% of the scrolls. There are also, as in 2023, $30,000 prizes for the first letters of scrolls 2, 3, and 4.

The deadline for submissions is December 31, 2024. Just like last year, the results seem to be taking time and nobody has yet managed to win any of these prizes. A certain sense of pessimism is beginning to be noticeable, but there is still plenty of time left before the end of the year. If the grand prize is achieved, it will be a historic success and a giant step toward the broader plan of scanning the remaining 300 papyri, most of them in Naples.

Nat Friedman has a long-term vision which he calls “The Master Plan”. Its final part, phase four, is to excavate the Villa of the Papyri again in order to recover the entire library, with the thousands of papyri that may still be buried there.

A very ambitious vision. We will see what can be achieved, and whatever happens, we will keep telling the story here.

Until next time, see you then! 👋👋

After a fortnight's delay and with my issues around organization, time management, and procrastination more or less under control, here I am with a new issue. This time we are reviewing not one but two fortnights.

A small announcement: from now on, I intend to publish on Fridays, after either the first or the second fortnight of each month. Most of the newsletters I read come out on a fixed day of the week, and it seems like a good idea to try that. That way, you will know that every other Friday I will show up in your inbox. And on some Fridays when it is not due, there will be a surprise. You will see next week.

A lot of news in a month. Above all, a lot of new models. Let us get to it, and thank you very much for reading me.

🗞 News

1️⃣ On April 18, Meta released its new versions of Llama, the Meta Llama 3 models [Build the future of AI with Meta Llama 3 - meta.com and Introducing Meta Llama 3: The most capable openly available LLM to date - meta.com].

Let us remember that Llama models are open LLMs, available to download, fine-tune, and use in any application. That said, if we use these models or create new models based on them, we will need to include the phrase “Built with Meta Llama 3” or include the name “Llama 3” at the beginning of the model name [META LLAMA 3 COMMUNITY LICENSE AGREEMENT - meta.com].

Specifically, Meta has published two models, one with 8 billion parameters and another with 70 billion parameters: Llama 3 8B and Llama 3 70B. According to the benchmarks published by Meta itself, these models are the best when compared with others of similar size.

They are still training the larger model, with more than 400 billion parameters, called Meta Llama 3 400B+. However, they have already shared preliminary results that place it ahead of GPT-4 and on a par with Claude 3 Opus.

Performance of the latest LLMs on the most important benchmarks [Hello GPT-4o - openai.com]

It is very interesting that these models have been built using very careful processes, for example by using high-quality datasets and texts, which have allowed them to achieve better results than much larger previous models. For example, the smaller model, Llama 3 8B, gets clearly better scores than the 70B model from a year ago, Llama 2 70B. Similarly, Llama 3 70B scores higher than GPT-3.5, with its 175 billion parameters.

Scores on the main benchmarks for Meta's models [Llama 3 Model Card - github.com].

The size of a model matters not only for performance, but also because it determines how much space it takes up. The Llama 3 8B model is a little over 13 GB in size. In its quantized version, which reduces parameter precision to save space, it takes up roughly 4.21 GB. That size would make it possible to run it on a high-end mobile device. Although there are advanced techniques that make it possible to use models from SSD storage memory [Apple Develops Breakthrough Method for Running LLMs on iPhones - macrumors.com], the most common approach is still that all the model weights need to be loaded into device memory in order to run it. For example, my iPhone 12 has 4 GB of RAM and could not run it. The latest models, the iPhone 15 family, have 6 GB in their basic configurations, iPhone 15 and 15 Plus, and 8 GB in their advanced configurations, 15 Pro and 15 Pro Max. The basic models would be cutting it very close, while the Pro models could run it. The ability to use a local LLM is surely going to become a differentiating factor in smartphones from now on.

If you want to play around with the models, you can find them on Hugging Face [Meta Llama - huggingface.co] and read the post published there as well [Welcome Llama 3 - Meta’s new open LLM - huggingface.co].

2️⃣ On April 23, Microsoft released its Phi-3 models [Introducing Phi-3: Redefining what’s possible with SLMs - microsoft.com and Microsoft’s Phi-3 shows the surprising power of small, locally run AI language models - arstechnica.com]. These are small models:

• Phi-3-mini, with 3.8 billion parameters and a 4,000-token context window, although Microsoft also introduced a 128K-token version called “phi-3-mini-128K”.

• Phi-3-small, with 7 billion parameters.

• Phi-3-medium, with 14 billion parameters.

The models are described in detail in the article published on arXiv [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone], where Microsoft shows that the smallest model, quantized to 4 bits, can run on an iPhone 14 Pro. It is curious that Microsoft tests its language models on Apple devices.

The paper also presents an evaluation of these models, showing that they achieve results on benchmarks that are similar to, or in some cases even slightly better than, other small models, and are in some cases comparable to GPT-3.5.

Results of Microsoft's small models on selected benchmarks, compared with other small models and GPT-3.5.

The models are also open and available on Hugging Face [Phi-3 family of models - huggingface.co].

An important competition is coming in the small-model space, where Apple is already taking its first steps, as the next item shows.

3️⃣ On April 22, Apple published several small models called OpenELM [Apple releases eight small AI language models aimed at on-device use - arstechnica.com and OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework - arxiv.org]. In the previous issue, we mentioned that Apple’s research department was publishing results from its first language models. Well, some of those models are already openly available on Hugging Face [OpenELM - huggingface.co] and can now be tested by the community.

These are still very basic models, with rather weak results compared with models of similar size. Even though Apple has really accelerated over the last year, it is still far behind research groups such as Meta and Microsoft, which in turn remain behind Google, Anthropic, and OpenAI.

I did a small investigation using the Wayback Machine into how the number of researchers in Apple's Hugging Face organization has evolved, with the result shown in the following chart:

Evolution of the number of researchers in Apple's organization on Hugging Face [Wayback Machine - archive.org].

The number of Apple researchers on Hugging Face has gone from just 4 two years ago to 308 on the day I am writing this. I made the chart a week ago, when there were 284 researchers. In a single week, they added another 24.

The question now is what models Apple is going to use in its new iPhone 16 lineup, which will launch next September and this time really should arrive full of AI-related features. According to Mark Gurman, Apple's strategy is to use its own models for APIs, both on-device and in the cloud, while presenting an assistant, or chatbot, that will come from an agreement with OpenAI.

Mark Gurman's post on X about the language models Apple is going to announce at the upcoming WWDC.

We will know soon, at Apple's developer conference, which will take place in less than a month [WWDC24 - apple.com], where iOS 18 and all the AI features it will include will be presented.

4️⃣ And now we come to the high point of the month: on May 13, OpenAI held a special event where it presented its new model: GPT-4o [Introducing GPT-4o - youtube.com]. After the 4 comes the letter “o”, for “omni”, not the number zero.

This is a multimodal model from the ground up. It is a project that began more than a year ago, as its director, Prafulla Dhariwal, explained.

Post on X from the director of the GPT-4o project at OpenAI.

The fact that the model is multimodal means that it was trained from scratch using text, audio, and images. Although it works with video, it has not been trained on video sequences1. Instead, it breaks down the video it is seeing into still frames and analyzes each one. It can also produce the same kinds of elements it was trained on: text, audio, or video.

The version they have put into production is the one that outputs text. In the demo at the event, they showed how the model was able to generate audio. And they are still testing image generation, by the model itself rather than external models such as DALL-E, before releasing that in production.

At the OpenAI event, the model was shown functioning as an assistant. It is able to understand what we say, the tone in which we say it, and what we show it, and it can respond with voice. A super-natural voice that expresses emotions and congratulates and encourages us. Obviously, it was trained to resemble Scarlett Johansson in Her.

Just look at how impressive this is:

As Antonio Ortiz points out [Monos estocásticos 2x17 - cuonda.com], it is worth noticing the moment when the AI makes a mistake at the beginning and says it “sees the image” before the image has actually been shown to it. What becomes overwhelming comes afterward, when it apologizes and does so with a specific, completely human intonation that conveys a certain embarrassment about the mistake.

If we watch the video carefully, we can notice that the AI produces very different intonations throughout the conversation. It is incredible that all of this is the result of a model trained only to generate the next most likely token. We have already seen that this works with text. Now OpenAI has shown that the same idea also works when the next token can be a piece of a word, or a fragment of an image or of audio. The later fine-tuning phase is also very important, in which the model's output is adapted to the preferences we want, in what is called RLHF, Reinforcement Learning with Human Feedback.

Let me stress one very important thing, because there has been a lot of confusion about it. In the current ChatGPT application it is already possible to have a spoken conversation. But that is absolutely not the same thing as what GPT-4o does. What we currently have on our devices is really three different models glued together by an app: one model recognizes speech and transcribes it into text, another model, GPT-4, takes text as input and returns text, and another model transforms that response text into speech. GPT-4o is totally different. There is only a single model, a single neural network, that receives tokens in any of the modalities and outputs other tokens in response, which can be text, audio, or even an image. The neural network itself, the model itself, is what produces the intonation, and also understands it. There is no post-processing layer or specific algorithm that converts text into audio. It is astonishing.

Another of the most important characteristics of the new assistant is its extremely low latency, it responds almost instantly, and the fact that it can be interrupted at any moment simply by talking to it. The AI is continuously listening to us and stops speaking at that moment in order to hear us and answer again.

As for text-generation performance, the resulting model is better than the latest version of GPT-4 and has gone straight to first place on all the major benchmarks and rankings. It is not the leap people were expecting from GPT-5, but there is still a lot of year left, and OpenAI will surely give us more surprises.

5️⃣ The day after the OpenAI event, on May 14, Google held Google I/O, its developer-focused event. It was obvious that OpenAI had scheduled its event as counter-programming against Google. And with a lot of success, given the impact of all the references to Her and the freshness of the OpenAI event, compared with a Google event that felt heavier, longer, and less focused.

In its keynote, Google presented many projects in progress [Google strikes back at OpenAI with “Project Astra” AI agent prototype - arstechnica.com and Gemini breaks new ground with a faster model, longer context, AI agents and more - blog.google], but few finished products. Among the most interesting things were Veo, a video generator in the style of Sora, a new version of the image-generation model Imagen 3, and Project Astra, a voice-controlled assistant very similar to OpenAI's.

Google's assistant is also multimodal and can see the environment through the phone. They have not released it as a product yet, but they showed a recorded video, not a live demo like OpenAI did, showing how it works.

Although what we see is quite spectacular, especially with the final “where did I leave my glasses?”, the interaction and the assistant's voice are not as polished as OpenAI's.

Google did not reveal many characteristics of the language model underlying this new AI. The only thing they said is that it is multimodal, like GPT-4o, but they did not provide many details and it has not yet been possible to try it.

They also presented a small update to the already existing Gemini 1.5 Pro model, which will support contexts of up to 2 million tokens, including text, images, and video. That is wild. Now that it is finally available in Europe, it is time to try it and see what it can do. It is true that it does not get as much attention as other models, Claude for instance gets talked about much more, but in the rankings it sits very high and it is one of the most advanced models around.

Even though they did not present too many real products, Google I/O was full of references to AI. The video TechCrunch put together and posted on X is very funny:

6️⃣ Speaking of rankings and benchmarks, many of them have been updated lately. Let us go through them.

The first, one of the most widely used, is MMLU, which has just launched a new and more advanced version called MMLU-Pro [MMLU-Pro - huggingface.co]. It is a collection of 12,000 questions from different fields, biology, mathematics, economics, computer science, and so on. In the latest version they have increased the number of options for each question to 10. It is produced by the TIGER-Lab at the University of Waterloo.

The following figure shows the results of the latest model evaluation, with GPT-4o clearly in first place.

Wenhu Chen's post on X with the latest MMLU-Pro benchmark results.

Another type of ranking is the LMSys arena [LMSYS Chatbot Arena Leaderboard - lmsys.org], where the score is obtained from head-to-head matchups between pairs of models. Users give them prompts, read their answers, and then choose a winner. We can see that the clear winner is once again GPT-4o, under the codename it used for a few days, im-also-a-good-gpt2-chatbot, some distance ahead of another leading group formed by versions of GPT-4, Gemini-1.5-pro, and Claude-3-opus.

Another very interesting chart is the one Maxime Labonne publishes regularly on X, in which you can see how models evolve over time. On the vertical axis he places the score obtained in LMSys-Arena, and on the horizontal axis the date the model was released. He also marks closed models in red and open ones in green. This makes it possible to analyze how the different types of models progress.

We can observe some very interesting things. For example, Meta's strongest current model, Llama-3 70B, is at the level of GPT-4 from a year ago. One year seems to be the time it takes for OpenAI's advances to diffuse into open models. I suppose the same will continue to happen in the future, and that by June 2025 we will see open models similar to the current GPT-4o. Another interesting thing to notice is that the upper curve keeps rising, which brings us to the next item.

7️⃣ The whole industry remains convinced that the scaling hypothesis [The scaling hipothesis - gwern.net] is true, and that larger models trained on larger and more varied datasets will obtain considerably better results. There are many ongoing investments, from Microsoft, OpenAI, Meta, and others, in building enormous data-processing centers, and even in building power plants to feed those centers.

For example, in the interesting interview with Dario Amodei [What if Dario Amodei Is Right About A.I. - apple.com] on the New York Times podcast The Ezra Klein Show, Anthropic's CEO shows himself to be fully convinced of this scaling hypothesis and talks about figures of 10 billion dollars to train future models:

"We are going to have to make bigger models that use more compute per iteration. We are going to have to run them for longer while feeding them more data. And that amount of chips multiplied by the time we run things on them essentially translates into a monetary value, because these chips are rented by the hour. That is the most common model for doing it. So current models cost on the order of 100 million dollars to train, give or take a factor of two or three. The models that are currently being trained, and that will come out at various points later this year or early next year, are closer to costing a billion dollars. So that is already happening. And then I think in 2025 and 2026, we get closer to five or ten billion.

So is it going to be 100 billion dollars? I mean, very quickly, the financial firepower you need to create one of these is going to exclude anyone except the biggest players."

Mark Zuckerberg, Meta's CEO, is somewhat more cautious in the interview on Dwarkesh Patel's podcast [Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus - dwarkeshpatel.com]:

"This is one of the big questions, right? [the exponential growth in model performance] I think nobody really knows. One of the hardest things to plan around is an exponential curve. How long will it keep going? I think it is probable enough that we will keep advancing that it is worth investing the 10 billion, or more than 100 billion dollars, into building the infrastructure and assuming that if it keeps going, you will get really amazing things that will make incredible products. I do not think anyone in the industry can tell you with certainty that it will continue scaling at this rate. In general, in history, you run into bottlenecks at certain points. Right now there is so much energy behind this that perhaps those bottlenecks will be overcome fairly quickly. I think that is an interesting question."

And regarding the size of future models and whether Meta will also release them openly:

"We have a roadmap of new releases that are going to bring multimodality, more multilingual capability, and larger context windows as well. Hopefully, at some point later this year, we will be able to release the 405 billion parameter model."

"Obviously we are very in favor of open source, but I have not committed to releasing absolutely everything we do. Basically I am strongly inclined to think that open source will be good for the community and also for us because we will benefit from the innovations. However, if at some point there is a qualitative change in what the thing is capable of doing, and we feel that it would not be responsible to make it open source, then we will not do it. It is very hard to predict."

👷‍♂️ My thirty days

Even though a whole month has passed since the previous issue, I do not have many updates to share. Not in books, I am still with The Dark Forest and have not progressed very much, and not in the things I have been tinkering with.

I am going to highlight the two films I liked most out of everything we watched.

📺 Two films

Two very different films, but I liked both of them a lot2.

The first is All of Us Strangers, by Andrew Haigh, with tremendous performances by Andrew Scott and Paul Mescal. A very personal and profound film, moving and thought-provoking. And with a soundtrack full of evocative songs for those of us who were young in the 1980s.

And the second is Kingdom of the Planet of the Apes, by Wes Ball. A return to adventure and to the themes I find most interesting in the saga: the formation of ape society, its relationship with humans, the inverted power relations between apes and humans, and the rewriting of history by the victors.

Some scenes reminded me of the original Charlton Heston film, and I had the feeling that the story is moving in that direction. I want more.

And that is it for this month. Until the next fortnight, see you then! 👋👋

We have already made it through the first third of the year and are fully entering 2024. We no longer hesitate when writing 2024. And this is already the 7th installment of this newsletter, which began as a happy New Year's idea ("let's see how it goes") and that I enjoy writing more and more each time.

There are less than 3 months left until Apple's developer conference, WWDC24, where it seems Apple is going to present some new developments around generative AI applications in its products. How much truth is there to that?

Let us get started. Thank you very much for reading me!

🗞 News

1️⃣ How hot is this summer going to be? I recently discovered the website https://climatereanalyzer.org, run by the Climate Change Institute at the University of Maine. They gather data every day from multiple sources and display it in a very convenient graphical format.

Two especially noteworthy pages on the site, which I have already added to my bookmarks so I can visit them every month, are the one for the evolution of surface temperature and the one for the evolution of sea temperature. You can select the part of the world you want to consult and get data, for example, for the Northern Hemisphere or the North Atlantic.

The figures show the latest data as of yesterday. Last year's record, in orange, is off the charts. That reflects what we all felt last summer: that it was the hottest one in recorded history. And the current year's record, in black, is above it, although the slope seems a little less steep and, if we are optimistic, it may not rise as much as last year did. We will have to keep checking the site to confirm it.

Why have temperatures risen so much since last year? It is not entirely clear. The temperature evolution models used to predict climate change are long-term models, and they do not help explain sharp fluctuations in a single year. One reason some scientists have proposed is the massive eruption of the underwater Hunga Tonga volcano, located in the South Pacific, in January 2022.

Eruption of the underwater Hunga Tonga volcano seen from space.

It was an extraordinary eruption that sent 146 million tons of water vapor into the Earth's stratosphere, around 10% of the water vapor already present in that layer of the atmosphere. Unlike earlier eruptions such as Krakatoa or Pinatubo, which injected gases and ash into the atmosphere that had a temporary cooling effect on the Earth's surface, water vapor has a greenhouse effect that increases temperature and causes chemical changes that may reduce ozone.

This greenhouse effect has not been studied very extensively, and there is no scientific consensus on its real impact, but it may be partly responsible for the extraordinary increase in temperature over the last few years. The effect will disappear once the extra water vapor is finally removed from the atmosphere. Although there are no clear estimates of how long that will take, experts speak of a few years. Let us hope not too many.

2️⃣ Some people are already saying Vision Pro has been a failure. Others are wondering whether they regret buying it.

YouTube video by AlenhandoroVR.

It is still far too early to pin labels on the launch. But we can start reflecting on the product and its possible evolution. I am interested in the evolution of Vision Pro for several reasons.

First, from a technological point of view, it is one of the computing devices that integrates the largest number of state-of-the-art components. Watching how the headset evolves will be almost equivalent to watching the evolution of the most advanced computing technologies: processors, displays, cameras, LiDAR sensors, and so on. On top of that, from a product design perspective, it will be very interesting to see the direction future models take over the next few years. Will Apple increase the headset's resolution at the expense of keeping its weight high? Will there be a lighter model? Will the external display showing the wearer's eyes change?

The second reason is software. What new styles of interaction are going to be created? To what extent will the device expand our capabilities? What will we be able to do with this new computational tool? I have always been interested in viewing computing through Steve Jobs's metaphor of the bicycle for the mind 1, and I think Vision Pro could be a spectacular step in that direction.

And the third reason is entertainment.

My wife and I go to the cinema almost every week. We go out for a while, watch a film, have dinner. That cannot be replaced by watching a movie with the headset, no matter how spectacular the Avengers Tower environment, the spatial audio, or the 3D effects may be. But that does not mean people will never watch films through the visor. For example, doing so while traveling by plane or train will probably become increasingly normal. Video did not kill the radio star; The Buggles got that one wrong. On the contrary: now we watch YouTube and listen to podcasts.

I am also curious about immersive experiences. Will many more of them be produced? Will we be able to watch plays, musicals, or concerts? We are not football fans, but we do sometimes watch Nadal or Alcaraz. Or the national team's basketball games. Ten years from now, will it be possible to watch them live as immersive experiences?

A spectacular feature Apple has introduced this fortnight in the new VisionOS 1.1 beta is spatial personas. Now the people you call on FaceTime no longer appear in a window. Instead, they appear next to you, in your space, in three dimensions. You can see their face and hands. You can move closer, crouch down, make gestures together, and even high-five each other.

This is what the new FaceTime video calls on Vision Pro look like. Frame from a video posted on X.

And in addition, almost as important as the above, everyone in the conversation can share and interact with apps using SharePlay . For example, they can write on a shared whiteboard or inspect 3D figures placed on it, or watch a movie or series together at the same time.

On the downside, Apple has released a short immersive film with a compilation of soccer highlights from the American league. The film has problems, because the editing is designed like a normal film, with cuts that are too fast for an immersive experience.

The future will tell us how the headset evolves. I do not share Marco Arment's opinion, from ATP, that it can already be considered a failure. It is still far too early to make such a categorical judgment, only three months have passed since it went on sale. I agree more with Siracusa, Mike Hurley and Casey Liss, or Jason Snell, who see it for now as an evolving experiment.

Apple has just given its promotion another push, publishing a press release about its use in business and putting it back on the home page, highlighting experiences such as using it while traveling by plane.

That is how products succeed: by finding success cases and then building from there. We will see how it evolves. As Siracusa says in the podcast linked above, we need to wait at least three years before calling it a success or a failure. I propose that we bet on three possible scenarios:

1. A hugely successful product, for which Apple presents multiple versions with different configurations and capabilities, similar to what the MacBook is today.

2. A product with limited but real success, a niche product. Well regarded and well supported by Apple, with frequent software and content updates, and with hardware updates spaced farther apart, similar to the Apple Watch Ultra today.

3. A product that Apple leaves without updates for a long time, with few diversification bets, little investment in software and content, and less and less success, similar to what is happening with the Mac Pro.

For now, I am moderately optimistic and I am betting on scenario 2. I will revisit this prediction in a year.

3️⃣ Still with Apple, there is a lot of buzz that at the upcoming WWDC24 they are going to launch major developments in generative AI. There are rumors of talks with Google and OpenAI, or of Apple developing its own models.

Some people are even saying that Apple is already launching LLMs, as the following tweet suggests.

How much truth is there to all this? I would like to comment on it, to explain how to tell rumors from actual news.

In hardware, we are already used to rumor cycles: Chinese supply chains leak some story based on prototype units or components, Mark Gurman more or less confirms it, a few moderately serious sites such as MacRumors, 9to5Mac, Applesfera, or Xataka pick up the rumors and identify them as such, and then everything gets out of hand in clickbait articles from outlets that sell the rumor as already confirmed news.