# Basics of LLMs and How They Work

Slides, a hands-on notebook and a no-install companion page for the
**SFB1709 Summer School** — *Cellular Plasticity in Myeloid Malignancies: From
Mechanisms to Therapies*.

A 60-minute session. Roughly **45 minutes of material** is here; the rest is
yours to fill (suggestions in [§7](#7-what-to-put-in-the-remaining-15-minutes)).

**Written for an audience that mostly has neither LLM background nor coding
experience.** The presenter demonstrates; anyone with a laptop can follow along;
anyone can take the notebook home and work through it alone. Nobody has to write
a line of code at any point.

The through-line: a language model is a machine that guesses the next word, and
everything else — the fluency, the apparent understanding, the confident wrong
citations — falls out of that. We take the machinery apart in order, with myeloid
haematology text at every step.

---

## 1. What's here

```
slides/deck.html                    the presentation — one file, opens offline, 5 live widgets
notebook/llm_workshop.ipynb         THE MAIN NOTEBOOK — beginner-friendly, no coding needed
web/explore.html                    zero-setup fallback — opens on any phone or laptop
notebook/llm_workshop_advanced.ipynb  the code-first version, for programmers
notebook/solutions.ipynb            worked answers to the advanced notebook's exercises
data/                               synthetic clinical corpus + precomputed real model output
src/                                teaching helpers, widgets, and the Act 5 prompts
scripts/                            everything is regenerable; see §6
CITATIONS.md                        every reused component with its licence
```

### Which notebook to use

`llm_workshop.ipynb` is the one to point people at. It assumes **no coding
ability and no prior knowledge of LLMs**: every code cell is a single line, all
interaction happens through text boxes and sliders, and the prose explains each
result so it still teaches when read alone afterwards. All the machinery lives in
`src/teaching.py`, so the screen shows results rather than loops.

`llm_workshop_advanced.ipynb` is the original: numpy written out, attention heads
hunted by hand, five exercises that require writing code. Point the programmers
at it.

**Two claims this repository makes good on:**

1. **Every number is real, and its provenance is stated.** Everything measured
   here comes from `scripts/precompute_widget_data.py` running the actual models —
   re-run it and you get the same figures. The two exceptions are the opening
   slides: the scale timeline and the MedQA benchmark chart are **published
   literature values**, cited on the slides themselves and in
   [CITATIONS.md](CITATIONS.md). Nothing anywhere is illustrative or invented.
2. **All clinical text is synthetic.** Ten discharge summaries and sixteen
   guideline snippets written for this session. No PHI, no real records, no
   credentialed corpora. The guideline sources are deliberately fictional —
   see [CITATIONS.md](CITATIONS.md) for why that's a teaching decision.

---

## 2. Running the slides

```
open slides/deck.html
```

That's it. No server, no build, no network. reveal.js is inlined.

| Key | Action |
|---|---|
| `→` / `Space` | next slide |
| `Esc` | slide overview |
| `S` | speaker notes window |
| `F` | fullscreen |

**The five widgets are live, not screenshots.** Type into the tokenizer, drag
the temperature slider, switch encoders, click through attention heads. They are
the demo — don't just talk over them.

The deck follows the viewer's light/dark setting. It is designed for 1280×720;
both themes are checked.

---

## 3. Running the notebook

### Google Colab (recommended for attendees)

1. Push this repository somewhere Colab can reach it.
2. Open `notebook/llm_workshop.ipynb` in Colab.
3. **Edit `REPO_URL` in the first code cell** to point at your copy.
4. Run all.

Attendees never need to type code — the notebook drives on Shift+Enter, text
boxes and sliders.

> **One thing worth checking on the day.** The sliders and text boxes use
> `ipywidgets`, which works in Colab but could not be verified from the machine
> this was built on. The notebook is designed so this cannot break the session:
> **every interactive cell prints a real, static result first**, and the widget is
> added underneath. If widgets fail to render, attendees still see the worked
> example and the explanation — they just cannot drag anything. Run the notebook
> once in Colab beforehand to see which you get.

The first cell installs what's missing and clones the data. (The *advanced*
notebook additionally prints a pre-flight report on the lab API; the beginner one
just says "Ready".) Model downloads total roughly **1.1 GB** (GPT-2 548 MB, PubMedBERT
440 MB, MiniLM 90 MB) — worth doing before attendees arrive if the venue wifi is
weak.

### Locally

Python 3.12 is pinned deliberately: as of writing, `torch` has no dependable
wheels for 3.14, which is the default `python3` on macOS via Homebrew.

```bash
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements.txt
.venv/bin/python -m jupyter lab notebook/llm_workshop.ipynb
```

### If Colab is not an option

`web/explore.html` is a zero-setup fallback: open it in any browser, including on
a phone. No sign-in, no install, works offline. It covers four of the six ideas —
tokenization (with a real GPT-2 encoder running in the page), the meaning/negation
comparison, the temperature dial, and the truncation bug — using precomputed
results rather than live models.

Good insurance against venue wifi, and a reasonable thing to hand out afterwards.

---

## 4. The lab API, and what happens when it isn't reachable

Act 5 uses the Heidelberg SAP proxy LiteLLM deployment
(`qwen-3.5-27b`, `nemotron-3-super`).

**Never hard-code the key.** It is read from the environment:

```bash
export HEIDELBERG_LITELLM_API_KEY="<your LiteLLM virtual key>"
```

`get_llm_client()` probes the endpoint with a 3-second timeout and returns
either a live client or a cached one **with an identical call signature**, so
every notebook cell is byte-for-byte the same in both modes. A banner always
states which mode is active. Replayed text is never presented as live inference.

### Before the session: settle the Colab question

The endpoint documentation says *lab network required*. Colab runs in Google's
cloud, so it may not reach it. **Run this in Colab beforehand** — no key needed:

```python
import socket
try:
    socket.create_connection(("129.206.138.38", 4000), timeout=3)
    print("reachable — Act 5 will run live")
except OSError as e:
    print("not reachable — Act 5 will replay:", e)
```

Either answer is fine. It only changes whether Act 5 is live or replayed;
Acts 1–4 run entirely on local open-weights models and never touch the network
after the model download.

### The replay cache ships as hand-written placeholders

`data/cached_responses.json` is currently **hand-authored**, because no lab key
was available when this was built. The notebook says so, loudly, every time.

To replace it with genuine recorded responses — do this once, on the lab
network, before the session:

```bash
export HEIDELBERG_LITELLM_API_KEY="..."
.venv/bin/python scripts/record_cache.py
```

The `_authored` flag flips to `false` and the warning banner changes to say the
responses are real recordings.

---

## 5. Run-book

Slides and notebook mirror each other. Present the concept from the deck, then
drop into the notebook and run the matching cell.

| Act | Slides | Notebook | Min | The moment that lands |
|---|---|---|---|---|
| **Opening** | 1–3 | — | 5 | The scale chart, then the MedQA bars. "AI passes the exam" is true — and measures none of what we're about to find. |
| Ground rules | 4 | Part 1 | 2 | `t.predict_next_word(...)` — the whole thesis in one bar chart. |
| **1 · Tokens** | 5–7 | Part 2 | 9 | Type `t(9;22)(q34;q11)` into the widget live: 16 characters, **12 tokens**. Then slide 7 — images are tokens too, and the window is smaller than advertised. |
| **2 · Vectors** | 8–11 | Part 3 | 12 | Two payoffs. `AML` vs `acute myeloid leukaemia` scores **0.178** in the general encoder. Then `no FLT3-ITD` vs `FLT3-ITD` scores **0.985**. |
| **3 · Attention** | 12–13 | Part 4 | 7 | Click *find the head that links the negation*. Then the turn: attention had it; pooling threw it away. |
| **4 · Sampling** | 14–15 | Part 5 | 6 | Drag temperature from 0.2 to 1.5 and watch the distribution melt. A governance point, not a style one. |
| **5 · Grounding** | 16–17 | Part 6 | 9 | The fabricated PMID, then the same question grounded. Then the planted allopurinol error. |
| **6 · Wrap** | 18–20 | Part 7 | 4 | The verdict list, GDPR / EU AI Act, and where to go next. |

**Callback worth making:** slide 3 says benchmark scores hide the failures. Slides
10, 11 and 16 are those failures. Say so out loud when you reach them — it is the
spine of the talk.

**Demonstrating from the notebook:** every cell is one line, so the output fills
the screen rather than the code. Zoom the browser to ~130% before you start and
the results read from the back of the room.

### Three things worth saying out loud

- **Act 2, the truncation bug.** `all-MiniLM-L6-v2` accepts 256 tokens; the notes
  average 527. It read half of every note and raised no error. This is the most
  immediately useful thing in the session for anyone building retrieval — it is
  a bug they probably have right now.
- **Act 2, the honest ending.** Chunking takes the general encoder from 4/10 to
  6/10 and does nothing for the biomedical one. Clinical retrieval stays hard.
  Resist the urge to end on a fix.
- **Act 5, the limitation.** Grounding *moves* the failure into retrieval, it
  doesn't remove it. A confident answer citing the wrong source is worse than an
  obvious fabrication, because it looks verified.

### If you're short on time

Cut **Act 3** (attention) and **slide 13**. Acts 1, 2 and 5 carry the argument on
their own. Do not cut Act 2 — it is where the clinical audience buys in.

---

## 6. Rebuilding everything

```bash
.venv/bin/python scripts/precompute_widget_data.py   # run the models, regenerate all figures
.venv/bin/python scripts/build_deck.py               # assemble slides/deck.html
.venv/bin/python scripts/build_beginner_notebook.py  # assemble the main notebook
.venv/bin/python scripts/build_advanced_notebook.py  # assemble the advanced notebook
.venv/bin/python scripts/build_solutions.py          # assemble the solutions notebook
.venv/bin/python scripts/build_explore_page.py       # assemble web/explore.html
.venv/bin/python scripts/build_authored_cache.py     # rebuild the placeholder replay cache
.venv/bin/python scripts/record_cache.py             # replace it with real recordings (needs key)
```

`build_deck.py --fetch-vendor` re-downloads reveal.js into `slides/_vendor/`
(needed once; that directory is gitignored).

Edit slides in `slides/deck.template.html` and `slides/widgets.js`, then rebuild
— never edit `slides/deck.html` directly, it is generated.

Likewise the notebooks: edit the matching `scripts/build_*_notebook.py` and
rebuild, so prose and code stay in step and diffs stay readable. Teaching helpers
live in `src/teaching.py` (pictures and tables) and `src/interactive.py` (boxes
and sliders) — change those to alter what every cell shows.

---

## 7. What to put in the remaining ~15 minutes

Natural continuations that don't overlap what's here:

- **Fine-tuning vs prompting vs RAG** — when each is actually worth it, and what
  each costs. This session covers only the last.
- **Evaluation** — the material stops at "measure the error rate" without saying
  how. Benchmark contamination, inter-annotator agreement, calibration.
- **Regulatory** — EU AI Act high-risk obligations, MDR, and what "clinical
  validation" means for a system that changes when you re-prompt it.
- **Local deployment** — why the Heidelberg endpoint exists; what it takes to run
  a 27B model on institutional hardware.
- **Multimodal** — the obvious next question from a myeloid audience is
  morphology and flow cytometry. Worth naming even briefly.

---

## 8. The public site

Attendee-facing materials are live at **https://sfb1709.jeffai.net** — landing page,
slides (`/deck`), the no-install version (`/try`), the notebook, and the zip.
The QR on the closing slide encodes exactly that address.

It is served by a dedicated `caddy:2-alpine` container (`~/sfb1709-site/` on the
server, alias `sfb1709-site` on the `web` network), reverse-proxied by the shared
ingress Caddy via `sites/sfb1709.caddy` in the `jeffai-infra` repo. The ingress
compose file was deliberately **not** touched, so publishing needed no container
recreation and no downtime for the other services.

To update the content after rebuilding the deck:

```bash
cp slides/deck.html deploy/public/deck.html
rsync -az --delete deploy/ healthbook:~/sfb1709-site/
ssh healthbook 'cd ~/sfb1709-site && sudo docker compose restart sfb1709-site'
```

Nothing on the server needs backing up — it all regenerates from this repo.

---

## 9. Security

- **No credential appears anywhere in this repository.** `.gitignore` covers
  `.env` and key files.
- Verify before sharing:
  ```bash
  grep -rniE "sk-[a-zA-Z0-9]{8,}|password|api[_-]?key *=" --exclude-dir=.venv .
  ```
- The lab key is a personal LiteLLM virtual key. Don't commit it, don't put it in
  a notebook cell, don't screenshot it, and rotate it if it leaks.

---

Attributions and licences for every reused component: **[CITATIONS.md](CITATIONS.md)**.
