monachus

The idea

Ask a chat model to talk like a man from the year 1600 and it will. It is an act, though. Underneath, it has read most of the modern internet, so it knows about telephones and world wars and is politely pretending it doesn't. I wanted one where that isn't pretend: a model that has only ever read old books.

A language model is a program that learns, from a huge pile of text, which word is likely to come next. Almost everyone bends one that is already trained. I started from an empty one and fed it Latin, French books from before 1900, and English from old public-domain books. The picture in my head was a medieval monk who reads French and answers you in old-fashioned English. Monachus is Latin for monk.

What happened

I trained two of these. They are small, about 100 million and 230 million adjustable numbers inside. Neither is perfectly "old": the first one's English is mostly from before 1930, so it is not a clean 1600 mind, and the second one's stops at 1900. The only score is how well each one predicts text it has not seen, which tells you little about whether it says anything sensible.

The cheaper experiment

So I also tried the cheap route to the same effect. I took an existing modern chat model (Qwen, an open one) and trained it on a small set of period-style conversations to be an Elizabethan Englishman in the year 1600. Then came 24 questions where a man in 1600 and a man today would disagree: is the Earth at the centre of everything, what causes the plague, what is the internet.

Asked what year it is: "Why, it is the one thousand six hundred, the two-and-fortieth of the reign of our sovereign lady Queen Elizabeth."

For most questions it held the character. Not always. Asked how a telephone works, it said it had never heard of one, and then added this:

"If thou meanest how men do hear one another by means of a wire, why, that is a curious notion."

That is a modern idea slipping out. A simple word-matching check flagged 4 of the 24 answers as leaks, two of them false alarms, and it missed slips like this one. It also proves less than it looks like: the model was told in its instructions to behave this way, and underneath it is still a modern model. Answering "I know no such thing" is not the same as not knowing.

What's next

Read what the two from-scratch models actually write and get them talking. After that, a Latin-only pair, classical against medieval, asked the same question to see how their worlds differ.

What I wanted

A language model whose knowledge stops at a date. Ordinary chat models are trained on an enormous amount of modern text, so asking one to play someone from 1600 gives you a costume, not an absence of knowledge. I wanted a medieval monk who reads French and answers in old-flavoured English, with no pretrained weights and nothing modern in the corpus.

That is the goal, and the data does not fully deliver it yet. The English for the first model is Project Gutenberg with no date filter, which is overwhelmingly from before 1930; only the second model's English is cut at 1900. The Latin includes neo-Latin. The French script keeps books dated 1500 to 1900 by default. Nothing catches modern text hiding inside old scans, apart from a crude line filter on the first model's English for web addresses and ISBN numbers.

How it works

The code is plain PyTorch in a handful of files, adapted from the ideas in Karpathy's nanoGPT. model.py is a GPT-2-style transformer: pre-LayerNorm, learned position embeddings, GELU, a tied output head, no bias terms, a 1,024-token context. The tokenizer is a byte-level BPE trained on the corpus itself, so its vocabulary goes on Latin word endings instead of web fragments. train.py is a training loop with warmup and cosine learning rate, gradient clipping, bf16, and multi-GPU through torchrun if I want it. It saves a checkpoint whenever validation loss improves.

What is a token, and what is byte-level BPE?

A model does not read letters or whole words, it reads tokens, which are chunks of text. BPE (byte pair encoding) builds the list of chunks by repeatedly merging the pair of symbols that occurs most often in the training text. Byte-level means it starts from raw bytes, so any text can be encoded and nothing is ever unknown. Because I train the list on Latin and French, common word endings can end up as single tokens.

Tap or click any box in the diagram to see what it does.

Corpora prepare.py train.py ckpt.pt finetune_chat.py chat.py

There are two pretrained checkpoints. monachus is a 12-layer, 768-wide model on Latin, French and English with a 16,384-token vocabulary. latin_v2 is 16 layers, 1,024 wide, with a 24,000-token vocabulary and more English: 6.0 GB of books from PG19 published by 1900, 14,080 books in all.

Results

The two from-scratch checkpoints. Token counts are the size of each prepared training split, not necessarily what the run actually saw. Validation loss is cross-entropy in nats on random batches of held-out text, so it is a noisy estimate. The two models use different tokenizers and corpora, so the losses are not comparable.

Checkpoint	monachus	latin_v2
Val loss	3.03	3.11
Layers x width	12 x 768	16 x 1,024
Params	about 98M	about 227M
Vocab	16,384	24,000
Training tokens	2.56 billion	2.42 billion
Iteration	13,000	24,000

What does a validation loss of 3 mean?

Validation loss measures how surprised the model is by text it was never trained on, in nats per token. A loss of 3 means that, on average, it is about as unsure as if it were choosing between 20 or so equally likely next tokens (e to the power 3 is about 20). Lower is better. It tells you how well the model predicts held-out text, not whether anything it writes is true or even reads well.

Two oddities. The presets say 9,000 and 12,000 iterations, but the checkpoints are at 13,000 and 24,000, so both runs went past them. And I first called the model 124M, which is only true with GPT-2's 50k-token vocabulary; with my 16k vocabulary it is about 98M.

The side experiment

Separately I tried the cheaper route: not from scratch, but a LoRA adapter on Qwen2.5-7B-Instruct, a modern model kept frozen at 4-bit, small enough for one 24 GB graphics card. The training set is period-voice conversations of an Elizabethan Englishman in the year 1600. The same system prompt is used in training and when I test it:

Thou art a learned Englishman living in the year of our Lord sixteen hundred, in the reign of good Queen Elizabeth. Thou art well read in the wisdom of the ancients — the Greek and the Latin tongues, the poets and philosophers of old, and the French learning. Thou speakest ever in the English of thine own age, with 'thee' and 'thou' and the old turns of phrase. Thou understandest the world as the learned of thy time understand it — the heavens turning about the Earth, the four humours of the body, the four elements. Of times yet to come thou knowest nothing, and if asked of things beyond thy ken thou sayest so honestly, in thy manner, for a man cannot know what is not yet.

What are LoRA and 4-bit training?

Fine-tuning a 7-billion-parameter model normally means changing all of its weights, which needs a lot of memory. LoRA freezes the original weights and learns a small add-on, the adapter, that nudges them. Here the frozen model is also stored in 4-bit numbers to save memory. The adapter is only about 323 MB, and the model underneath is unchanged.

The 24 probes

To see whether the costume holds, there are 24 questions where a 1600 mind and a modern one would disagree: the Sun and the Earth, the plague, a wound, lightning, a telephone, the internet, whether it is an AI, what year it is. Each answer goes through a keyword matcher, with a list of modern words that would give it away and a list of period words that fit. Step through them and read the answers yourself.

The matcher counted 4 leaks, 16 period answers and 4 where nothing matched. Of the four leaks, two are plainly false alarms: "car" matched "carts", and "atom" matched the model crediting atoms to the ancient Atomists, which is period-appropriate. The other two are borderline. For the internet it said it knew no such thing but offered "network", and the AI answer says it does not know the word yet calls itself "a program" and "a complex automaton".

The matcher also misses slips. The wound answer recommends warm water and mild soap, and the telephone answer speaks of hearing "by means of a wire". The flies answer starts with "No" and calls the idea of flies rising from rot a delusion, which is the modern view, but the matcher counted it as period because it contains the word "rotting". A keyword list cannot catch subtle leaks. There are slips of other kinds too: one answer calls Paracelsus a Frenchman, and one has a stray Chinese word in the middle of a sentence.

Every probe ran with the persona prompt on, which tells the model it knows nothing of later times, so the tally says little about what the model knows. Mostly it shows whether the model follows that prompt.

What didn't work, and what is missing

For the from-scratch models the only quality number is validation loss, around 3. The chat stage is code plus a chat set of 16 different examples repeated up to 640 rows, with no mined pairs.

The side experiment is a different kind of thing. The base stays modern underneath, so answering "I know not the word" does not show the knowledge is gone. That is why internal_probes.py exists. It skips the persona and the chat format entirely and feeds the raw model bare stems like "World War II ended in the year", to see whether it completes them with the modern fact. It is also why rmu.py replaced NPO, which damaged ordinary text completion. That table and the 24 answers are all the evaluation there is.

What are unlearning, NPO and RMU?

Unlearning tries to remove specific knowledge from a model after training, instead of just telling it not to say things. NPO (negative preference optimisation) pushes down the probability of the facts I want gone, while keeping ordinary text prediction intact. RMU (representation misdirection for unlearning) works inside the model instead: on those facts it steers one middle layer's internal state toward random noise, and elsewhere it holds that layer close to the original. Both are published research methods, which I tried on my adapter.

Unfinished

Still on the list: read samples from both checkpoints, run the chat stage on real mined data, and score the unlearning branch. I also have a list of untried ideas: separate classical and medieval Latin twins to compare their worldviews, a model trained on a single author such as Cicero or Aquinas, and sampling one run at early checkpoints to watch grammar arrive before meaning.