ScreenTranslator

I wanted to read German documents on a screen share as they appear, live. Think of live audio translation, but for what you see instead of what you hear, or Google Translate's camera mode, except live and drawn on the screen itself instead of through a phone camera. ScreenTranslator is the Windows program I wrote for that. You drag a box over the part of the screen you care about, press a key, and a moment later the same words show up on top of it in the language you picked.

A screen share is just video, so the text in it can't be selected or copied, and the same goes for subtitles baked into a film, words inside a picture, or an app that draws its own letters. German is one of the roughly 200 languages the translation model covers.

Think of someone sticking a translated Post-it over every sentence, with the paper coloured to match what's underneath. Under the hood it does three things, over and over. It takes a picture of your box. It reads the letters off that picture, using the text recognition that already comes with Windows. Then it hands the words to a translation model running on the PC itself, and draws the result in a see-through window laid over the original.

I wanted the translating to happen on my own computer, so nothing on my screen gets sent anywhere. That has a price. The model I use is called NLLB-200, a free translation model from Meta. It covers about 200 languages and is decent, but it isn't as good as Google Translate or DeepL. The idea came from a free screen translator called Translumo, whose code is public, and I borrowed some of it for judging how alike two pieces of text are.

The text recognition doesn't always read the same pixels the same way. My test case is a lowercase l in "lazy" being read as a capital I. The first version I put online only recognises exact repeats, so a one-letter wobble means translating the whole thing again. The newer version on my computer compares each new reading with recent ones and, if they're close enough, keeps showing the old translation.

It does have limits. It isn't instant: the model is a download of about 2.4 GB, it wants around 3 GB of the computer's memory, and each translation takes a moment, longer on a machine without a graphics card. If you don't tell it what language the text is in, it guesses English, so anything else has to be set by hand. If the translator fails it quietly shows you the original text and doesn't say so. Some screen-recording tools can still see the overlay. And it only runs on Windows.

What's next. The newer matching and timing work is only on my own machine so far and needs to go up online. The copy I've shared is out of date and won't run yet, and I need to fix that.

What I wanted

I wanted to read German documents on a screen share as they come up, live. It's the visual version of live audio translation, or Google Translate's camera mode without the camera: the translation appears on the screen itself, over the original. A screen share is video, so the text can't be selected or copied, and the same is true of subtitles burned into a film, words inside an image, or apps that draw their own letters. So you drag a box over the text and the translation is drawn in place.

I also didn't want my screen going to a cloud service, so everything runs on my PC. The cost is quality: the local model is decent, and Google Translate and DeepL are better.

It started from Translumo, an open-source screen translator (Apache-2.0). I ported its text-similarity code, wrote my preprocessing to match one of its image helpers, and took from it the idea of matching OCR output loosely instead of exactly.

How it works

One background thread runs the loop below. The Qt window on the main thread only draws what the thread hands it. Step through one pass with the buttons.

I use Windows OCR because it's built in, needs no model download and gives back lines with word boxes. If Windows doesn't support the language I ask for, it falls back to the languages in my Windows profile. For translation I use Meta's NLLB-200 through Hugging Face Transformers, on the GPU when there is one. It's the only model the tool downloads and runs itself.

What is NLLB-200?

No Language Left Behind, a translation model from Meta AI that handles about 200 languages in a single network, so any supported pair works without a separate model for each. I use the distilled 600M version: a smaller copy trained to imitate a bigger one, with 600 million parameters. The download is about 2.4 GB.

The overlay is meant to look like the original with different words, so I estimate a background and a text colour for every block. The background is the median of a ring of pixels around the block, and the text colour comes from sorting the pixels inside it into two groups.

What is k-means?

A way of sorting pixels into groups by colour. Pick two starting colours, give every pixel to the nearer one, move each colour to the average of its pixels, and repeat. With two groups on a block of text you end up with roughly "ink" and "paper".

The overlay is a single transparent Qt window across all monitors. Windows extended styles make it click-through and keep it from stealing focus, and it's drawn in two passes, all the backgrounds first and all the text second, so one block's text can't be covered by another block's background. The translation is clipped to its backing rectangle.

Why keep the overlay out of the screenshot?

The tool photographs the screen, and the overlay sits on the screen. Without a fix it would photograph its own translation, read it as new text and translate that, forever. Windows has a call, SetWindowDisplayAffinity, that tells the system to leave a window out of captures. Some capture tools may still pick the overlay up anyway.

Why the cache is loose

The OCR doesn't always read the same pixels the same way. The example I test with is a lowercase l in "lazy" coming back as a capital I. With an exact-match cache that's a new string: it gets translated again and can come out slightly different, which would make the overlay flicker. The version on GitHub only matches exactly.

So in my local version the cache compares loosely. Every new block is scored against what's stored for the same language pair, and a score above 0.65 counts as the same text and reuses the stored translation. For Chinese, Japanese, Korean, Thai, Burmese, Khmer, Lao and Vietnamese the bar is lower, 0.55. The cache forgets everything after 3 seconds without a lookup. A second check works on the output: the joined translation is compared with the last six, and a Jaro score above 0.955 counts as a repeat. That check currently has no effect, because the overlay has already been cleared by then (stage 9).

What are Jaro and Dice?

Two ways to score how alike two strings are, from 0 to 1. Jaro counts the characters the strings share in roughly the same places, and penalises swapped ones. Dice splits both strings into pairs of neighbouring letters and counts how many pairs they share. I use the average of the two, the way Translumo does, which makes a single wrong letter cost very little.

How often it looks

On GitHub the loop wakes every 500 ms. In my local version the wait depends on what just happened. After a pass that found nothing, or a screen that hadn't changed, it waits 115 ms. After a full pass of OCR and translation it waits 320 ms. When the new OCR text is similar to the last (between 0.5 and 0.95) and no shorter, it looks like typing or scrolling, and the wait is 620 ms. Time spent working counts against the wait, and there's a floor of 50 ms.

What doesn't work

Source language "auto" isn't detection. In the tool it means English, for both the OCR and the translator, so anything that isn't English needs the source language set by hand.

When translation throws, the engine logs the error and returns the original text, so the overlay shows untranslated words with no sign anything failed. Input is cut at 128 tokens, and a translation more than three times longer than its source is truncated on screen.

The loose cache can be loose in the wrong direction. A sentence cut off by scrolling still scores 0.858 against the whole sentence, so it gets the whole sentence's old translation. The binarisation cut-off is a fixed 150. Paragraph merging only compares each line with the next one down, so text in side-by-side columns stays in separate lines.

The moment the screen changes, the old translation is wiped, so there is a blank while the new one is worked out. And the overlay may still show up in some capture tools, even with the capture exclusion on.

What is unfinished

GitHub has my first version, from February 2026, and it won't start as pushed: the language list and settings modules never made it up, so it fails on import. The fuzzy cache, the similarity code, adaptive polling, preprocessing, auto-clear and the multi-monitor BetterCam mapping are only on my own machine. One set of tests covers the similarity maths and the cache, and nothing tests the pipeline end to end. It only runs on Windows.