TextDetector

This one is a bit of a sketchy project, to be honest. It started as a random idea: how much of a person's gender shows in the way they write? I had no use for the answer and I still don't. I just wanted to see what would happen, so I built a program that reads a piece of text and guesses whether the writer is a man or a woman, and, for blog posts, roughly which age bracket they were in.

One thing first. The only labels it has are male and female, because that is all the datasets I trained on contain. So it is not measuring gender. It is picking up the habits of particular online communities: bloggers from 2004, people on Reddit, people on a fitness forum, people on Twitter.

What it found

I trained it on about a million pieces of writing from five public collections, then tested it only on writers it had never seen. Otherwise it could just remember people, and the score would mean nothing.

The result depends almost entirely on how much text it gets. It is like guessing someone's job from one sentence, against an hour of chatting. One sentence is barely better than a guess, and the more you hear the better you do, up to a limit. Slide this and see.

On a single short message it is right about 6 times out of 10, where flipping a coin gets 5. On a blog post that runs a page, it is right about 7 or 8 times out of 10. Average its scores over 100 tweets from one person and it reaches about 8 out of 10, though that comes from only 83 people, so it is rough.

So the program is built to say how sure it is. Hand it a short message and it will give a weak answer, because a few words simply do not carry much. It tells you so instead of sounding sure.

What it learned, and what did not work

When I looked at what it had picked up, it was mostly stereotypes. "My husband", "hubby" and "knitting" pushed it towards female. "My wife", "halo" and "chess" pushed it towards male, and so, oddly, did the word "the". Anyone who writes against type gets guessed wrong, consistently. If you take one thing from this, take that.

Single tweets stayed around 60 to 62% whatever I tried, because most tweets simply do not carry the signal. The age guess is right a little over half the time, and always guessing the biggest age group would already get 45%, so that is not much of a result. And one run on a rented graphics card was wasted by a bug that wiped the labels from the data: the model trained happily, then scored no better than chance.

What's next

If I ever come back to it, I would test what it does when you give it several messages from the same person at once, which I have not tried yet. It will most likely stay what it is now, an experiment I poked at for curiosity and not used for anything. Part of the data is licensed for research only, so I am not releasing the trained model either way.

The question

This started as a random idea and it is a bit of a sketchy project: I wanted to see how far a text-only guess of gender could go, and nothing I do uses it. Three things interested me. Is gender detectable from text at all? How good can a guess get? And does more data, or a bigger model, change that? So it is an experiment, not a product. The labels are binary M/F, exactly as the source corpora give them, so what the model measures is the statistical habits of specific online communities, not gender.

How it works

Training data is five published corpora: the Blog Authorship Corpus, a Twitter set with one tweet per author, a mirror of PAN 2018, and RtGender's Reddit and Fitocracy sets, the last two research-only. Each text is cut into chunks of random length, from tweet-sized to page-sized, so one model sees both. Click or tap any box below to see what it does.

CorporaSplitChunkerTF-IDFDistilBERTRaw scoreCalibratorP(male)Age head

Five corpora. Five public collections of writing with a male or female label: the Blog Authorship Corpus (2004 blogs, which also has ages), a set of 200K tweets with one tweet per author, a mirror of the PAN 2018 tweet set (1,900 authors, 100 tweets each), and two RtGender sets, one from Reddit and one from a fitness site called Fitocracy. Together that is about 960K training chunks, or 760K for the linear model, which gets a capped share of the blog. The two RtGender sets are research-only, which matters at the end.

Split by author. Each author goes entirely into train, validation or test, 80/10/10, so the model is never tested on someone whose other writing it has seen. Twitter is the exception: that dataset comes with its own split, and every author in it has exactly one tweet anyway.

Chunker. Every text is cut into pieces. The target length of each piece is drawn at random from 8, 16, 32, 64, 128 or 256 words, with short ones drawn more often (22% each for 8 and 16 words, down to 8% for 256), and pieces under 5 words are thrown away. This is what lets one model handle a tweet and an essay.

TF-IDF + SGD logistic. The linear backend. Word 1-grams and 2-grams plus character 3 to 5-grams, fed into a logistic regression trained with averaged SGD. It trains in minutes on a CPU, and I can read which words it leans on.

DistilBERT. The transformer backend, fine-tuned for two epochs on all 960K chunks on a rented RTX 4090, about 30 minutes. It adds about 2 to 3 points over the linear model on blog, PAN18 and Reddit, 1.6 on Fitocracy, and nothing on the one-tweet-per-author Twitter set. It truncates at 256 tokens.

Raw score. Both backends hand back the same thing: logit(male) minus logit(female). It is a number, not yet a probability, and it means different things at different lengths, which is why the next box exists.

Length-binned calibrator. The part I care about most. A raw score from 10 words and one from 500 words are not worth the same, so each length bin gets its own Platt scaler, fit on the validation split. Both backends use the same calibrator code. Bins: up to 12, 24, 48, 96, 192, 384 words, and above.

P(male) and the band. The calibrated number is the probability of male. The tool turns it into words: under 55% it says essentially unsure, up to 65% a slight lean, up to 80% a moderate lean, and above that a strong lean. For a short message the calibrated probability stays close to 50%, so the tool reports low confidence, on purpose.

Age head. A separate linear model on the same text. It trains only on the blog labels (three brackets: 13-17, 23-27, 33-48), so it only means anything for blog-like text, and I did not calibrate it.

Both backends sit behind one interface, so everything after the raw score is shared. From the command line it looks like this. The message is one I made up and the output is an illustration of the format, not a real run.

$ python -m textdetector.predict "ok but we should go to that new cafe this weekend"
backend: transformer
input: 1 message(s), 11 words total
gender: female 56% (slight lean)
age: 13-17 55%, 23-27 32%, 33-48 13%

What is TF-IDF with a logistic regression?

TF-IDF counts how often each word or word pair appears in a text and turns down the ones that appear everywhere. Logistic regression then learns one weight per feature, so a positive weight pushes the guess towards male and a negative one towards female. Mine also uses fragments of 3 to 5 characters, which catches spelling and slang. Because the weights are plain numbers, I can list the words it leans on.

What is DistilBERT?

A smaller, faster version of BERT, a language model that has already read a large amount of English. I fine-tuned it, which means I kept what it knew about language and trained it for two passes (epochs) over my chunks to output male or female instead. Unlike the linear model, it sees word order and context.

What is Platt scaling?

A model's raw score is not a probability: a score of 2 does not mean 88%. Platt scaling fits a small logistic curve on held-out data that maps scores to probabilities, so that when the tool says 70% the aim is that about 70% of such guesses are right. I fit a separate curve per length range, because the same raw score means something different on 10 words than on 500.

Why split by author?

If posts from the same person land in both training and test, the model can learn that person's quirks and look much better than it really is. Splitting by author means every test author is a stranger to the model, so the accuracy is what you would get on someone new.

Results

Accuracy on held-out authors, chunk level, calibrated at the 0.5 threshold. Linear means TF-IDF + logistic. Chance is 50%.

Domain	Linear	DistilBERT
Blog	62.9%	65.9%
Twitter, 1 tweet each	59.4%	59.3%
PAN18, per tweet	59.9%	62.0%
Reddit	65.9%	68.4%
Fitocracy	65.0%	66.6%

Length matters more than the model. On blog text DistilBERT goes from 59.7% at 12 words or fewer to 76.1% at the longest bin, and the linear model goes from 58.1% to 72.6%.

Blog climbs steadily with length. Twitter, which is one tweet per author, does not: DistilBERT stays between 57% and 61% in every bin. Reddit and Fitocracy are mostly short texts, so their longer bins hold few chunks (437 for Fitocracy at 49 to 96 words, 480 for Reddit at 97 to 192), and those bins are noisier.

Averaging scores over many tweets from one PAN18 author reaches 79.5% at 100 tweets, against 74.7% for the linear model. Published PAN18 winners reach about 82% with heavier ensembles, so this is not a new result.

Age

Age, in three blog brackets (13-17, 23-27, 33-48), is 56.5% accurate at chunk level with the linear model only. Uniform chance is 33%, but always guessing the largest bracket gives 44.5%. Macro-F1 is 0.51. The oldest bracket is right only about 29% of the time, and most errors for the youngest and oldest brackets land in the neighbouring one.

What didn't work

Single tweets from the one-tweet-per-author Twitter set stay at 59 to 60% for both models, and PAN18 tweets score 60 to 62% per chunk. I think that is a ceiling in the data: many short texts carry no gender signal at all.

The 79.5% figure rests on 83 authors, because only those have 100 tweets. That is a standard error of roughly 4 points, and the curve is not even monotonic (63.2% at 3 tweets, below 64.2% at 1, and 69.5% at 10, below 71.1% at 5). Between model versions the k=100 result moved by a point or two, which is one or two authors.

Adding Reddit and fitness data cost the transformer 0.7 points on blogs (66.6% to 65.9%) in exchange for coverage of the other domains. A Bluesky corpus with undocumented auto-labels agreed with my model at only AUC 0.56, so I dropped it. PANDORA needs an access request, and the one open mirror lacks the demographic columns. TwiSty has no English subset.

One bug cost me a GPU run. Newer pandas silently drops the grouping column from groupby().apply(), which destroyed the labels on the remote rebuild. The model had a healthy training loss and chance-level evaluation. I now check content hashes between machines. The rented GPU time came to about 3.5 hours at $0.30/h, including that wasted run.

The linear model's strongest features are "my husband", "hubby" and "knitting" for female, and "my wife", "halo", "chess" and even "the" for male. It is largely a stereotype detector, and people who write against type are consistently misclassified.

Unfinished

I have no plans to turn this into anything. It was a curiosity, and if I come back to it, it will be to test the several-messages case described next. The command-line tool concatenates several messages into one text for a joint guess, whereas my author-level numbers average per-tweet scores, so the two are not the same measurement. I have not evaluated the concatenation path. The transformer also truncates at 256 tokens, which cuts the longest chunks a little. RtGender is research-only, so the trained models are not for release or commercial use.