Clustering

Nobody in YJMob100K comes with a label saying commuter or homebody. Clustering asks whether such kinds exist anyway, using nothing but how people moved, and this tab then asks whether people still belong to their kind once the emergency starts.

Overview

Clustering is a way of finding groups in data when nobody has said what the groups are. Every person, place or object is described by a list of numbers, and a clustering method puts the ones whose numbers are alike into the same group and the ones whose numbers differ into different groups. It is unsupervised learning: there is no right answer to train on, so the groups have to be judged afterwards by whether they are stable and whether they make sense.

There are two broad families. Partitional methods such as k-means decide on a number of groups k in advance and split the data into exactly k groups at once. K-means places k centres, assigns every point to its nearest centre, moves each centre to the average of its points, and repeats until nothing changes (Figure 1). It is fast enough for tens of thousands of people, but it tends to find round groups of similar size, and because it can end in a local optimum it is restarted several times from different starting centres. Hierarchical methods do not fix k. Agglomerative clustering starts with every point as its own group and merges the two closest groups again and again until one is left, and the history of merges forms a tree called a dendrogram (Figure 3). Cutting the tree at some height gives the groups; the taller the vertical line before a merge, the more different the two groups being merged.

Figure 1. How k-means finds groupsClose

Everything depends on what counts as alike, which is the job of the distance metric. Euclidean distance is the straight line between two points: it asks how far apart two people are on all their traits together, and it is the distance k-means minimises. Cosine distance is one minus the cosine of the angle between two points seen as arrows from the average person: it asks whether two people differ from the average in the same directions, regardless of by how much. In Figure 2, person A is closest to C by Euclidean distance but closest to B by cosine distance, because B is a stronger version of A. Manhattan distance, the sum of the differences trait by trait, is a third common choice. The hierarchical clustering on this tab uses cosine distance and the k-means uses Euclidean distance, so the two methods look at the people in different ways.

Figure 2. Euclidean and cosine distanceClose

Figure 3. A dendrogram from five pointsClose

In this project clustering is used twice. The main use is on people: each of 24,514 people is described by fifteen numbers about their ordinary weeks, and k-means and hierarchical clustering look for kinds of movers among them. Those kinds are then frozen and used as a ruler, to see who still fits their kind in a normal fortnight and who stops fitting it in the emergency. That is the reliability question of the whole project in a simple form, and the kinds become the groups that the later studies report their results by. A second, smaller use is on places: 11,858 grid cells are grouped by the time of day at which people are in them. One earlier study sorted the same YJMob100K people into returners and explorers by fixed thresholds on how far they travel (Armantalab, Hawkins and Kontar, 2025). The kinds here come from the data instead of a threshold, and they include when people move as well as how far.

Data

Clustering needs a table of numbers with no labels: one row per thing to be grouped, one column per measured property, and nothing that says in advance which group a row belongs to. It also needs the columns to be on comparable scales, because a column measured in kilometres would otherwise outweigh a column measured as a share between 0 and 1.

The cleaned YJMob100K table from the DataPrep_EDA tab is not like that. It has one row per record, 29.4 million of them, each saying that a person was seen in a grid cell during a half-hour of a day (Table 1). To group people, every person first has to become one row of numbers that sums up how they live their days.

Before any number is computed, a few things are fixed for each person:

• Home is the cell where the person is seen most often between midnight and 5 am on days 0-44.
• The daytime place is the cell where they are seen most often between 10 am and 4 pm on workdays of days 0-44, which for many people is a workplace or a school.
• “At home” means anywhere in the 3 by 3 block of cells around the home cell, because a person who stays in one spot is often recorded in a neighbouring 500 m cell as well. The same goes for the daytime place.
• Five blocks of fifteen days split each person’s 75 days: days 0-14, 15-29 and 30-44 are the reference period, days 45-59 are an ordinary period that nothing is fitted on, and days 60-74 are the emergency. The same fifteen numbers are computed for every person in every block.

The first six numbers say how far a person ranges.

• Radius of gyration (rg_km): how far, in kilometres, a person’s records typically lie from the middle point of all their positions. It is small for someone whose life fits in one neighbourhood and large for someone spread across the city.
• Cells per day (cells_per_day): how many different 500 m cells the person is seen in, divided by the number of days they are seen.
• Spread over places (entropy): how evenly the records are shared among the cells the person visits, from 0 when everything happens in one cell to 1 when every visited cell gets the same share.
• Moving share (moving_share): of all pairs of records exactly half an hour apart, the share in which the person is in a different cell the second time.
• Step length (step_km): the typical length of those half-hour moves, in kilometres.
• Daily reach (extent_km): on each day, the farthest the person gets from home, averaged over the days.

The next three say how rooted a person is.

• Home share (home_share): the share of all records that are at home.
• Two main places (top2_share): the share of records in the person’s two most visited cells, whatever those cells are.
• Regular places (n_anchors): how many cells hold at least 5% of the person’s records, a count of the places they keep coming back to.

The last six say when a person is out, mostly on workdays.

• Away in the morning, at midday, in the evening (morning_away, midday_away, evening_away): on workdays, of the records between 7 and 9 am, between 10 am and 4 pm, and between 6 and 10 pm, the share that are away from home.
• Night records (night_share): the share of all the person’s records that fall between midnight and 5 am. Records are mostly taken while people are out and moving, so this says how active, or how visible, someone is at night.
• Weekend ratio (weekend_away_ratio): the share of daytime records away from home on weekend-like days divided by the same share on workdays. Above 1 means a person is out more on weekends than on workdays.
• Regularity (regularity): the share of workdays on which the place where the person spends most of 10 am to 4 pm is their usual daytime place.

From these traits to the table that is clustered:

• Rates, not counts. Every trait is a share, a rate or a typical value, never a count of records, so a person who is simply recorded less often does not look different because of it.
• One reference row per person. It is the average of the person’s blocks in days 0-44, kept for the 24,514 people who have at least two usable blocks and all fifteen traits (Table 2).
• Log transform. Six traits were strongly skewed, with a few people far out in the tail (step length, weekend ratio, daily reach, night records, radius and cells per day), so they were log-transformed first.
• Standardisation. All fifteen were then scaled to mean 0 and standard deviation 1 (Table 3), so a trait in kilometres and a trait between 0 and 1 count the same.
• Fitted on development people only. The scaler was fitted on the 17,164 development people and applied unchanged to the 7,350 evaluation people and to every later block.
• Recording kept apart. Three more numbers describe how well a person is recorded rather than how they move: records per day, the share of days with any record, and the hours between the first and last record of a day. They stay out of the clustering and are used afterwards as a check.

Table 1. Before: the cleaned record tableClose

Table 2. After: fifteen traits per personClose

Table 3. After: the same people, standardisedClose

Links: the raw YJMob100K files are on Zenodo, record 10836269; the cleaned record table is in the release Processed tables, Part 1; a 500-person sample of the trait table used here is data/samples/person_traits_sample.csv.

Code

Python 3.11 with pandas, NumPy, scikit-learn, SciPy and Matplotlib, run on the Alpine cluster at the University of Colorado Boulder.

• notebooks/03_kinds_of_movers.ipynb: the traits, k-means, hierarchical clustering, the frozen-kinds test, the checks and every figure on this tab.
• code/profiles.py: turns the 29.4 million records into the person and block traits.

Results

How many kinds?

K-means was run for every k from 2 to 10, ten times from different starting centres each, on the 17,164 development people. The elbow plot has no clear bend (Figure 4). The mean silhouette, which compares how close a person is to their own group with how close they are to the next group, is highest at k = 2 (0.29), falls to 0.21 at k = 3 and stays near 0.15 from k = 6 on. Values this low mean that people do not form separate islands. They form one continuous cloud, and any k cuts it rather than finds it. So stability was checked as well: k-means was fitted on two different bootstrap samples, both fits labelled everyone, and their agreement was measured with the adjusted Rand index, twenty times per k. k = 2 and k = 3 come back almost identical every time (0.98 and 0.97). k = 4 and k = 5 do not (0.84 and 0.76, with bad draws down to 0.53). k = 6 recovers to 0.83.

Figure 4. Choosing kClose

The silhouette’s answer, k = 2, only splits people into those who stay close to home and those who range wide. k = 3 is the most robust description with some content. This tab works with k = 6, which is as stable as anything above k = 3 and separates kinds that k = 3 merges, such as people who go to the same place every workday and people who range as far without a fixed place. Figure 5 shows k = 3, 5 and 6 on the first two principal components from the PCA tab.

Figure 5. Three values of kClose

Six kinds of movers

Named from their centres, the six kinds read as six ways of living a fortnight. Far roamers (3,890 development people) range widest, with a radius of 14.5 km, 8.5 different cells a day, few fixed places and the least regular days. Mid-range movers (4,556) live the same kind of life at about half the scale. Regular commuters (3,020) are away from home on 90% of workday mornings and spend 78% of workday middays at the same place. Local regulars (3,242) stay within about 6 km and keep half their records near home. Weekend movers (1,028) are away from home more than twice as much on weekends as on workdays. Homebodies (1,428) keep three quarters of their records near home and move within 3 km.

The clustering was never told anything about the time of day, and still each kind has its own shape of day (Figure 6). On workdays regular commuters and far roamers are both away more than 94% of the time between 8 am and 3 pm, but far roamers are also away 60% of the time in the middle of the night, and part of what makes them far roamers is that they have no fixed place at night. Weekend movers leave early on workdays, are mostly home through the middle of the day and go out again in the evening. Homebodies never pass 41%.

Figure 6. When each kind is away from homeClose

A second opinion from the tree

Hierarchical clustering with cosine distance was run on a random sample of 3,000 development people, since a tree of all 17,164 would need 147 million pairwise distances. Average linkage, which joins groups by the mean distance between their members, peeled off tiny groups of 45 and 60 people as the tree was cut lower. Complete linkage, which joins groups by their farthest pair, kept the groups balanced and is the one shown in Figure 7. Its cosine silhouette is 0.37 at two groups, 0.35 at three and 0.32 at four, and then falls to 0.28 and 0.23. So the tree suggests three or four groups. The last merges in the dendrogram are tall and close together, and there is no height where a cut is obviously right, which is the same message as the low silhouette of k-means.

Figure 7. Dendrogram, cosine distance, complete linkageClose

Cut at six groups and laid against the six k-means kinds of the same people, the tree agrees on the main lines (Figure 8). The wide-ranging people form one branch in both methods: 578 far roamers and 340 mid-range movers share branch H2 with almost nobody else. Regular commuters have a branch of their own, H6. The rooted side is a single branch in the tree, holding local regulars, weekend movers and homebodies together. Cosine distance looks only at the direction in which a person differs from the average, and all three kinds differ toward home, so the tree joins them where k-means, which also counts how far, splits them. The adjusted Rand index between the two methods is 0.54 at two and three groups and about 0.36 at six.

Figure 8. Where each k-means kind lands in the treeClose

Do people keep their kind in the emergency?

The scaler and the six centres were frozen on the development people and days 0-44. Then every person’s block, for development and evaluation people alike, was measured again with the same fifteen traits and given the nearest of the same six kinds. A person is still in their kind if a new block lands where their reference period did. Nothing is refitted in between: the six kinds are one ruler, and only the person’s behaviour is measured anew.

In the ordinary test period, days 45-59, which nothing was fitted on, 74% of people are still in their kind. In the emergency, days 60-74, it is 59%. The evaluation people, who were never part of any fit, give 74% and 60%. The drop is not the same for everyone (Figure 9). Homebodies barely change (73% to 69%), and far roamers stay the most stable kind (87% to 74%). Regular commuters lose the most, 20 points, from 76% to 56%, followed by weekend movers and local regulars, who end at 49%, the lowest of all.

Figure 9. Who keeps their kind of routine in the emergencyClose

The people who leave their kind do not scatter at random (Figure 10). In the ordinary period the moves go both ways in similar amounts. In the emergency they mostly point toward the rooted end: regular commuters become mid-range movers (20%) or local regulars (17%), far roamers become mid-range movers (22%), and local regulars become homebodies (16%).

Figure 10. Where people go when they leave their kindClose

People are also recorded less in the emergency, a median 12.4% fewer records per day, so part of this could be thinner data rather than different behaviour. Two checks bound that part. Among the 5,587 people whose records per day changed by less than 10%, the share still in their kind falls from 74% to 64%, against 74% to 60% for everyone, so at most about a quarter of the drop comes from recording. And removing 30% of every person’s records at random leaves nine in ten people in the same kind and barely moves the result (74% to 73% in the ordinary period, 60% to 59% in the emergency).

Kinds of places

The same tool was used on places. For the 11,858 cells with at least 500 records on days 0-59, which hold 92% of all records, each cell was described by the share of its records in each half-hour of workdays and of rest days. K-means finds no natural groups here, with a silhouette of 0.15 at two groups and 0.06 at five: cells range continuously from places full at night to places full at midday. Five groups still describe that range well, and they are named by the hour at which they are busiest on workdays (Figure 11).

Figure 11. Five rhythms of placesClose

Every group loses records in the emergency, because everyone is recorded less, but not equally (Figure 12). The noon cells, the most workplace-like, lose the most, 15% on workdays and 21% on rest days, while the busy afternoon and evening centres lose the least, about 8% on workdays.

Figure 12. Which places lose records in the emergencyClose

Conclusions

People in this city do not come in a few sharply separated types. Their ordinary weeks form one continuous range, from people who stay around home to people who cover a wide area, and any grouping cuts that range at some point rather than discovering a natural border. Within it, six kinds give a fair and stable description: far roamers, mid-range movers, regular commuters, local regulars, weekend movers and homebodies. Each has its own shape of day even though the method never saw a clock, and two quite different clustering methods agree on the main lines.

The more important finding is what happens to these kinds when the emergency comes. In an ordinary fortnight about three people in four still fit the kind their earlier weeks gave them. During the emergency it is three in five. The people with the most fixed routines change the most: regular commuters, who in normal times go to the same place almost every workday, are the kind most often unrecognisable, while people who were already mostly at home hardly change. Those who leave their kind mostly move toward smaller ranges and more time near home, and places that look like workplaces lose the most visitors. Thinner recording explains only a small part of this.

For the project this matters directly. A simulated stand-in built from someone’s ordinary weeks would describe roughly three quarters of people correctly in a normal fortnight, but fewer during the emergency, and it would fail most often for exactly the people whose ordinary routine looked the most predictable. The later studies report how reliable such agents are separately for each of these six kinds, because an average over everyone would hide where they fail.