Forecasts against DB’s own

What was measured

Every ten minutes, DB’s own forecast was recorded for twenty stations chosen in advance across the whole network — six major hubs down to three village halts. The next day the archive says when each train really arrived, and both forecasts are scored against that.

The model is the app’s own prediction code at commit eee57d9ba683, which declares version 0.3.0 (versionCode 7). That commit is not a released version, and the version number above is the last released one rather than this code's — the bump happens in the release commit, so a candidate carries its predecessor's. The app code has moved on in 7 commit(s) since 0.3.0 was tagged. The app published in the stores does not contain this model unless a later release says so. It is only ever shown history from before the day it predicts. It appears twice: as shipped, which when DB actually reports a delay anchors on that report and widens it by the error such reports are measured to carry at that much notice, and leans on the train’s own past runs when DB says nothing; and history only, which never looks at the live number at all. The same treatment is given to the connecting train’s departure report, so a change is the difference of two admitted errors rather than one distribution and one promise.

Two kinds of answer are scored, and they correspond to the two kinds of journey the app plans. For a journey without a change the answer is an arrival time, scored as a distribution against the arrival that happened. For a journey with one change the answer is the probability of making that change, scored against whether it was made. The two use different scores and are not comparable with each other; each is compared only with DB’s answer to the same question. A third answer joins them below: the complete two-leg journey — the predicted arrival at the far end of a change, against the arrival that happened. It is in the same units as a direct journey, so those two are comparable with each other.

The comparison, with its uncertainty

Each row is BayesianBahn’s score minus DB’s over the same predictions, so a negative number is the lower score for BayesianBahn, and lower is better for both scores used here. The interval is what decides it: delays arrive in clusters — one late train produces a dozen correlated predictions — so the range comes from resampling whole trains, not individual predictions. Where the interval crosses zero, the collected days are not enough to claim anything, however suggestive the middle number looks.

How far ahead, and how wrong

Lead time is counted back from the train’s scheduled departure, because that is when a passenger can still act on the answer. Lower is better; the scores are in minutes and are directly comparable — see the definitions at the foot of the page for why a point forecast and a distribution can be put on one axis.

Read the history-only series as a flat reference, not as a trend. It never looks at the live number and the arrival it predicts does not move, so its answer for a given stop is the same however early it is asked — checked on every render, and for the stops polled more than once the quantiles are identical at every lead time. What little the column does move across the buckets is composition and not behaviour: a train first seen forty minutes out is in the near buckets and in none of the far ones, so each bucket averages a different set of trains. Held to the stops present in all six, the history-only figure is the same in every bucket to three decimal places, while DB still climbs from 1.49 to 3.08 and the shipped model from 1.60 to 2.28. The degradation with lead time is real for the two series that read the live number, and only for those two.

The two variants cross over in this bucket. In the >3h bucket the shipped model scores 2.23 against history alone at 2.22 — leaning on DB’s number makes the answer 0.01 minutes worse there. That is the shape this model was changed to remove, so a crossover appearing here means a reported delay is being believed in a range where it should not be.

How the errors are distributed

The means above summarise a badly skewed distribution, and the two forecasts differ in its upper tail rather than in its middle. That is worth reading in full, because a forecast a minute out and one twenty minutes out have very different consequences for a passenger.

One asymmetry has to be held in mind while reading the boxes. DB answers with a single minute, so its error is a whole number of minutes and is exactly zero — the train arrived in the minute DB named — for 41% of all predictions here. A distribution cannot score zero: it always pays something for its own spread, however well placed it is. So the bottom of the two boxes is not a like-for-like comparison and should not be read as one; the medians and everything above them are. The same asymmetry is why DB's median is exactly 1.00 minute in every bucket.

The last two columns of the table are the point. DB's score is the lower one on slightly more than half of the individual predictions, and its mean is higher all the same. Being closer more often and worse on average is what a heavy tail looks like from the inside.

The box spans the middle half of the predictions, the ruled line across it — overhanging on both sides, so it stays visible where it meets a box edge — is the median, and the whiskers reach the 10th and 90th percentiles. The worst tenth of each forecast therefore reaches past the whisker, and is given exactly in the table below.

The last two columns give the share of forecasts out by more than 15 minutes. Over 296,619 predictions, most of which are uneventful, this is the part of the distribution a mean is least able to convey.

Does the 80% range hold?

The app gives not only a time but a range stated to contain the true arrival four times in five. That is checkable: count how often the real arrival fell inside it. A bar at the dashed line matches the stated probability; below it the range is narrower than the forecast’s accuracy supports, and above it wider.

Connections

A connection here is a train arriving at one of the sampled stations and another leaving it a few minutes later, judged from before the first train set off — the last moment at which the answer could still change a decision. DB’s timetable answers yes or no; the app answers with a probability.

Split by what actually happened

Almost every connection is caught, so answering “yes” every time scores well on the pooled average, and a yes/no answer taken from the timetable is close to doing that. The two outcomes are therefore worth reading apart: the connections that failed are the smaller group and the one the pooled figure says least about.

Does it hold from one day to the next?

Everything above pools the collected days. Pooling gives the more precise estimate; it cannot show whether a result is a property of the model or of one day’s conditions. Here each day stands alone. A column that points the same way in every row is the more durable result; one that changes sign between days is within the range of day-to-day variation.

Read the row lengths too. The first day was collected from the evening onwards, so it is both smaller and drawn only from the busiest hours — a day with fewer missed connections here is not necessarily a calmer day.

The missed connections, day by day

This is the smallest sample on the page and the one most easily over-read.

Journeys with a change, end to end

Everything above scores a journey with a change by whether the change works. That is not the question a passenger has, which is when they get there — and it cannot be compared with the direct-journey figures, because a probability and a distribution over arrival times are not in the same units. This section asks the passenger's question. The answer is scored the same way a direct journey is, in minutes of CRPS, so the two rows can finally be read against each other.

Both forecasters are asked before the feeder sets off and see the same 5.7 candidate trains on average. DB answers with the arrival of whichever train its own forecasts say the passenger catches; ours answers with a distribution over the arrival, which includes missing the planned connection and taking a later train — it puts 0.8% of its weight on catching none of the candidates at all. The truth is the arrival of whichever train was actually caught.

Truth is not held to that list. A passenger who misses every train the app offered waits for the next one that runs, and the forecast is then wrong by however long that took — so the walk to the train actually boarded goes on past the end of the list, and both forecasters are charged for the difference. That happens on 28.4% of the journeys here. Scoring only the ones that fit inside the list would drop exactly the answers that were most wrong: it raised this model's margin over DB by about 0.8 minutes of CRPS while discarding a third of the evidence.

This is the newest part of the page and rests on the fewest days: the far end of a change only began being polled once the second tier existed, so this table starts later than every other one here.

The table is measured over the journeys both variants answered. They do not answer the same set: a connecting train with too little shared history can be carried by a live report and cannot be carried without one, so the shipped model answers 63,720 journeys against the blinded model's 62,031, and 60,308 are answered by both. Measuring each over its own set would put answers to different questions in one row — it made the live number look worth four times what it is worth on the journeys both answered. Answering more journeys is a real advantage of the shipped model, but it is a difference in coverage and is counted here rather than folded into a score.

Which kinds of train, and which kinds of line

Everything above pools every train at every sampled station. The model does not treat them alike: it carries a separate prior per class of train — a mean of 4.0 minutes for long distance against 1.0 for an S-Bahn — so the classes are the axis those constants were chosen on, and splitting by them is the only way to see whether each one earns its value. Lower is better; the gap is BayesianBahn's score minus DB's, with the interval that decides it.

A category that is neither long distance nor one of the named regional ones still falls through to the regional prior, because that is what TrainClass.fromCategory does with anything beginning with a letter. Replacement buses reach it that way, and they are a third of every regional prediction here. They are not trains — they do not share the tracks and are not held by the same signals — and they score about half a minute worse than the regional trains they were pooled with, so they are given their own row. The model is unchanged: the row name says which prior they were scored under.

Working days and the weekend

The two are different networks to forecast. DB’s own mean error is 2.82 minutes from Monday to Friday against 2.08 minutes at the weekend, so there is substantially less to improve on in the quieter half. Each figure below is BayesianBahn’s score minus DB’s, computed within that part of the week alone.

An interval that spans zero says the collected days cannot separate the two forecasters in that part of the week, not that they are equal.

Delay through the day

Delay is not a property of a train alone: a late train occupies a platform and the next one can inherit some of it, so delay accumulated in the morning may still be on the network in the afternoon. If so, an hour-of-day term describes the data rather than merely fitting it.

The collected days show that pattern. The mean arrival delay climbs from 1.16 min at 01:00 to 4.18 min at 16:00, a factor of 3.6, and then drains overnight. Each train counts once, at its scheduled arrival hour, however many times it was polled. Only days that cover the whole clock are included, and only hours carrying at least 100 trains.

The maximum falls in the mid-afternoon rather than at the evening rush. One reading is that accumulated delay is worked off after the last peak departures, so the worst hour to arrive is set by how much delay the network is still carrying rather than by passenger numbers.

What that means for the time bands

The model keeps separate delay statistics per time band, which only makes sense if the hours inside a band resemble each other. Spread is the distance between the latest and earliest hourly mean inside the band: a large spread means the band is averaging together hours that behave differently.

The bands that ship were chosen from the commuter timetable rather than from this curve, and do not line up with it: the peak band ends before the peak, and the night band spans the range from the quietest hour of the night to the tail of the evening. This is an observation about the model’s bucketing rather than about the trains, and it is only as good as the days collected so far.

Definitions

CRPS — continuous ranked probability score

The integral of (F(x) − 1{x ≥ y})² over all x, where F is the forecast’s cumulative distribution and y is what happened. In minutes; lower is better. A point forecast is a distribution with all its weight in one place, and for that case CRPS is exactly the absolute error — which is why DB’s column and ours sit on the same axis. It is the only score here that credits the whole distribution rather than just its middle.

Brier score

Mean squared distance between a stated probability and the 0/1 outcome; lower is better. A yes/no answer is a probability of 1 or 0, so its Brier score is simply the share it got wrong. This makes “yes” and “72%” comparable.

DB bias

Mean signed error. Negative means DB predicted the train earlier than it arrived — optimism, which is the direction that costs a passenger a connection.

DB surprises

Share of arrivals more than 5 minutes later than DB said. This is the error that turns into a missed change.

As shipped vs history only

Both are the same model. As shipped feeds it DB’s live number for the station, but only when that number is a reported delay: DB states a stop in four shapes and three of them mean “on time”, which is the plan restated rather than an observation, so those are passed through as no report at all. History only withholds the live number in every case. The model was built to take the measured delay at the train’s previous stop, and the live path substitutes this station’s forecast instead — an approximation documented in the source, and where DB does report a delay the gap between these two columns is what it costs.

Ground truth

The archive’s recorded arrival minus the scheduled arrival, the same definition used to build the history the model learns from. Cancelled stops are excluded from the delay scores.

Time to change

Scheduled gap between the feeder arriving and the connection leaving, less the 5 minutes assumed for walking between platforms.

What this does not establish

• 16 days of data. Delays cluster by line, by weather and by incident, so the effective sample is far smaller than the counts suggest. A single quiet day is one draw, not a result — which is why every comparison above carries an interval and every day is also shown on its own.
• The days do not cover the same hours. Collection began in the evening on the first day and ran round the clock afterwards, so a difference between days mixes the date with the time of day. Both matter: DB’s yes/no on connections is far more often right during the day than in the evening, so the pooled figure for missed connections depends on the mix.
• Every feasible pair counts as a connection, including changes nobody would make. That inflates how often connections are caught and flatters both forecasters against real journeys.
• DB’s forecast is read from IRIS, the feed behind the station displays, not from the Navigator app itself. The two normally agree; the cross-check that would confirm it was unavailable while this ran.
• Twenty stations, chosen before any data existed but still twenty. Nothing here is weighted to how often people actually travel.
• Weekdays and weekend days are pooled. Traffic, staffing and the timetable itself differ between weekdays and weekends, so a figure drawn from one does not transfer to the other — the hour-of-day curve above least of all. Where both kinds of day are present, the two are also shown apart, above.

Reproducing this

Every figure above comes from the commands below. The collector must have been running on the day in question; the archive publishes the ground truth the following morning.

# collect (runs continuously, survives restarts and power cuts)
pixi run -e evaluate collect
pixi run -e evaluate collect-status      # is it still healthy?

# once the archive has published the day, score it and rebuild
# this page — stages already done are skipped, so adding a day
# costs only that day
pixi run -e evaluate evaluate 2026-08-17 2026-08-18 2026-08-19 2026-08-20 2026-08-21 2026-08-22 2026-08-23 2026-08-24 2026-08-25 2026-08-29 2026-08-30 2026-08-31 2026-09-01 2026-09-02 2026-09-03 2026-09-04