Update behaviour after disk is full.

[StyXman]

I run out of disk this morning:

mdione@ioniq:~/src/projects/elevation/data/osm$ osm2pgsql-replication update --port 5432 --database europe
2025-07-12 13:07:33 [INFO]: Using replication service 'https://download.geofabrik.de/europe-updates'.
2025-07-12 13:11:05  osm2pgsql version 2.1.1
2025-07-12 13:11:05  Database version: 16.4 (Debian 16.4-3)
2025-07-12 13:11:05  PostGIS version: 3.4
2025-07-12 13:11:05  Loading properties from table '"public"."osm2pgsql_properties"'.
2025-07-12 13:11:05  Using flat node file '/home/mdione/src/projects/elevation/data/osm/europe.cache' (same as on import).
2025-07-12 13:11:05  Using output 'flex' (same as on import).
2025-07-12 13:11:05  Using style file '/home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto-flex.lua' (same as on import).
2025-07-12 13:11:05  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 13:11:05  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 13:43:47  Reading input files done in 1962s (32m 42s).
2025-07-12 13:43:47    Processed 2953438 nodes in 335s (5m 35s) - 9k/s
2025-07-12 13:43:47    Processed 711976 ways in 291s (4m 51s) - 2k/s
2025-07-12 13:43:47    Processed 18907 relations in 1336s (22m 16s) - 14/s
2025-07-12 13:43:48  Going over 88258 pending ways (using 4 threads)
Left to process: 0.......
2025-07-12 13:45:13  Processing 88258 pending ways took 85s (1m 25s) at a rate of 1038.33/s
2025-07-12 13:45:13  Going over 60567 pending relations (using 4 threads)
Left to process: 38687...2025-07-12 14:05:59  ERROR: DB copy thread failed: Ending COPY mode for 'planet_osm_polygon' failed: ERROR:  could not extend file "base/87679982/91569895.60": No space left on device
HINT:  Check free disk space.
Left to process: 38676...2025-07-12 14:06:16 [CRITICAL]: Exception during execution: Command '['/usr/bin/osm2pgsql', '--append', '--slim', '--prefix', 'planet_osm', '-d', 'europe', '-P', '5432', '/tmp/tmpfkov18ov/osm2pgsql_diff.osc.gz']' died with <Signals.SIGSEGV: 11>.

This is expected since I am, well, running out of space :)

In that graph, we can see the correlation with some of the logs up there.

So, I free some space, and I try again. What surprises me is what happens then:

2025-07-12 20:00:02 [INFO]: Using replication service 'https://download.geofabrik.de/europe-updates'.
2025-07-12 20:02:17  osm2pgsql version 2.1.1
2025-07-12 20:02:17  Database version: 16.4 (Debian 16.4-3)
2025-07-12 20:02:17  PostGIS version: 3.4
2025-07-12 20:02:17  Loading properties from table '"public"."osm2pgsql_properties"'.
2025-07-12 20:02:17  Using flat node file '/home/mdione/src/projects/elevation/data/osm/europe.cache' (same as on import).
2025-07-12 20:02:17  Using output 'flex' (same as on import).
2025-07-12 20:02:17  Using style file '/home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto-flex.lua' (same as on import).
2025-07-12 20:02:17  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-07-12 20:02:17  Storing properties to table '"public"."osm2pgsql_properties"'.
Processing: Node(2953k 10.5k/s) Way(711k 2.84k/s) Relation(10010 10.9/s)
2025-07-12 20:26:30  ERROR: DB copy thread failed: Ending COPY mode for 'planet_osm_line' failed: ERROR:  could not extend file "base/87679982/91569896.125": No space left on device
HINT:  Check free disk space.
2025-07-12 20:26:31 [CRITICAL]: Exception during execution: Command '['/usr/bin/osm2pgsql', '--append', '--slim', '--prefix', 'planet_osm', '-d', 'europe', '-P', '5432', '/tmp/tmpj6hf90ab/osm2pgsql_diff.osc.gz']' returned non-zero exit status 2.

Yes I run out of space again.

So my question is: how does the update process works? In particular, I would expect a second pass not to use more space until it hits the place where it left before. This clearly is wrong, so I don't understand how the update process. Moreover, this is not the first (and not the last, until I buy a new SSD) that I run out of space, so I wonder if there is some cleanup that can be done before retrying and update.

[pnorman]

This is all up to PostgreSQL as osm2pgsql's only direct usage of disk space is the flat nodes file. When importing data twice on the second import it will delete rows then insert new rows with identical contents. This consumes disk space.

You should run a vacuum. You probably don't have enough room for a vacuum full or cluster.

[cquest]

Not only index bloat, but also data bloat, especially on planet_osm_line... and I don't know why this table is more affected by bloat than the point/polygon/roads ones.
From time to time I do some manual re-cluster to recompact data order by geography which then also saves a lot of reads I/O.

[StyXman]

To be honest, right now I barely have space for the whole import; just 56GiB left while reclustering.

You can do a 'REINDEX' on an existing database

Anything to add to reindex?

From time to time I do some manual re-cluster to recompact data order by geography

Do you use the same reclustering queries that osm2pgsql does at the end of the import?

If so, could we get an option to do that like osm2pgsql-replication update --reindex --recluster?

[StyXman]

just 56GiB left while reclustering

plus the 30GiB I reclaimed after removing the pbf :) If the input file is read sequentially and only once, then one could probably do some bash magic (<(...)) to pipe it directly from a curl command :)

[pnorman]

If you're that short on disk space are you running with --disable-parallel-indexing?

SSDs aren't designed to have heavy loads on them when 100% full so what you're doing isn't ideal long-term.

[StyXman]

ouch, no. but I shouldn't be running out of space in the first place (I'm at a healthy 286GiB now :). Note taken.