Why does Prometheus monitoring data briefly increase after some data is deleted, and then decrease again?

ZSvirt was open-sourced from the ZStack ZSphere commercial project and has since developed independently. Some code, configuration, and documentation still retain historical ZStack/ZSphere names (such as environment variables, configuration items, dependencies, paths, and commands). These names are for compatibility only and do not affect the normal use of ZSvirt.

Description

  • Prometheus compresses older monitoring data. During compression, it generates new compressed data from the source data, which temporarily increases total storage usage. After compression completes, the source data is deleted, so the total usage decreases. When retained monitoring data exceeds the configured threshold, Prometheus also deletes older data blocks.

  • The monitoring data retention size refers to the total size of all directories under /var/lib/zstack/prometheus/data2/, excluding the wal directory. Prometheus deletes monitoring data in block units. Each block corresponds to a subdirectory under /var/lib/zstack/prometheus/data2/. Because each block contains a different amount of data, the amount deleted in each operation varies.

  • You can modify the monitoring data retention size in Global Settings as needed to control the maximum local directory size for stored monitoring data.

On this page