[[2021-06-14_Mon]] #prometheus #sockshop Crashloop backoffになっている。 `unepxpected fault address` ```shell-session level=info ts=2021-06-14T08:05:04.335Z caller=main.go:343 msg="Starting Prometheus" version="(version=2.20.0, branch=HEAD, revision=e5a06b483527d4fe0704b8fa3a2b475b661c526f)" level=info ts=2021-06-14T08:05:04.340Z caller=main.go:344 build_context="(go=go1.14.6, user=root@ac954b6d5c6e, date=20200722-18:51:45)" level=info ts=2021-06-14T08:05:04.340Z caller=main.go:345 host_details="(Linux 5.4.89+ #1 SMP Sat Feb 13 19:45:14 PST 2021 x86_64 prometheus-statefulset-0 (none))" level=info ts=2021-06-14T08:05:04.340Z caller=main.go:346 fd_limits="(soft=1048576, hard=1048576)" level=info ts=2021-06-14T08:05:04.341Z caller=main.go:347 vm_limits="(soft=unlimited, hard=unlimited)" unexpected fault address 0x7f241a8da000 fatal error: fault [signal SIGBUS: bus error code=0x2 addr=0x7f241a8da000 pc=0x468938] goroutine 1 [running]: runtime.throw(0x286a02d, 0x5) /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0006a1300 sp=0xc0006a12d0 pc=0x434b92 runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:692 +0x443 fp=0xc0006a1330 sp=0xc0006a1300 pc=0x44b733 runtime.memmove(0x7f241a8da000, 0x2867456, 0x1) /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc0006a1338 sp=0xc0006a1330 pc=0x468938 github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffc08efc993, 0x13, 0x14, 0x30898a0, 0xc00064d860, 0x30898a0) /app/promql/query_logger.go:120 +0x3a3 fp=0xc0006a1490 sp=0xc0006a1338 pc=0x1ee2433 main.main() /app/cmd/prometheus/main.go:374 +0x4f08 fp=0xc0006a1f88 sp=0xc0006a1490 pc=0x2099d08 runtime.main() /usr/local/go/src/runtime/proc.go:203 +0x1fa fp=0xc0006a1fe0 sp=0xc0006a1f88 pc=0x4371fa runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0006a1fe8 sp=0xc0006a1fe0 pc=0x467671 goroutine 59 [runnable]: github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc0007a4680) /app/storage/remote/write.go:81 created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage /app/storage/remote/write.go:77 +0x19a goroutine 38 [select]: go.opencensus.io/stats/view.(*worker).start(0xc0002f4320) /app/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100 created by go.opencensus.io/stats/view.init.0 /app/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57 goroutine 39 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x44d9e80) /app/vendor/k8s.io/klog/v2/klog.go:1107 +0x8b created by k8s.io/klog/v2.init.0 /app/vendor/k8s.io/klog/v2/klog.go:416 +0xd6 goroutine 53 [runnable]: github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc0004fe480) /app/pkg/logging/dedupe.go:70 created by github.com/prometheus/prometheus/pkg/logging.Dedupe /app/pkg/logging/dedupe.go:61 +0xcf ``` [CrashLoop with "unexpected fault address 0x7f15db1b1000" · Issue #8298 · prometheus/prometheus](https://github.com/prometheus/prometheus/issues/8298) によると、Prometheusのディスクサイズが埋まっているときのエラーではないか。 [[Kubernetesディスクサイズ表示]]で確認すると、満杯だった。