[[opentelemetry-ebpf-profiler]]を用いて、[[Xtsdb]]と[[KairosDB]]のそれぞれのプロファイルを取得する。 ``` $ uname -a Linux xtsdb-ng-01 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ``` https://docs.google.com/spreadsheets/d/1oNkWODIpF_WxBXcyUBUYyKjH_jWrOrV2y3vmGIgKt0A/edit?gid=1904500240#gid=1904500240 [[Xtsdb]] [[Grafana Alloy]] http://10.0.5.100:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=unspecified&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=&from=2025-02-09T07:14:59.025Z&to=2025-02-09T07:44:37.362Z&var-filters=instance%7C%3D%7Cxtsdb-ng-01&var-filtersBaseline=&var-filtersComparison=&maxNodes=16384&refresh=30s ``` loaded 181800000 items in 1787.429070sec with 4 workers (mean point rate 101710.329660/sec, mean value rate 101710.329660/sec, 20.66MB/sec from stdin) ``` 2/15: scalevar 100 ``` loaded 181800000 items in 1157.330309sec with 2 workers (mean point rate 157085.663933/sec, mean value rate 157085.663933/sec, 31.80MB/sec from stdin) ``` - xtsdb-ingester (7.09m) - GC 1.05(m) - プロトコル処理 3.91m - Redisプロトコル処理 1.44m - xtsdb-flusher (1.29m) - - redis (8.86m) - lua 1.27m - AE: 24.2s - コマンド処理 5.21m - lookup 46.0s + 48.2s + 1.45m + 9.08s + 13.9s - cassandra (5.05m) - 2/15: scalevar 10000 ``` loaded 181800000 items in 1816.705157sec with 2 workers (mean point rate 100071.274255/sec, mean value rate 100071.274255/sec, 20.33MB/sec from stdin) ``` http://10.0.5.100:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=unspecified&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=&from=2025-02-09T15:19:57.632Z&to=2025-02-09T15:35:54.829Z&var-filters=&var-filtersBaseline=&var-filtersComparison=&maxNodes=16384&refresh=30s > loaded 181800000 items in 957.880139sec with 2 workers (mean point rate 189794.101108/sec, mean value rate 189794.101108/sec, 38.42MB/sec from stdin) ## [[KairosDB]] [[opentelemetry-ebpf-profiler]] http://10.0.5.100:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=unknown&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=all&from=2025-02-09T10:48:30.000Z&to=2025-02-09T12:35:15.000Z&var-filters=host_name%7C%3D%7Ckairosdb-ng-01&var-filtersBaseline=&var-filtersComparison=&maxNodes=16384&refresh=30s 19:48:30 - 21:35:15 11 + 60 + 35 = 106:45: total 2.42hours compoents - kairosdb-app (18.9m) - メトリクス受信 プロトコル処理 (8.43m ) - Cassandraへの書き込みの問い合わせ (10.2m) - cassandra (47.8m + 20.0) - memtable flush (19.3m, ) - flush (12m, 8.07%) - コンパクション (5.42+1.9 m, ) - JVM処理 GCなど (2.42 - 1.58) - scalvar 100 > loaded 181800000 items in 4294.664170sec with 2 workers (mean point rate 42331.598654/sec, mean value rate 42331.598654/sec, 8.57MB/sec from stdin) http://10.0.5.100:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=unknown&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=all&from=2025-02-10T01:55:50.818Z&to=2025-02-10T03:06:25.384Z&var-filters=&var-filtersBaseline=&var-filtersComparison=&maxNodes=16384&refresh=30s scalevar 10000 > loaded 181800000 items in 9147.085281sec with 2 workers (mean point rate 19875.183669/sec, mean value rate 19875.183669/sec, 4.04MB/sec from stdin) http://10.0.5.100:3000/a/grafana-pyroscope-app/profiles-explorer?searchText=&panelType=time-series&layout=grid&hideNoData=off&explorationType=flame-graph&var-serviceName=unknown&var-profileMetricId=process_cpu:cpu:nanoseconds:cpu:nanoseconds&var-dataSource=local-pyroscope&var-groupBy=all&from=2025-02-10T13:19:45.000Z&to=2025-02-10T15:51:44.000Z&var-filters=host_name%7C%3D%7Ckairosdb-ng-01&var-filtersBaseline=&var-filtersComparison=&maxNodes=16384&refresh=30s ## Dataset ``` $ zcat graphite_bulk_records__usecase_devops__scalevar10000_30m.gz | wc -l 181800001 ``` ``` $ zcat graphite_bulk_records__usecase_devops__scalevar100_2d2.5h.gz | wc -l 183618001 ``` ``` $ ./bulk_data_gen_cmd -format graphite-line -use-case devops -scale-var=100 -seed 123 -timestamp-start "2020-06-01T00:00:00Z" -timestamp-end "20 20-06-03T02:30:00Z" | gzip | cat > graphite_bulk_records__usecase_devops__scalevar100_2d2.5h.gz using random seed 123 2025/02/09 23:11:39 Using sampling interval 10s 2025/02/09 23:11:39 Using cardinality of 900 2025/02/09 23:18:55 Written 16362000 points, 183618000 values, took 435.951316 seconds 2025/02/09 23:18:55 bulk_data_gen - main() took 7m15.9534315s ``` ``` $ ./bulk_data_gen_cmd -format graphite-line -use-case devops -scale-var=100 -seed 123 -timestamp-start "2020-06-01T00:00:00Z" -timestamp-end "20 20-06-03T02:00:00Z" | gzip | cat > graphite_bulk_records__usecase_devops__scalevar100_2d2h.gz using random seed 123 2025/02/09 23:26:32 Using sampling interval 10s 2025/02/09 23:26:32 Using cardinality of 900 2025/02/09 23:33:42 Written 16200000 points, 181800000 values, took 430.605374 seconds 2025/02/09 23:33:42 bulk_data_gen - main() took 7m10.611194539s ```