[[2021-06-18_Fri]] #litmus #sockshop
pod-cpu-hogの注入後に、steady stateに戻らない。
![[Pasted image 20210618145117.png|600]]
特定ノード（gke-microservices-experi-default-pool-66a015a7-xhqvと、gke-microservices-experi-default-pool-66a015a7-so7i）のCPU負荷が常に高いままになっている。
![[Pasted image 20210618145530.png|600]]
おそらく orders-db があやしい。
![[Pasted image 20210618145618.png|600]]
```shell-session
kubectl get pods -o wide | grep orders
orders-66969b485-qpb7w 1/1 Running 3 9d 10.8.5.5 gke-microservices-experi-default-pool-66a015a7-xhqv <none> <none>
orders-db-8678fb4cc-w6p4p 2/2 Running 5 9d 10.8.0.3 gke-microservices-experi-default-pool-66a015a7-s07i <none> <none>
```
orders-dbのログをみると大量の接続エラーがでている。
```shell-session
$ kubectl logs -f deployment/orders-db --all-containers=true
...
{"t":{"$date":"2021-06-18T04:28:59.333+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.8.3.63:56584","connectionId":42223,"connectionCount":19}}
{"t":{"$date":"2021-06-18T04:28:59.339+00:00"},"s":"I", "c":"NETWORK", "id":22988, "ctx":"conn42223","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Client sent an HTTP request over a native MongoDB connection"},"remote":"10.8.3.63:56584","connectionId":42223}}
...
```
ordersのログをみると、Connection refused例外ででている。
```shell-session
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_111-internal]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_111-internal]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_111-internal]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_111-internal]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_111-internal]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_111-internal]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_111-internal]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.8.0_111-internal]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.8.0_111-internal]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[na:1.8.0_111-internal]
at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.8.0_111-internal]
at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) ~[na:1.8.0_111-internal]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) ~[na:1.8.0_111-internal]
at zipkin.reporter.urlconnection.URLConnectionSender.send(URLConnectionSender.java:174) ~[zipkin-sender-urlconnection-0.6.9.jar!/:na]
at zipkin.reporter.urlconnection.URLConnectionSender.sendSpans(URLConnectionSender.java:135) ~[zipkin-sender-urlconnection-0.6.9.jar!/:na]
at zipkin.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:228) [zipkin-reporter-0.6.9.jar!/:na]
at zipkin.reporter.AsyncReporter$Builder.lambda$build$0(AsyncReporter.java:153) [zipkin-reporter-0.6.9.jar!/:na]
at zipkin.reporter.AsyncReporter$Builder$Lambda$1.run(Unknown Source) [zipkin-reporter-0.6.9.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-internal]
```
podを再起動してみる。 => ログではエラーが出続ける。
```shell-session
$ kubectl rollout restart deploy orders
deployment.apps/orders restarted
```
```shell-session
$ kubectl rollout restart deploy orders-db
```
orders-dbのログの流量が減少した。
すこし待つと、元の水準に戻っていた。
![[Pasted image 20210618152334.png|600]]