Envoy Mesh, Java Agent & Fluentd #7566

andrewgkew · 2021-08-25T11:09:55Z

andrewgkew
Aug 25, 2021

I am looking at configuring SW to communicate with an Envoy Mesh, and because the mesh installs their own agent for each application I am assuming I dont need have the Java Agent setup?

But then how will the log collection work? If I am not using the Java agent I wont using logj42 gRPC client appender and satellite to forward the logs so I assume I can use fluentd instead, but looking at the fluentd configuration it is dependent on the log format to be specific.

Should I ensure that I setup my log4j2 appender with the right format so that it matches this express

expression /^\[SW_CTX:\s*(?<SW_CTX>[^ ]*]*)\] \[(?<level>[^ ]*)\] (?<logtime>[^\]]*) \[(?<thread>[^ ]*)\] (?<logger>[^\]]*):(?<line>[^\]]*) - (?<body>[^\]]*)$/

Any docs on this log format?

Do you recommend install fluentd as a daemonSet?

It looks like with fluentd we can push the logs through kafka to elastic search too, so if we are using ES as the backend for SW could we not us that or does OAP need to fetch the data from kafka instead?

kezhenxu94 · 2021-08-25T11:19:22Z

kezhenxu94
Aug 25, 2021
Collaborator

I am looking at configuring SW to communicate with an Envoy Mesh, and because the mesh installs their own agent for each application I am assuming I dont need have the Java Agent setup?

The 2 agents are of different functionalities and they can co-exist, feel free to install Java agent if you need our functionalities.

But then how will the log collection work? If I am not using the Java agent I wont using logj42 gRPC client appender and satellite to forward the logs so I assume I can use fluentd instead, but looking at the fluentd configuration it is dependent on the log format to be specific.

For Envoy mesh logs, it consists of 2 parts, Envoy access logs and user application logs.

Envoy access logs: (assuming you're using Istio) you can get metrics, topology and collect envoy access logs themself, see https://skywalking.apache.org/blog/2020-12-03-obs-service-mesh-with-sw-and-als/
User application logs: you can use fluentd / fluent bit, see our test cases https://github.com/apache/skywalking/blob/master/test/e2e/e2e-test/docker/log/docker-compose.fluentbit.yml and https://github.com/apache/skywalking/tree/master/test/e2e/e2e-test/docker/log/fluent-bit, there are some customization to reformat the logs into SkyWalking acceptable format.

Should I ensure that I setup my log4j2 appender with the right format so that it matches this express
expression /^\[SW_CTX:\s*(?<SW_CTX>[^ ]*]*)\] \[(?<level>[^ ]*)\] (?<logtime>[^\]]*) \[(?<thread>[^ ]*)\] (?<logger>[^\]]*):(?<line>[^\]]*) - (?<body>[^\]]*)$/

The doc format depends on your applications, just modify the expression to match your format.

Any docs on this log format?

Do you recommend install fluentd as a daemonSet?

I recommend fluent-bit (daemonset) as it's specifically optimized to be used in containers.

It looks like with fluentd we can push the logs through kafka to elastic search too, so if we are using ES as the backend for SW could we not us that or does OAP need to fetch the data from kafka instead?

Yes, fluentd doesn't support gRPC or http, and SkyWalking only supports gRPC, HTTP, Kafka, so you can only choose kafka if you insist using FluentD

1 reply

andrewgkew Sep 5, 2021
Author

I am looking at configuring SW to communicate with an Envoy Mesh, and because the mesh installs their own agent for each application I am assuming I dont need have the Java Agent setup?

The 2 agents are of different functionalities and they can co-exist, feel free to install Java agent if you need our functionalities.

@kezhenxu94 - can you explain how this would work.

If I installed Istio and configured the mesh to talk to Skywalking, I would get the APM data and Topology from the mesh ALS (I think) which is great.
But then if I also add the Java agent to my pods they would be sending the same info to Skywalking as well? I realize if I wanted to use functionality from the java agent like for logging then maybe I need both setup and configured.

My questions:

Would there be any issue configuring both an agent and mesh at the same time?
Am I right in saying they are doing the same thing? i.e. sending the same data across to OAP
If I had, mesh, an agent and satellite all configured together would that be ok?

wu-sheng · 2021-08-25T11:24:16Z

wu-sheng
Aug 25, 2021
Collaborator

Do you want to switch to Service Mesh observability solution? That is totally different. It analyzes Envoy logs.

Read this
https://skywalking.apache.org/docs/main/v8.7.0/en/setup/envoy/als_setting/

We have DP and CP monitoring without tracing agents. If you have agent deployed, you have more data already.

19 replies

kezhenxu94 Sep 6, 2021
Collaborator

One of the Istio manifest settings suggested in your docs was

--set 'meshConfig.defaultConfig.ProxyStatsMatcher.inclusionRegexps[0]=.*'

But when apply that I get the following error

Run the command with the --force flag if you want to ignore the validation error and proceed.

Error: failed to install manifests: failed to unmarshall mesh config: unknown field "ProxyStatsMatcher" in v1alpha1.ProxyConfig

But looking at the Istio docs that field seems to be there, have you come across this before?

This requires istio version >= 1.8.x, make sure you are looking up the right doc and using the needed version.

andrewgkew Sep 6, 2021
Author

Morning @kezhenxu94

I tried that just installed Istio 1.11.2 and got the same error

 ~/istio$ istioctl  manifest install \
 >     --set meshConfig.enableEnvoyAccessLogService=true \
 >     --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.skywalking.svc.cluster.local:11800 \
 >     --set meshConfig.defaultConfig.envoyMetricsService.address=skywalking-oap.skywalking.svc.cluster.local:11800 \
 >     --set 'meshConfig.defaultConfig.ProxyStatsMatcher.inclusionRegexps[0]=.*'
 Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See 
 https://istio.io/v1.11/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
 Run the command with the --force flag if you want to ignore the validation error and proceed.
 Error: generate config: failed to unmarshall mesh config: unknown field "ProxyStatsMatcher" in v1alpha1.ProxyConfig

kezhenxu94 Sep 6, 2021
Collaborator

s/ProxyStatsMatcher/proxyStatsMatcher

andrewgkew Sep 6, 2021
Author

Sorry that was my fault, I was trying to fix it (before the version) and thought it might have been a spelling mistake, let me give this a go. thanks

andrewgkew Sep 6, 2021
Author

Update that and it worked, thanks @kezhenxu94

Envoy Mesh, Java Agent & Fluentd #7566

Uh oh!

andrewgkew Aug 25, 2021

Replies: 2 comments · 20 replies

Uh oh!

Uh oh!

kezhenxu94 Aug 25, 2021 Collaborator

Uh oh!

andrewgkew Sep 5, 2021 Author

Uh oh!

wu-sheng Aug 25, 2021 Collaborator

Uh oh!

Uh oh!

kezhenxu94 Sep 6, 2021 Collaborator

Uh oh!

andrewgkew Sep 6, 2021 Author

Uh oh!

kezhenxu94 Sep 6, 2021 Collaborator

Uh oh!

andrewgkew Sep 6, 2021 Author

Uh oh!

andrewgkew Sep 6, 2021 Author

andrewgkew
Aug 25, 2021

Replies: 2 comments 20 replies

kezhenxu94
Aug 25, 2021
Collaborator

andrewgkew Sep 5, 2021
Author

wu-sheng
Aug 25, 2021
Collaborator

kezhenxu94 Sep 6, 2021
Collaborator

andrewgkew Sep 6, 2021
Author

kezhenxu94 Sep 6, 2021
Collaborator

andrewgkew Sep 6, 2021
Author

andrewgkew Sep 6, 2021
Author