Agents pipeline customization¶

This document describes how you can customize a data pipeline Input -> filter 1 -> ... -> filter N -> Output in the logging agents (FluentD, Fluent-bit).

FluentD¶

FluentD Input customization¶

FluentD Input Customization of the out-of-box configuration¶

Parameters that affect out-of-box input configuration:

containerRuntimeType
fluentd.systemLogging
fluentd.excludePath
fluentd.systemLogType

You can find the full list of FluentD parameters in the installation guide.

FluentD Custom input configuration¶

You can add your own custom part of the input pipeline configuration by using fluentd.customInputConf. Example of the custom input configuration:

fluentd:
  install: true
  #...
  customInputConf: |-
    <source>
      custom_input_configuration
    </source>

FluentD Filters customization¶

FluentD Filters Customization of the out-of-box configuration¶

Parameters that affect out-of-box filters configuration:

containerRuntimeType
fluentd.cloudEventsReaderFormat
fluentd.billCycleConf
fluentd.multilineFirstLineRegexp

You can find the full list of FluentD parameters in the installation guide.

Append fields to every log message¶

You can add your own custom fields (labels) to every log messages, for example, to identify the source of sent logs on the Graylog server later.

You can use fluentd.extraFields parameter to add custom key-value pairs to every log message processed by the FluentD agent. Example:

fluentd:
  install: true
  #...
  extraFields:
    foo_key: foo_value
    bar_key: bar_value

Warning: This filter is based on record_transformer plugin, so it can override existing fields if their keys are identical.

This filter works after all other filters except the custom filter.

Custom filter configuration¶

You can add your own custom part of the filtering pipeline configuration by using fluentd.customFilterConf. Example of the custom filtration configuration:

fluentd:
  install: true
  #...
  customFilterConf: |-
    <filter raw.kubernetes.var.log.**>
      custom_filter_configuration
    </filter>

FluentD Output customization¶

FluentD Output Customization of the out-of-box configuration¶

Parameters that affect out-of-box output configuration:

fluentd.graylogOutput
fluentd.fileStorage
fluentd.totalLimitSize
fluentd.tls

You can find the full list of FluentD parameters in the installation guide.

FluentD Custom output configuration¶

You can add your own custom part of the output pipeline configuration by using fluentd.customOutputConf. Example of the custom output configuration:

fluentd:
  install: true
  #...
  customOutputConf: |-
    <store ignore_error>
      custom_output_configuration
    </store>

FluentD Scenarios¶

Send a specific type of logs to custom output¶

Objective:

Some of your pods have logs that you want to send in your custom output. For example, access logs that are represented like this:

2024-05-20T09:57:38.406 CEF:0|Qubership|access-control|5.13.0|IMPORT_RULES|ABAC configuration was imported|6|suser=Some User

You can define these type of logs by including access-control marker in log (or stricter rule).

Configuration:

Add your custom filter that will rewrite tag in parameter fluentd.customFilterConf Documentation of rewrite_tag_filter link: rewrite_tag_filter. For example:

fluentd:
  customFilterConf: |-
    <match parsed.kubernetes.var.log.your-app**>
      @type rewrite_tag_filter
      <rule>
        key log
        pattern /access-control*/
        tag my-tag.var.log.access-pods.log
      </rule>
    </match>

The filter in the example checks if the logs from your-app* pods fits regular expression. If true, a new tag my-tag.var.log.access-pods.log applied to the log and emitted again.

Note: Rewriting tag increases time of processing log, because log is sent to the beginning of pipeline and each filter applies to log again.

Configure custom output in Fluent Bit. There are different custom output integration described in documentation.

Now we can send logs with a new tag my-tag.var.log.access-pods.log. All we need to do is to set parameter fluentd.customOutputConf.

For instance, you need to send logs with the new tag to Splunk:

  fluentd:
    customOutputConf: |-
      <match my-tag.var.log.access-pods.log>
        @type copy
        @log_level fatal

        @type splunk_hec
        protocol http
        insecure_ssl false
        hec_host <splunk_host>
        hec_port <splunk_port>
        hec_token <splunk_token>
      </match>

In case if you need to send these logs both to Splunk and Graylog:

fluentd:
  customOutputConf: |-
     <match my-tag.var.log.access-pods.log>
       @type copy

       <store ignore_error>
         @type gelf
         host <graylog_host>
         port <graylog_port>
         protocol tcp
         retry_wait 1s
         <buffer>
           flush_interval 30s
           retry_max_interval 64
           chunk_limit_size 2m
           queue_limit_length 160
           flush_thread_count 32
           retry_forever false
         </buffer>
       </store>

       @type splunk_hec
       protocol http
       insecure_ssl false
       hec_host <splunk_host>
       hec_port <splunk_port>
       hec_token <splunk_token>
    </match>
  ```

  **Note:** These are just examples, not recommended configurations.

## FluentBit

### FluentBit Input customization

#### Customization of the out-of-box configuration

Parameters that affect out-of-box input configuration:

* `containerRuntimeType`
* `fluentbit.systemLogging`
* `fluentbit.systemLogType`

You can find the full list of FluentBit parameters in
[the installation guide](../installation.md#fluentbit).

#### Custom input configuration

You can add your own custom part of the input pipeline configuration by using `fluentbit.customInputConf`.
Example of the custom input configuration:

```yaml
fluentbit:
  install: true
  #...
  customInputConf: |-
    [INPUT]
      Name   <name>

With fluentbit.output.http.routing.enabled set, the exact tags out_audit, out_system, out_pods, out_nginx, out_k8s_event, out_access, out_int, and out_default are reserved for routing. Do not assign them in fluentbit.customInputConf or in custom rewrite_tag rules: records with these tags skip level normalization. Other tags that start with out_ remain available and are always normalized. When HTTP routing is disabled, every tag is normalized.

A custom input reaches the HTTP output and whatever you declare in fluentbit.customOutputConf. The built-in GELF, Loki and OpenTelemetry outputs select the tags the operator produces, so custom records do not reach them. Sending records there also needs a short_message field, which the built-in filters add only to container, audit and system records. GELF rejects a record without it.

FluentBit Filters customization¶

FluentBit Customization of the out-of-box configuration¶

Parameters that affect out-of-box filters configuration:

containerRuntimeType
fluentbit.billCycleConf
fluentbit.multilineFirstLineRegexp
fluentbit.multilineOtherLinesRegexp

You can find the full list of FluentBit parameters in the installation guide.

Append fields to every log message in FluentBit¶

You can add your own custom fields (labels) to every log messages, for example, to identify the source of sent logs on the Graylog server later.

You can use fluentbit.extraFields parameter to add custom key-value pairs to every log message processed by the FluentBit agent. Example:

fluentbit:
  install: true
  #...
  extraFields:
    foo_key: foo_value
    bar_key: bar_value

This filter works after all other filters except the custom filter. The filter is based on record_modifier plugin.

Custom filter configuration in FluentBit¶

You can add your own custom part of the filtering pipeline configuration by using fluentbit.customFilterConf. Example of the custom filtration configuration:

fluentbit:
  install: true
  #...
  customFilterConf: |-
    [FILTER]
      Name record_modifier
      Match *
      Record testField fluent-bit

FluentBit Output customization¶

FluentBit Output Customization of the out-of-box configuration¶

Parameters that affect out-of-box output configuration:

fluentbit.graylogOutput
fluentbit.graylogHost
fluentbit.graylogPort
fluentbit.graylogProtocol
fluentbit.totalLimitSize
fluentbit.tls

You can find the full list of FluentBit parameters in the installation guide.

FluentBit HTTP stream routing¶

fluentbit.output.http.routing.enabled makes the HTTP output send an X-Log-Type header, so that vmauth can route each stream to its own VictoriaLogs instance. The header name comes from fluentbit.output.http.routing.headerTag, and is X-Log-Type by default.

The rules run in the order below and the first match wins, so a record marked as audit never reaches the access rule.

`X-Log-Type`	Records
`out_audit`	Container records the audit filters marked, and everything the audit inputs read
`out_k8s_event`	Container records whose `kind` is `KubernetesEvent`
`out_nginx`	Container records whose pod name contains `ingress-nginx`
`out_access`	Container records whose `logType` or `log_type` is `access`
`out_int`	Container records whose `logType` or `log_type` is `int`
`out_pods`	Every other container record
`out_system`	Everything the system input reads
`out_default`	Anything else, a custom input for example

A service sets logType or log_type itself; the pipeline never writes either. Both spellings are read, and both are matched exactly, the same way the Route Access logs and Route Integration logs processing rules match them on the Graylog side. A value of integration is therefore not int.

FluentBit Custom output configuration¶

You can add your own custom part of the output pipeline configuration by using fluentbit.customOutputConf. Example of the custom output configuration:

fluentbit:
  install: true
  #...
  customOutputConf: |-
    [OUTPUT]
      Name null
      Match fluent.*

FluentBit Scenarios¶

Send a specific type of logs to custom output using FluentBit¶

Objective:

Some of your pods have logs that you want to send in your custom output. For example, access logs that are represented like this:

2024-05-20T09:57:38.406 CEF:0|Qubership|access-control|5.13.0|IMPORT_RULES|ABAC configuration was imported|6|suser=Some User

You can define these type of logs by including access-control marker in log (or stricter rule).

Configuration:

Add your custom filter that will rewrite tag in parameter fluentbit.customFilterConf Documentation of rewrite_tag filter link: rewrite-tag. For example:

fluentbit:
  customFilterConf: |-
    [FILTER]
        Name                  rewrite_tag
        Match                 parsed.raw.kubernetes.var.log.pods.your-app*
        Rule                  $log /access-control*/ my-tag.var.log.access-pods.log true
        Emitter_name          re_my_audit
        Emitter_Storage.type  filesystem
        Emitter_Mem_Buf_Limit 10M

The filter in the example checks if the logs from your-app* pods fits regular expression. If true, the log is copied with a new tag my-tag.var.log.access-pods.log and sent to the beginning of Fluent Bit pipeline to be processed (about Fluent Bit pipeline you can read in article router. In brief, each log message has a tag that is needed to decide should any filter be applied to log and to route log to output according to tag). If the log should be sent both in Graylog and your custom output, the flag true set in Rule parameter.

Note: Rewriting tag increases time of processing log, because log is sent to the beginning of pipeline and each filter applies to log again.

Configure custom output in Fluent Bit. There are different custom output integration described in documentation.

Now we can send logs with a new tag my-tag.var.log.access-pods.log. All we need to do is to set parameter fluentbit.customOutputConf.

For instance, you need to send logs with the new tag to Splunk:

fluentbit:
  customOutputConf: |-
    [OUTPUT]
        Name         splunk
        Match        my-tag.var.log.access-pods.log
        Host         <splunk_host>
        Port         <splunk_port>
        Splunk_Token <splunk_token>
        TLS          On
        TLS.Verify   Off

Note: These are just examples, not recommended configurations.

FluentBit Aggregator¶

The way to customize the data pipeline in the FluentBit in Aggregator mode is almost the same as for the standard FluentBit except the following differences:

The aggregator configuration have fixed input config
The aggregator configuration doesn't have billCycleConf
You have to set parameters into the different fluentbit.aggregator section instead of fluentbit