Merge pull request #2112 from alanconway/fluentd-plugin-loki

Add fluentd-plugin-grafana-loki to fluentd image

Author: openshift-merge-robot

HACKING.md

@@ -11,22 +11,11 @@ images into a public registry like `quay.io`).
 
 ## Deploying to OpenShift
 
-See the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator/blob/master/docs/HACKING.md) hacking document for deploying OpenShift Logging using the operator make targets.  
+See the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator/blob/master/docs/HACKING.md) hacking document for deploying OpenShift Logging using the operator make targets.
 
 ## Updating Sources for Fluentd
 
-Use the script `hack/update-fluentd-vendor-gems.sh` to update the vendored source
-code for Fluentd.  It is highly recommended to use something like [rvm](https://rvm.io) in order to isolate specific gem versions being used by Fluentd. 
-
-**Note:** Be sure to add `fluentd/rh-manifest.txt`
-or add it to the commit if it was updated.
-
-### Add a new dependency or update a dependency version
-1. Use a ruby version: `rvm use $RUBY-VERSION`
-1. Create a gemset: `rvm gemset create $GEMSETNAME`
-1. Use gemset: `rvm gemset use $GEMSETNAME`
-1. Edit: `./fluentd/Gemfile`
-1. Vendor dependencies: `./hack/update-fluentd-vendor-gems.sh`
+	See [fluentd/README.md](fluentd/README.md)
 
 ### Updating all the dependencies
 
@@ -39,5 +28,4 @@ Set the clobber env var: `CLOBBER_VENDOR=true ./hack/update-fluentd-vendor-gems.
 
  Edit the file `fluentd/source.jemalloc` to also update the
 vendored jemalloc source.  You will have to use `git add` or `git rm` or otherwise
-fix any conflicts, then commit and submit a PR.  
-
+fix any conflicts, then commit and submit a PR.

fluentd/Gemfile

@@ -8,6 +8,7 @@ gem 'fluent-plugin-cloudwatch-logs'
 gem 'fluent-plugin-concat'
 gem 'fluent-plugin-elasticsearch'
 gem 'fluent-plugin-kafka'
+gem 'fluent-plugin-grafana-loki'
 gem 'fluent-plugin-multi-format-parser'
 gem 'fluent-plugin-record-modifier'
 gem 'fluent-plugin-rewrite-tag-filter'

fluentd/Gemfile.lock

@@ -55,16 +55,18 @@ GEM
       elasticsearch
       excon
       fluentd (>= 0.14.22)
+    fluent-plugin-grafana-loki (1.2.7)
+      fluentd (>= 0.14.10, < 2)
     fluent-plugin-kafka (0.13.1)
       fluentd (>= 0.14.10, < 2)
       ltsv
       ruby-kafka (>= 0.7.8, < 2)
-    fluent-plugin-label-router (0.2.4)
-      fluentd (>= 0.14.10, < 2)
     fluent-plugin-kubernetes_metadata_filter (2.5.2)
       fluentd (>= 0.14.0, < 1.12)
       kubeclient (< 5)
       lru_redux
+    fluent-plugin-label-router (0.2.4)
+      fluentd (>= 0.14.10, < 2)
     fluent-plugin-multi-format-parser (1.0.0)
       fluentd (>= 0.14.0, < 2)
     fluent-plugin-prometheus (1.7.3)
@@ -175,6 +177,7 @@ DEPENDENCIES
   fluent-plugin-cloudwatch-logs
   fluent-plugin-concat
   fluent-plugin-elasticsearch
+  fluent-plugin-grafana-loki
   fluent-plugin-kafka
   fluent-plugin-kubernetes_metadata_filter
   fluent-plugin-label-router

fluentd/README.md

@@ -1,9 +1,13 @@
 # Fluentd
+
 [Fluentd](https://www.fluentd.org/) is the log collector that resides on each OpenShift node to gather application and node logs
+
 ## Configuration
 This image is intended to be run in conjunction with the configuration and `run.sh` files provided by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator).  Experiences with the image outside that context may vary.
+
 ## Utilities
 Several utilities are available with the final image.
+
 ### sanitize_msg_chunks
 Sanitize file buffer chunks by removing corrupt records.
 
@@ -19,48 +23,86 @@ file chunk, perform a limited set of operations to confirm message validity. Use
 the files before running this utility.
 
 ## Hacking
+
 This repo vendors the dependencies required to build the final image. Following are instructions for hacking on fluentd.
+
 ### Directories
+
 Dependencies are organized according in several basic directories:
 * `lib` - Dependencies for which this repo is considered the single source of truth.  This is where we make changes that are permanent forks of other libraries or libraries that do not have any other upstream
 * `vendored_gem_src` - Dependencies that are copied in from upstream sources based on what is defined in the `Gemfile`.  Changes should never be made to content in this directory as it is likely to be clobbered on updates
 
 
 ### Modifying a gem Dependency
-Modifying or updating any of the gems requires a local ruby installation.  The version of ruby should match that defined in the `Dockerfile`.  It is highly recommended to setup the environment using something like [rvm](https://rvm.io/) which makes it possible to install multiple versions of ruby and define sets of gems.  The following instructions assume the use of rvm.
-#### Install ruby
+
+Modifying or updating any of the gems requires a local ruby installation.
+The version of ruby should match the ruby version in the first line of `Dockerfile`.
+It is highly recommended to setup the environment using something like [rvm](https://rvm.io/) which makes it possible to install multiple versions of ruby and define sets of gems.
+
+#### Install Ruby
+
+The version of ruby should match the ruby version in the first line of `Dockerfile`.
+
 ```
-$ rvm install ruby-2.5
-$ rvm use ruby-2.5
+rvm install ruby-2.5
+rvm use ruby-2.5
 ```
-#### Install Vendored gems
-Gems required by fluentd are maintained in the [Gemfile](https://bundler.io/gemfile.html) which is the list of dependencies.  The repository also includes the `Gemfile.lock` which is the explicit list of the versions of the dependencies being used including the transitive ones.
+
+#### Install Vendored Gems
+
+Gems required by fluentd are maintained in the [Gemfile](https://bundler.io/gemfile.html) which is the list of dependencies. The repository also includes the `Gemfile.lock` which is the explicit list of the versions of the dependencies being used including the transitive ones.
+
+Note the bundler version should match the version from the end of `Gemfile.lock`
 
 ```
-$ rvm gemset create fluentd
-$ rvm gemset use fluentd
-$ gem install bundler -v 1.17.3
-$ bundle install
+rvm gemset create fluentd
+rvm gemset use fluentd
+gem install bundler -v 1.17.3
+bundle install
 ```
+
 #### Updating Vendored Gems
-* Edit the Gemfile as needed
+
+* Edit `Gemfile` to add new gems.
 * Run the update script from the repo root dir
-* Commit changes and rebuild image
+  ```
+  cd ..
+  ./hack/update-fluentd-vendor-gems.sh
+  ```
+* Commit changes and rebuild images
+
+**Note:** The hack script accepts the following env vars:
+
+* `BUNDLE_TASK=install`: (default) download and install all missing Gems.
+* `BUNDLE_TASK=update SOMEGEM` : update a single gem to latest.
+* `BUNDLE_TASK=update` : update _all_ the gems to latest.
+  **Note**: Not to be undertaken lightly!
+* `CLOBBER_VENDOR`: If set remove everything in `vendored_gem_src` for cases where the intent is to clean up the vendored gems to remove unused dependencies
+
+If you get an error like this:
+
+``` shell
+[32mFetching fluent-plugin-kafka 0.13.1[0m
+[31mDownloading fluent-plugin-kafka-0.13.1 revealed dependencies not in the API or the lockfile
+(fluentd (>= 0.10.58, < 2)).
 ```
-$ ./hack/update-fluentd-vendor-gems.sh
+
+This may help (replace the plugin name and version appropriately)
+``` shell
+gem uninstall fluent-plugin-kafka
+gem install fluent-plugin-kafka --version 0.13.1
 ```
-**Note:** The hack script accepts the following env vars:
-* `BUNDLE_TASK`: 'install'(default) or 'update' to modify what bundler is doing.  'update' may be used if the intent is to update all the gems to the latest.  This will also update the `Gemfile.lock` which should additionally be committed.
-* `CLOBBER_VENDOR`: This removes everything in `vendored_gem_src` for cases where the intent is to clean up the vendored gems to remove unused dependencies
 
 #### Make lib Changes
+
 * May need to install additional dependencies from `lib/<gemdir>` using `bundle install`
 * Make any needed changes to the files in lib
 * Test change `bundle exec rake test`
 * Commit change and rebuild image
 
 ### Building the Container Image
+
 Build the container image from the fluentd directory like:
 ```
 $ podman build -t fluentd:test .
-```
+```

fluentd/rh-manifest.txt

@@ -1,77 +1,78 @@
+addressable 2.7.0 https://github.com/sporkmonger/addressable
 aws-eventstream 1.1.0 https://github.com/aws/aws-sdk-ruby
 aws-partitions 1.396.0 https://github.com/aws/aws-sdk-ruby
 aws-sdk-cloudwatchlogs 1.38.0 https://github.com/aws/aws-sdk-ruby
 aws-sdk-core 3.109.3 https://github.com/aws/aws-sdk-ruby
 aws-sigv4 1.2.2 https://github.com/aws/aws-sdk-ruby
+bigdecimal 3.0.0 https://github.com/ruby/bigdecimal
+concurrent-ruby 1.1.6 http://www.concurrent-ruby.com
+connection_pool 2.2.3 https://github.com/mperham/connection_pool
+cool.io 1.6.0 http://coolio.github.com
 digest-crc 0.6.1 https://github.com/postmodern/digest-crc#readme
 dig_rb 1.0.1 https://github.com/jrochkind/dig_rb
+domain_name 0.5.20190701 https://github.com/knu/ruby-domain_name
+elasticsearch 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
+elasticsearch-api 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
+elasticsearch-transport 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
+ethon 0.12.0 https://github.com/typhoeus/ethon
+excon 0.75.0 https://github.com/excon/excon
+faraday 1.0.1 https://lostisland.github.io/faraday
+ffi 1.11.3 https://github.com/ffi/ffi/wiki
 ffi-compiler 1.0.1 http://wiki.github.com/ffi/ffi
 fluent-config-regexp-type 1.0.0 https://github.com/okkez/fluent-config-regexp-type
 fluentd 1.7.4 https://www.fluentd.org/
+fluent-mixin-config-placeholders 0.4.0 https://github.com/tagomoris/fluent-mixin-config-placeholders
 fluent-plugin-cloudwatch-logs 0.7.6 https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs
 fluent-plugin-concat 2.4.0 https://github.com/fluent-plugins-nursery/fluent-plugin-concat
+fluent-plugin-elasticsearch 4.1.1 https://github.com/uken/fluent-plugin-elasticsearch
+fluent-plugin-grafana-loki 1.2.7 https://github.com/grafana/loki/
 fluent-plugin-kafka 0.13.1 https://github.com/fluent/fluent-plugin-kafka
+fluent-plugin-kubernetes_metadata_filter 2.5.2 https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
 fluent-plugin-label-router 0.2.4 https://github.com/banzaicloud/fluent-plugin-label-router
 fluent-plugin-multi-format-parser 1.0.0 https://github.com/repeatedly/fluent-plugin-multi-format-parser
+fluent-plugin-prometheus 1.7.3 https://github.com/fluent/fluent-plugin-prometheus
+fluent-plugin-record-modifier 2.1.0 https://github.com/repeatedly/fluent-plugin-record-modifier
+fluent-plugin-remote-syslog 1.1 https://github.com/docebo/fluent-plugin-remote-syslog
+fluent-plugin-rewrite-tag-filter 2.3.0 https://github.com/fluent/fluent-plugin-rewrite-tag-filter
 fluent-plugin-splunk-hec 1.1.2 https://github.com/splunk/fluent-plugin-splunk-hec
+fluent-plugin-systemd 1.0.2 https://github.com/reevoo/fluent-plugin-systemd
+http 4.4.1 https://github.com/httprb/http
 http-accept 1.7.0 https://github.com/ioquatix/http-accept
+http-cookie 1.0.3 https://github.com/sparklemotion/http-cookie
+http-form_data 2.3.0 https://github.com/httprb/form_data.rb
 http-parser 1.2.1 https://github.com/cotag/http-parser
+http_parser.rb 0.6.0 http://github.com/tmm1/http_parser.rb
 jemalloc 5.2.1 https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2
 jmespath 1.4.0 http://github.com/trevorrowe/jmespath.rb
 jsonpath 1.0.5 https://github.com/joshbuddy/jsonpath
+kubeclient 4.8.0 https://github.com/abonas/kubeclient
+lru_redux 1.1.0 https://github.com/SamSaffron/lru_redux
 ltsv 0.1.2 https://github.com/condor/ltsv
-rubygem-addressable 2.7.0 https://github.com/sporkmonger/addressable
-rubygem-bigdecimal 3.0.0 https://github.com/ruby/bigdecimal
-rubygem-concurrent-ruby 1.1.6 http://www.concurrent-ruby.com
-rubygem-connection_pool 2.2.3 https://github.com/mperham/connection_pool
-rubygem-cool.io 1.6.0 http://coolio.github.com
-rubygem-domain_name 0.5.20190701 https://github.com/knu/ruby-domain_name
-rubygem-elasticsearch 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
-rubygem-elasticsearch-api 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
-rubygem-elasticsearch-transport 7.8.0 https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.x/index.html
-rubygem-ethon 0.12.0 https://github.com/typhoeus/ethon
-rubygem-excon 0.75.0 https://github.com/excon/excon
-rubygem-faraday 1.0.1 https://lostisland.github.io/faraday
-rubygem-ffi 1.11.3 https://github.com/ffi/ffi/wiki
-rubygem-fluent-mixin-config-placeholders 0.4.0 https://github.com/tagomoris/fluent-mixin-config-placeholders
-rubygem-fluent-plugin-elasticsearch 4.1.1 https://github.com/uken/fluent-plugin-elasticsearch
-rubygem-fluent-plugin-kubernetes_metadata_filter 2.5.2 https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
-rubygem-fluent-plugin-prometheus 1.7.3 https://github.com/fluent/fluent-plugin-prometheus
-rubygem-fluent-plugin-record-modifier 2.1.0 https://github.com/repeatedly/fluent-plugin-record-modifier
-rubygem-fluent-plugin-remote-syslog 1.1 https://github.com/docebo/fluent-plugin-remote-syslog
-rubygem-fluent-plugin-rewrite-tag-filter 2.3.0 https://github.com/fluent/fluent-plugin-rewrite-tag-filter
-rubygem-fluent-plugin-systemd 1.0.2 https://github.com/reevoo/fluent-plugin-systemd
-rubygem-http 4.4.1 https://github.com/httprb/http
-rubygem-http-cookie 1.0.3 https://github.com/sparklemotion/http-cookie
-rubygem-http-form_data 2.3.0 https://github.com/httprb/form_data.rb
-rubygem-http_parser.rb 0.6.0 http://github.com/tmm1/http_parser.rb
-rubygem-kubeclient 4.8.0 https://github.com/abonas/kubeclient
-rubygem-lru_redux 1.1.0 https://github.com/SamSaffron/lru_redux
-rubygem-mime-types 3.3.1 https://github.com/mime-types/ruby-mime-types/
-rubygem-mime-types-data 3.2020.0512 https://github.com/mime-types/mime-types-data/
-rubygem-msgpack 1.3.3 http://msgpack.org/
-rubygem-multi_json 1.15.0 https://github.com/intridea/multi_json
-rubygem-multipart-post 2.1.1 https://github.com/nicksieger/multipart-post
-rubygem-net-http-persistent 3.1.0 http://docs.seattlerb.org/net-http-persistent
-rubygem-netrc 0.11.0 https://github.com/geemus/netrc
-rubygem-oj 3.11.3 http://www.ohler.com/oj
-rubygem-prometheus-client 0.9.0 https://github.com/prometheus/client_ruby
-rubygem-public_suffix 4.0.5 https://simonecarletti.com/code/publicsuffix-ruby
-rubygem-quantile 0.2.1 http://github.com/matttproud/ruby_quantile_estimation
-rubygem-rake 13.0.1 https://github.com/ruby/rake
-rubygem-recursive-open-struct 1.1.2 https://github.com/aetherknight/recursive-open-struct
-rubygem-rest-client 2.1.0 https://github.com/rest-client/rest-client
-rubygem-sigdump 0.2.4 https://github.com/frsyuki/sigdump
-rubygem-syslog_protocol 0.9.2 https://github.com/eric/syslog_protocol
-rubygem-systemd-journal 1.3.3 https://github.com/ledbettj/systemd-journal
-rubygem-typhoeus 1.4.0 https://github.com/typhoeus/typhoeus
-rubygem-tzinfo 2.0.2 https://tzinfo.github.io
-rubygem-tzinfo-data 1.2020.1 https://tzinfo.github.io
-rubygem-unf 0.1.4 https://github.com/knu/ruby-unf
-rubygem-unf_ext 0.0.7.7 https://github.com/knu/ruby-unf_ext
-rubygem-uuidtools 2.1.5 https://github.com/sporkmonger/uuidtools
-rubygem-yajl-ruby 1.4.1 http://github.com/brianmario/yajl-ruby
+mime-types 3.3.1 https://github.com/mime-types/ruby-mime-types/
+mime-types-data 3.2020.0512 https://github.com/mime-types/mime-types-data/
+msgpack 1.3.3 http://msgpack.org/
+multi_json 1.15.0 https://github.com/intridea/multi_json
+multipart-post 2.1.1 https://github.com/nicksieger/multipart-post
+net-http-persistent 3.1.0 http://docs.seattlerb.org/net-http-persistent
+netrc 0.11.0 https://github.com/geemus/netrc
+oj 3.11.3 http://www.ohler.com/oj
+prometheus-client 0.9.0 https://github.com/prometheus/client_ruby
+public_suffix 4.0.5 https://simonecarletti.com/code/publicsuffix-ruby
+quantile 0.2.1 http://github.com/matttproud/ruby_quantile_estimation
+rake 13.0.1 https://github.com/ruby/rake
+recursive-open-struct 1.1.2 https://github.com/aetherknight/recursive-open-struct
+rest-client 2.1.0 https://github.com/rest-client/rest-client
 ruby-kafka 1.1.0 https://github.com/zendesk/ruby-kafka
 serverengine 2.2.1 https://github.com/fluent/serverengine
+sigdump 0.2.4 https://github.com/frsyuki/sigdump
 strptime 0.2.4 https://github.com/nurse/strptime
+syslog_protocol 0.9.2 https://github.com/eric/syslog_protocol
+systemd-journal 1.3.3 https://github.com/ledbettj/systemd-journal
 to_regexp 0.2.1 https://github.com/seamusabshere/to_regexp
+typhoeus 1.4.0 https://github.com/typhoeus/typhoeus
+tzinfo 2.0.2 https://tzinfo.github.io
+tzinfo-data 1.2020.1 https://tzinfo.github.io
+unf 0.1.4 https://github.com/knu/ruby-unf
+unf_ext 0.0.7.7 https://github.com/knu/ruby-unf_ext
+uuidtools 2.1.5 https://github.com/sporkmonger/uuidtools
+yajl-ruby 1.4.1 http://github.com/brianmario/yajl-ruby