Support base64 encoded config files

[dengelke]

For configuring the collector the current two options that seem to be available are:

Environment variables:

docker run \
  snowplow/scala-stream-collector-kafka:2.4.1 \
  -Dcollector.interface=0.0.0.0 \
  -Dcollector.port=80 \
  -Dcollector.streams.good=goodstream \
  -Dcollector.streams.bad=badstream \
  -Dcollector.streams.sink.brokers=localhost:9092

or a pre-existing Config File:

docker run \
  snowplow/scala-stream-collector-kinesis:2.5.0 \
  --config /snowplow/config/snowplow-stream-collector-kinesis-2.5.0.hocon

However in the situation you are running on Fargate ECS it's not possible to use either one of these two approaches if you are setting paths i.e.

paths {
  "/com.acme/track"    = "/com.snowplowanalytics.snowplow/tp2"
  "/com.acme/redirect" = "/r/tp2"
  "/com.acme/iglu"     = "/com.snowplowanalytics.iglu/v1"
}

As ECS with Fargate does not make it possible to use config files and setting env variables with "/" in the key did not seem to be possible.

On the other hand, the enricher makes this easy to do by accepting base64 encoded config files see here for source code which would make it much easier to configure in different environments.

As a workaround in the meantime for anyone else with this issue it's possible to take the existing docker image and modify it by adding a new entrypoint.sh which seems to work nicely!

Dockerfile

# Use the original image
FROM snowplow/scala-stream-collector-kinesis:2.9.2

# Copy your entrypoint.sh into the image
COPY entrypoint.sh /home/snowplow/bin/entrypoint.sh

USER root

# Ensure the script is executable
RUN chmod +x /home/snowplow/bin/entrypoint.sh
RUN mkdir -p /home/snowplow/config
RUN chown 1001:0 /home/snowplow/config

USER 1001:0

# Set your new entrypoint
ENTRYPOINT ["/home/snowplow/bin/entrypoint.sh"]

entrypoint.sh

#!/bin/sh

# Check if the BASE64_CONFIG is not empty
if [ -n "$BASE64_CONFIG" ]; then
  # Decode the Base64 string and write to a config file
  echo "$BASE64_CONFIG" | base64 -d > /home/snowplow/config/config.hocon
else
  echo "BASE64_CONFIG is empty, using default configuration..."
  # You may handle this case differently based on your needs
fi

# Start your application using the decoded configuration file
exec /home/snowplow/bin/snowplow-stream-collector --config /home/snowplow/config/config.hocon "$@"