'How to run Fluent-Bit in AWS-Lambda
I have built a more simple test image that works. The Dockerfile installs python, then Fluent-Bit, and then the CMD is a shell script that starts both processes.
Here is my Dockerfile
###############################################################################
# Python Install
###############################################################################
FROM python:3.8
RUN apt-get update -y \
&& apt-get install -y libgdal-dev\
&& apt-get install -y nano
RUN python -m venv /env \
&& /env/bin/pip install --upgrade pip \
# && /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)"
COPY logs/python.log /fluent-bit/etc/
COPY ["script.py", "./"]
###############################################################################
# FluentBit Install
###############################################################################
COPY ["install.sh", "./install.sh"]
RUN chmod +x ./install.sh
RUN ./install.sh
# Configuration files
COPY conf/fluent-bit.conf \
conf/parsers.conf \
conf/parsers_java.conf \
conf/parsers_extra.conf \
conf/parsers_openstack.conf \
conf/parsers_cinder.conf \
conf/plugins.conf \
/fluent-bit/etc/
COPY logs/python.log /fluent-bit/etc/
###############################################################################
# Start with a wrapper script
###############################################################################
COPY ["runboth.sh", "./runboth.sh"]
RUN chmod +x ./runboth.sh
CMD ./runboth.sh
Here is runboth.sh
#Start Python
#!/bin/bash
python script.py
#Start Fluent-Bit
/opt/fluent-bit/bin/fluent-bit -c ./fluent-bit/etc/fluent-bit.conf -d
And here is script.py
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler('/fluent-bit/etc/python.log')
file_handler.setLevel(logging.INFO)
logger.addHandler(file_handler)
i=0
while i<3:
logging.info(json.dumps({"Time": str(datetime.datetime.now()),"Message": "Here's a log from a python script"}))
sleep(10)
i+=1
The python script runs for a few seconds during which it prints logs to a log file. Once that process is done, Fluent-Bit starts up, collects the logs, and ships them to ElasticSearch.
Now I can't figure out how to adapt this so that it runs live as a Lambda function.
Lambda complains like this:
entrypoint requires the handler name to be the first argument
My current runboth.sh for Lambda is below, but I have tried many versions
#Start Python
#!/bin/bash
/usr/local/bin/python -m awslambdaric app.handler
#Start Fluent-Bit
/opt/fluent-bit/bin/fluent-bit -c ./fluent-bit/etc/fluent-bit.conf -d
There's not a lot of information out there on Fluent-Bit for Lambda other than an extension that's been abandoned.
If this is a bad idea, I'm happy to consider alternative ways of shipping Lambda logs to ES.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|