'How to run Fluent-Bit in AWS-Lambda

I have built a more simple test image that works. The Dockerfile installs python, then Fluent-Bit, and then the CMD is a shell script that starts both processes.

Here is my Dockerfile

 ###############################################################################
# Python Install
###############################################################################

FROM python:3.8

RUN apt-get update -y \
    && apt-get install -y libgdal-dev\
    && apt-get install -y nano

RUN python -m venv /env \
    && /env/bin/pip install --upgrade pip \
    # && /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
    && runDeps="$(scanelf --needed --nobanner --recursive /env \
        | awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
        | sort -u \
        | xargs -r apk info --installed \
        | sort -u)"

COPY logs/python.log /fluent-bit/etc/
COPY ["script.py", "./"]


###############################################################################
# FluentBit Install
###############################################################################
COPY ["install.sh", "./install.sh"]
RUN chmod +x ./install.sh
RUN ./install.sh

# Configuration files
COPY conf/fluent-bit.conf \
     conf/parsers.conf \
     conf/parsers_java.conf \
     conf/parsers_extra.conf \
     conf/parsers_openstack.conf \
     conf/parsers_cinder.conf \
     conf/plugins.conf \
     /fluent-bit/etc/

COPY logs/python.log /fluent-bit/etc/

###############################################################################
# Start with a wrapper script
###############################################################################

COPY ["runboth.sh", "./runboth.sh"]
RUN chmod +x ./runboth.sh
CMD ./runboth.sh

Here is runboth.sh

#Start Python
#!/bin/bash
python script.py

#Start Fluent-Bit
/opt/fluent-bit/bin/fluent-bit -c ./fluent-bit/etc/fluent-bit.conf -d

And here is script.py

logger = logging.getLogger()
logger.setLevel(logging.INFO)

file_handler = logging.FileHandler('/fluent-bit/etc/python.log')
file_handler.setLevel(logging.INFO)


logger.addHandler(file_handler)
i=0
while i<3:
    logging.info(json.dumps({"Time": str(datetime.datetime.now()),"Message": "Here's a log from a python script"}))
    sleep(10)
    i+=1

The python script runs for a few seconds during which it prints logs to a log file. Once that process is done, Fluent-Bit starts up, collects the logs, and ships them to ElasticSearch.

Now I can't figure out how to adapt this so that it runs live as a Lambda function.

Lambda complains like this:

entrypoint requires the handler name to be the first argument

My current runboth.sh for Lambda is below, but I have tried many versions

#Start Python
#!/bin/bash
/usr/local/bin/python -m awslambdaric app.handler

#Start Fluent-Bit
/opt/fluent-bit/bin/fluent-bit -c ./fluent-bit/etc/fluent-bit.conf -d

There's not a lot of information out there on Fluent-Bit for Lambda other than an extension that's been abandoned.

If this is a bad idea, I'm happy to consider alternative ways of shipping Lambda logs to ES.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source