Azure Blob storage SDK: Switch off logging

Question

I have an application which is using azure.storage.blob python module.

Evidently, when I am executing it, the module is dumping its log data in the logger which I dont want because it is large and my own application info gets lost in between it.

Is there any way to remove the logger from azure.storage.blob python module?

log sample:

INFO:azure.storage.common.storageclient:Client-Request-ID=d5afebc0-aa84-11e8-be16-000d3ae070ae Outgoing request: Method=PUT, Path=/marketingcloud-raw-events/2018/8/27/bounce.csv, Query={'timeout': None}, Headers={'Content-Length': '38176', 'x-ms-blob-type': 'BlockBlob', 'x-ms-version': '2018-03-28', 'x-ms-lease-id': None, 'x-ms-client-request-id': 'd5afebc0-aa84-11e8-be16-000d3ae070ae', 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'User-Agent': 'Azure-Storage/1.3.0-1.3.1 (Python CPython 2.7.15; Linux 4.15.0-1021-azure)'

Answer 1

As Thomas mentioned in the comment, you could find the code of log data is encapsulated in the azure.storage.blob python sdk. As I know, there's no such switch to close this logging mechanism.

1.You can encapsulate the SDK by yourself and get rid of these redundant logs.

2.Or you just could try to add special tags to the application logs and filter out redundant other log data.

3.You could commit feedback to azure to make the the log data as optional.

Hope it helps you.

Answer 2

This is actually very easy to do. You can adjust any logger.

Just do this near the start of your program to switch verbosity from INFO to WARNING, or whatever you prefer.

logging.getLogger("azure.storage.common.storageclient").setLevel(logging.WARNING)

Answer 3

Some messages are related to http calls. Also try the following:

logger=logging.getLogger('azure.core.pipeline.policies.http_logging_policy')
logger.setLevel(logging.WARNING)

Answer 4

I had the same issue that the azure blob API was spamming my log stream. When you build a BlobClient you can also pass a custom logger which you can control as you like.

logger = logging.getLogger("logger_name")
logger.disabled = True
my_blob_client = BlobClient.from_connection_string("azure blob connection string", container_name="container", blob_name="blob", logger=logger)

Answer 5

See also https://docs.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-logging

where it shows how you can also use more general namespaces. For example:

import logging

# Set the logging level for all azure-storage-* libraries
logger = logging.getLogger('azure.storage')
logger.setLevel(logging.INFO)

# Set the logging level for all azure-* libraries
logger = logging.getLogger('azure')
logger.setLevel(logging.ERROR)

Answer 6

The heavy INFO logging from the azure.storage.blob library was filling my logs with REDACTED junk when doing basic requests.

2022-04-19 13:46:15,760 [Thread-3 (worker)] [INFO]  Request URL: 'https://storage-url/folder/blob'
Request method: 'GET'
Request headers:
'x-ms-range': 'REDACTED'
'x-ms-version': 'REDACTED'
'Accept': 'application/xml'
'User-Agent': 'azsdk-python-storage-blob/12.11.0 Python/3.10.4 (Windows-10-10.0.17763-SP0)'
'x-ms-date': 'REDACTED'
'x-ms-client-request-id': '43a4b6e8-bf93-11ec-a3c9-00224893b074'
'Authorization': 'REDACTED'
No body was attached to the request

Using @MRTN's answer I found that managing the logger where you generate the client and adding it to the logging parameter stopped the spam.

import logging 
from azure.storage.blob import BlobServiceClient

logger = logging.getLogger('azure.core.pipeline.policies.http_logging_policy').setLevel(logging.WARNING)

class AzureBlobDownloader:
    def __init__(self, account_url, access_key):
        self.blob_service_client: BlobServiceClient = BlobServiceClient(
            account_url=account_url, 
            credential=access_key,
            logging=logger
        )