How to use and update same variable across multiple modules in python independently?

Question

I have 3 files and I want to do something like below

[1] conf.py

var = 10  # Intialized with 10 (start)

[2] file_1.py

import conf
print(conf.var)   # Prints 10
conf.var = 1000   # Updated to 1000

[3] file_2.py

import conf
print(conf.var)   # Prints 1000
conf.var = 9999   # Updated to 9999

I want something like this. Assume that the files, file_1 and file_2 will be running and will stay in memory unless pressed CTRL+C. How can I change var in other 2 files and persist the value for it? Final value for var should be 9999 if we were to use it in file_3 like other 2 files. [It should print 9999] and so on.

Execution order file_1.py -> file_2.py.

Help me with identifying some way to do it or some module/package that can handle this.

Thanks! :)

Answer 1

I worked on this problem. I will suggest to use shared memory cache providers like Redis or memcached. Both start a server instance that we need to connect and use it like a key-value store. Note than values should be of type str or bytes.

Memcached

Install memcached

sudo apt install memcached

Start Server with default settings

You need to start the server before using it or you can set it as a service in linux

sudo service memcached start

Install python memcached client

pip install pymemcache

Sample Code

from pymemcache.client import base

client = base.Client(('localhost', 11211))
client.set('some_key', 'stored value')

client.get('some_key')  # Returns "stored value"

Redis

The process is exact same for redis

Install redis

sudo apt install redis

Start Server with default settings

You need to start the server before using it or you can set it as a service in linux

sudo service redis start

Install python redis client

pip install redis

Sample Code

import redis

client = redis.Redis()  # Default settings/credentials
client.set('some_key', 'stored value')

client.get('some_key')  # Returns "stored value"

Usage

• Redis gives more safety and fault tolerance.
• Both has some memory limits but is configurable. If your data is more in size, use files or db suitable to your problem.
• Both are easy to configure

Answer 2

Couldn't you use a class and never initialize an object?

class Conf:
    var = 10

You would then update Conf with:

from conf import Conf

...

Conf.var = ...

Just never use Conf()..., as this creates an object instance.

Answer 3

Consider this approach:

class Lookup:
    # For allowing storing and looking up variables using variable sets of key pairs.
    # Each key pair is a key-value pair tuple.
    # The entire set of key pair tuples uniquely stores and retreives a variable value.
    # frozenset is used to achieve the benifit of a set while ensuring immutability, so it can be hashed for use as a dictionary key.
    lookupDictionary = {}
    def put(*, keyPairs, value):
        Lookup.lookupDictionary[frozenset(keyPairs)] = value
    def get(*, keyPairs, default, matchWildcard = False): # No substring searching. only '*' can be used for searching for an entire string
        if matchWildcard and True in [bool(x) for x in [(y[1] == '*') for y in keyPairs]]:
            # If we are here, we need to match using wildcard. 
            valuedKeyPairs = set([y for y in keyPairs if not y[1] == '*'])  # Separating value pairs from wildcard pairs
            starKeyPairs = set(keyPairs) - valuedKeyPairs
            starKeyPairNames = [x[0] for x in starKeyPairs]
            return [Lookup.lookupDictionary[i] for i in Lookup.lookupDictionary.keys() if set(valuedKeyPairs).issubset(i) and sorted(starKeyPairNames) == sorted([x[0] for x in i-valuedKeyPairs])]
        return Lookup.lookupDictionary.get(frozenset(keyPairs), default)
    def getAllValues(*, keyPairs):
        # Returrns all qualifying values to part or all of the key pairs
        return [Lookup.lookupDictionary[i] for i in set(Lookup.lookupDictionary.keys()) if set(keyPairs).issubset(i)]
class utils:
    def setCacheEntry(*, zipFileName, functionName, value):
        Lookup.put(keyPairs = [('__zipFileName__',zipFileName),('__functionName__',functionName)],value=value)

    def getCacheEntry(*, zipFileName, functionName, default):
        return Lookup.get(keyPairs = [('__zipFileName__',zipFileName),('__functionName__',functionName)],default=default)

from Lookup import utils
from Lookup import Lookup
utils.setCacheEntry(zipFileName='FileName.zip',functionName='some_func',value='some_value')
utils.getCacheEntry(zipFileName='FileName.zip',functionName='some_func', default=1)

Output: 'some_value'

This is a more simple case:

from Lookup import Lookup

def setParamValue(*, paramName, value):
    Lookup.put(keyPairs = [('__paramName__',paramName),],value=value)
    
def getParamValue(*, paramName, default):
    return Lookup.get(keyPairs = [('__paramName__',paramName),],default=default, matchWildcard=True)
    
setParamValue(paramName='name', value='John')
setParamValue(paramName='age', value=23)

getParamValue(paramName='name', default='Unknown')

Output: John

getParamValue(paramName='age', default=-1)

Output: 23

getParamValue(paramName='*', default='Unknown')

Output: ['John', 23]

If you need to communicate variables/objects between different programs, the above does not work. To do it you may use pickle to dump and load objects across the OS: I have not tested it to see if it does deep or shallow dumping of objects, but probably the later.

import pickle

# Dumping

with open('object_pickle','wb') as f:
    pickle.dump('some_value', f)    

# Loading
    
with open('object_pickle','rb') as f2:
    var = pickle.load(f2)

print(var)

Output: 'some_value'