Elegant way to check if a nested key exists in a dict?

Question

Is there are more readable way to check if a key buried in a dict exists without checking each level independently?

Lets say I need to get this value in a object buried (example taken from Wikidata):

x = s['mainsnak']['datavalue']['value']['numeric-id']

To make sure that this does not end with a runtime error it is necessary to either check every level like so:

if 'mainsnak' in s and 'datavalue' in s['mainsnak'] and 'value' in s['mainsnak']['datavalue'] and 'nurmeric-id' in s['mainsnak']['datavalue']['value']:
    x = s['mainsnak']['datavalue']['value']['numeric-id']

The other way I can think of to solve this is wrap this into a try catch construct which I feel is also rather awkward for such a simple task.

I am looking for something like:

x = exists(s['mainsnak']['datavalue']['value']['numeric-id'])

which returns True if all levels exists.

Answer 1

To be brief, with Python you must trust it is easier to ask for forgiveness than permission

try:
    x = s['mainsnak']['datavalue']['value']['numeric-id']
except KeyError:
    pass

---

The answer

Here is how I deal with nested dict keys:

def keys_exists(element, *keys):
    '''
    Check if *keys (nested) exists in `element` (dict).
    '''
    if not isinstance(element, dict):
        raise AttributeError('keys_exists() expects dict as first argument.')
    if len(keys) == 0:
        raise AttributeError('keys_exists() expects at least two arguments, one given.')

    _element = element
    for key in keys:
        try:
            _element = _element[key]
        except KeyError:
            return False
    return True

Example:

data = {
    "spam": {
        "egg": {
            "bacon": "Well..",
            "sausages": "Spam egg sausages and spam",
            "spam": "does not have much spam in it"
        }
    }
}

print 'spam (exists): {}'.format(keys_exists(data, "spam"))
print 'spam > bacon (do not exists): {}'.format(keys_exists(data, "spam", "bacon"))
print 'spam > egg (exists): {}'.format(keys_exists(data, "spam", "egg"))
print 'spam > egg > bacon (exists): {}'.format(keys_exists(data, "spam", "egg", "bacon"))

Output:

spam (exists): True
spam > bacon (do not exists): False
spam > egg (exists): True
spam > egg > bacon (exists): True

It loop in given element testing each key in given order.

I prefere this to all variable.get('key', {}) methods I found because it follows EAFP.

Function except to be called like: keys_exists(dict_element_to_test, 'key_level_0', 'key_level_1', 'key_level_n', ..). At least two arguments are required, the element and one key, but you can add how many keys you want.

If you need to use kind of map, you can do something like:

expected_keys = ['spam', 'egg', 'bacon']
keys_exists(data, *expected_keys)

Answer 2

You could use .get with defaults:

s.get('mainsnak', {}).get('datavalue', {}).get('value', {}).get('numeric-id')

but this is almost certainly less clear than using try/except.

Answer 3

Try/except seems to be most pythonic way to do that.
The following recursive function should work (returns None if one of the keys was not found in the dict):

def exists(obj, chain):
    _key = chain.pop(0)
    if _key in obj:
        return exists(obj[_key], chain) if chain else obj[_key]

myDict ={
    'mainsnak': {
        'datavalue': {
            'value': {
                'numeric-id': 1
            }
        }
    }
}

result = exists(myDict, ['mainsnak', 'datavalue', 'value', 'numeric-id'])
print(result)
>>> 1

Answer 4

Python 3.8 +

dictionary = {
    "main_key": {
        "sub_key": "value",
    },
}

if sub_key_value := dictionary.get("main_key", {}).get("sub_key"):
    print(f"The key 'sub_key' exists in dictionary[main_key] and it's value is {sub_key_value}")
else:
    print("Key 'sub_key' doesn't exists or their value is Falsy")

Extra

A little but important clarification.

In the previous code block, we verify that a key exists in a dictionary but that its value is also Truthy. Most of the time, this is what people are really looking for, and I think this is what the OP really wants. However, it is not really the most "correct" answer, since if the key exists but its value is False, the above code block will tell us that the key does not exist, which is not true.

So, I leet here a more correct answer:

dictionary = {
    "main_key": {
        "sub_key": False,
    },
}

if "sub_key" in dictionary.get("main_key", {}):
    print(f"The key 'sub_key' exists in dictionary[main_key] and it's value is {dictionary['main_key']['sub_key']}")
else:
    print("Key 'sub_key' doesn't exists")

Answer 5

I suggest you to use python-benedict, a solid python dict subclass with full keypath support and many utility methods.

You just need to cast your existing dict:

s = benedict(s)

Now your dict has full keypath support and you can check if the key exists in the pythonic way, using the in operator:

if 'mainsnak.datavalue.value.numeric-id' in s:
    # do stuff

Here the library repository and the documentation: https://github.com/fabiocaccamo/python-benedict

Note: I am the author of this project

Answer 6

You can use pydash to check if exists: http://pydash.readthedocs.io/en/latest/api.html#pydash.objects.has

Or get the value (you can even set default - to return if doesn't exist): http://pydash.readthedocs.io/en/latest/api.html#pydash.objects.has

Here is an example:

>>> get({'a': {'b': {'c': [1, 2, 3, 4]}}}, 'a.b.c[1]')
2

Answer 7

The try/except way is the most clean, no contest. However, it also counts as an exception in my IDE, which halts execution while debugging.

Furthermore, I do not like using exceptions as in-method control statements, which is essentially what is happening with the try/catch.

Here is a short solution which does not use recursion, and supports a default value:

def chained_dict_lookup(lookup_dict, keys, default=None):
    _current_level = lookup_dict
    for key in keys:
        if key in _current_level:
            _current_level = _current_level[key]
        else:
            return default
    return _current_level

Answer 8

The accepted answer is a good one, but here is another approach. It's a little less typing and a little easier on the eyes (in my opinion) if you end up having to do this a lot. It also doesn't require any additional package dependencies like some of the other answers. Have not compared performance.

import functools

def haskey(d, path):
    try:
        functools.reduce(lambda x, y: x[y], path.split("."), d)
        return True
    except KeyError:
        return False

# Throwing in this approach for nested get for the heck of it...
def getkey(d, path, *default):
    try:
        return functools.reduce(lambda x, y: x[y], path.split("."), d)
    except KeyError:
        if default:
            return default[0]
        raise

Usage:

data = {
    "spam": {
        "egg": {
            "bacon": "Well..",
            "sausages": "Spam egg sausages and spam",
            "spam": "does not have much spam in it",
        }
    }
}

(Pdb) haskey(data, "spam")
True
(Pdb) haskey(data, "spamw")
False
(Pdb) haskey(data, "spam.egg")
True
(Pdb) haskey(data, "spam.egg3")
False
(Pdb) haskey(data, "spam.egg.bacon")
True

Original inspiration from the answers to this question.

EDIT: a comment pointed out that this only works with string keys. A more generic approach would be to accept an iterable path param:

def haskey(d, path):
    try:
        functools.reduce(lambda x, y: x[y], path, d)
        return True
    except KeyError:
        return False

(Pdb) haskey(data, ["spam", "egg"])
True

Answer 9

I had the same problem and recent python lib popped up:
https://pypi.org/project/dictor/
https://github.com/perfecto25/dictor

So in your case:

from dictor import dictor

x = dictor(s, 'mainsnak.datavalue.value.numeric-id')

---

Personal note:
I don't like 'dictor' name, since it doesn't hint what it actually does. So I'm using it like:

from dictor import dictor as extract
x = extract(s, 'mainsnak.datavalue.value.numeric-id')

Couldn't come up with better naming than extract. Feel free to comment, if you come up with more viable naming. safe_get, robust_get didn't felt right for my case.

Answer 10

Another way:

def does_nested_key_exists(dictionary, nested_key):
    exists = nested_key in dictionary
    if not exists:
        for key, value in dictionary.items():
            if isinstance(value, dict):
                exists = exists or does_nested_key_exists(value, nested_key)
    return exists

Answer 11

I wrote a data parsing library called dataknead for cases like this, basically because i got frustrated by the JSON the Wikidata API returns as well.

With that library you could do something like this

from dataknead import Knead

numid = Knead(s).query("mainsnak/datavalue/value/numeric-id").data()

if numid:
    # Do something with `numeric-id`

Answer 12

Using dict with defaults is concise and appears to execute faster than using consecutive if statements.

Try it yourself:

import timeit

timeit.timeit("'x' in {'a': {'x': {'y'}}}.get('a', {})")
# 0.2874350370002503

timeit.timeit("'a' in {'a': {'x': {'y'}}} and 'x' in {'a': {'x': {'y'}}}['a']")
# 0.3466246419993695

Answer 13

The selected answer works well on the happy path, but there are a couple obvious issues to me. If you were to search for ["spam", "egg", "bacon", "pizza"], it would throw a type error due to trying to index "well..." using the string "pizza". Like wise, if you replaced pizza with 2, it would use that to get the index 2 from "Well..."

Selected Answer Output Issues:

data = {
    "spam": {
        "egg": {
            "bacon": "Well..",
            "sausages": "Spam egg sausages and spam",
            "spam": "does not have much spam in it"
        }
    }
}

print(keys_exists(data, "spam", "egg", "bacon", "pizza"))
>> TypeError: string indices must be integers

print(keys_exists(data, "spam", "egg", "bacon", 2)))
>> l

I also feel that using try except can be a crutch that we might too quickly rely on. Since I believe we already need to check for the type, might as well remove the try except.

Solution:

def dict_value_or_default(element, keys=[], default=Undefined):
    '''
    Check if keys (nested) exists in `element` (dict).
    Returns value if last key exists, else returns default value
    '''
    if not isinstance(element, dict):
        return default

    _element = element
    for key in keys:
        # Necessary to ensure _element is not a different indexable type (list, string, etc).  
        # get() would have the same issue if that method name was implemented by a different object
        if not isinstance(_element, dict) or key not in _element:
            return default

        _element = _element[key]
        
    return _element 

Output:

print(dict_value_or_default(data, ["spam", "egg", "bacon", "pizza"]))
>> INVALID

print(dict_value_or_default(data, ["spam", "egg", "bacon", 2]))
>> INVALID

print(dict_value_or_default(data, ["spam", "egg", "bacon"]))
>> "Well..."

Answer 14

If you can suffer testing a string representation of the object path then this approach might work for you:

def exists(str):
    try:
        eval(str)
        return True
    except:
        return False

exists("lst['sublist']['item']")

Answer 15

There are many great answers. here is my humble take on it. Added check for array of dictionaries as well. Please note that I am not checking for arguments validity. I used part Arnot's code above. I added this answer because a I got a use case that requires checking array or dictionaries in my data. Here is the code:

def keys_exists(element, *keys):
    '''
    Check if *keys (nested) exists in `element` (dict).
    '''
    
    retval=False
    if isinstance(element,dict):
        for key,value in element.items():
            for akey in keys:
                if element.get(akey) is not None:
                    return True
            if isinstance(value,dict) or isinstance(value,list):
                retval= keys_exists(value, *keys)
            
    elif isinstance(element, list):
        for val in element:
            if isinstance(val,dict) or isinstance(val,list):
                retval=keys_exists(val, *keys)

    return retval