how to insert a nested dict to dynamodb with boto3?

Question

we construct some data which is in nested format, it contains 3 keys. the time, id and others where id contains another nested dict. We have no idea how to create table for this or just how to insert such data to that specific table. We have tried to create a table with hashkey called id, but when we use

dynamodb.batch_write_item(RequestItems={
    'table': [{ 'PutRequest': { 'Item':whatweget}}
    }]
})

it shows thatbotocore.exceptions.ClientError: An error occurred (Validation Exception) when calling the BatchWriteItem operation: The prov ided key element does not match the schema can anyone offer some help?

Answer 1

You can create a table using DynamoDB console at your aws dashboard, you can view detailed instructions through dynamodb documentations.

For the nested dict you have multiple options, first you can just insert the values in id as xmls or jsons, since dynamodb will accept it as a string, if you want to have more strict rules regarding table operations you can just create a field for id and other fields for the supposed to be nested inside id fields, they don’t have to be present physically inside id

You can insert data to your created table using many ways, simplest is to add them manually through the aws dynamoDB consol, or you can use boto3, but you have to have a table first of course.

Answer 2

The first thing to clarify is how you are interacting with DynamoDB. When using boto3, there are two approaches to making requests:

• Boto3 DynamoDB Client - low level: you write more code - more fine grained control over operations.
• Boto3 DynamoDB Service Resource - high level - you write less code, a lot of the heavily lifting is done for you.

Generally, the Service Resource is easier to use and is preferable unless you have specific needs.

The other thing to consider is that storing nested data makes querying and filtering difficult, you will likely need to handle this after retrieving the data.

In the following, we'll create a table with a single index and add additional data alongside, including nested data, without defining the structure for the other fields.

Create table

Assuming you use the CLI, create a table with:

aws dynamodb create-table \
    --table-name my-table \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --key-schema AttributeName=id,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST

This creates a table with a single indexed id attribute.

Insert items

The following will insert 2 items (max 25), and will automatically handle converting Python data types to DynamoDB attribute types - one of the benefits of the Service Resource over the low-level client.

import boto3
dynamodb = boto3.resource('dynamodb')
dynamodb.batch_write_item(
  RequestItems={
    'my-table': [
      {
        'PutRequest': {
          'Item': {
            'id': 'id1',
            'newAttribute': 'some new value 1',
            'anotherAttribute': 'another new value 1',
            'someDict': {
              'a': 1,
              'b': 2
            }
          }
        },
      },
      {
        'PutRequest': {
          'Item': {
            'id': 'id2',
            'newAttribute': 'some new value 2',
            'anotherAttribute': 'another new value 2',
          }
        },
      },
    ]
  }
)

In what follows, we see automatic creation of the new attributes and auto inference of the data type, and are therefore able to query and filter. The map, not so. Consider carefully your querying strategy if you decide to store nested data.