DynamoDB Parallel Scan not splitting results
<p>I'm using the <code>Segment</code> and <code>TotalSegments</code> parameters to split my DynamoDB scan over multiple workers (as shown in the <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScanParallelScan" rel="nofollow">Parallel Scan section</a> of the developer guide).</p>

<p>However, all of the results get returned to one worker. What could be the issue here? Is there perhaps an issue with how I've implemented threading?</p>

<pre><code>import threading
import boto3

def scan_foo_table(foo, segment, total_segments):
    print 'Looking at segment ' + str(segment)
    session = boto3.session.Session()
    dynamoDbClient = session.client('dynamodb')
    response = dynamoDbClient.scan(
        TableName='FooTable',
        FilterExpression='bar=:bar',
        ExpressionAttributeValues={
            ':bar': {'S': bar}
        },
        Segment=segment,
        TotalSegments=total_segments,
    )
    print 'Segment ' + str(segment) + ' returned ' + str(len(response['Items'])) + ' items'

def create_threads(bar):
    thread_list = []
    total_threads = 3

    for i in range(total_threads):
        # Instantiate and store the thread
        thread = threading.Thread(target=scan_foo_table, args=(bar, i, total_threads))
        thread_list.append(thread)

    # Start threads
    for thread in thread_list:
        thread.start()

    # Block main thread until all threads are finished
    for thread in thread_list:
        thread.join()

def lambda_handler(event, context):
    create_threads('123')
</code></pre>

<p>Output:</p>

<pre><code>Looking at segment 0
Looking at segment 1
Looking at segment 2
Segment 1 returned 0 items
Segment 2 returned 0 items
Segment 0 returned 10000 items
</code></pre>

<p>One thing that jumps at me is the filter expression.</p>

<p>It is possible that the items that match your filter expression are located all in the first segment.</p>

<p>It is also worth noting that a parallel scan doesn't split the items, but it splits the key-space that is searched for items. Think of it as dividing a very large highway into multiple lanes.  It is possible le that most cars are in the fast lane and you won't see any cars in other lanes.</p>

<p>Though in this case it seems more likely that the filter expression is what is causing only one segment to return items.</p>

python amazon-dynamodb boto aws-lambda