Faster way to perform bulk insert, while avoiding duplicates, with SQLAlchemy
<p>I'm using the following method to perform a bulk insert, and to optionally avoid inserting duplicates, with SQLAlchemy:</p>

<pre><code>def bulk_insert_users(self, users, allow_duplicates = False):
    if not allow_duplicates:
        users_new = []

        for user in users:
            if not self.SQL_IO.db.query(User_DB.id).filter_by(user_id = user.user_id).scalar():
                users_new.append(user)

        users = users_new

    self.SQL_IO.db.bulk_save_objects(users)
    self.SQL_IO.db.commit()
</code></pre>

<p>Can the above functionality be implemented such that the function is faster?</p>

<p>How many users do you have? You're querying for the users one at a time, every single iteration of that loop. You might have more luck querying for ALL user Ids, put them in a list, then check against that list.</p>

<pre><code>existing_users = #query for all user IDs
for user in new_users:
    if user not in existing_users:
        #do_stuff
</code></pre>

python sqlalchemy