Skip to content

Feature request: let client get the correct lastrowid when bulk insert #431

Description

@cbi-gshen

Hi,
When bulk insert a long list of data(thousands of rows) using cursor.executemany(), the cursor.lastrowid is not correct.
code is like this:

def get_data():
    return [
        ("db_tests_112312313", 1, 1, "testsetestst", i)
        for i in range(1, 3000)
    ]


def using_mysql_db(insert):
    db = MySQLdb.connect(hostname, username, password)

    cursor = db.cursor()
    sql_statement = """
        INSERT INTO test (name, id_creator, id_owner, description, id_collection)
        VALUES (%s, %s, %s, %s, %s)
    """

    results = cursor.executemany(sql_statement, get_data())

    print("lastrowid ", cursor.lastrowid)

I think this is an issue a lot of people experienced.
I am wondering if it's possible the PyMySQL can support that.
I did a small experiment like this, only assign the lastrowid once(the first execute() will set the correct lastrowid) when using executemany. and it works.

    def _do_execute_many(self, prefix, values, postfix, args, max_stmt_length, encoding):
        conn = self._get_db()
        escape = self._escape_args
        if isinstance(prefix, str):
            prefix = prefix.encode(encoding)
        if isinstance(values, str):
            values = values.encode(encoding)
        if isinstance(postfix, str):
            postfix = postfix.encode(encoding)
        sql = bytearray(prefix)
        args = iter(args)
        v = values % escape(next(args), conn)
        sql += v
        rows = 0
        tmp_lastrowid = 0  # added this line
        for arg in args:
            v = values % escape(arg, conn)
            if len(sql) + len(v) + len(postfix) + 1 > max_stmt_length:
                rows += self.execute(sql + postfix)
                if tmp_lastrowid:                              # added this line
                    self.lastrowid = tmp_lastrowid  # added this line
                else:                                                    # added this line
                    tmp_lastrowid = self.lastrowid   # added this line
                sql = bytearray(prefix)
            else:
                sql += b","
            sql += v
        rows += self.execute(sql + postfix)
        if tmp_lastrowid:                              # added this line
            self.lastrowid = tmp_lastrowid    # added this line
        self.rowcount = rows
        return rows

I am wordering if this is good idea. are there bigger concerns to do this?

Thanks,

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions