Updating MYSQL from daily CSV file - Skip already imported

问题: I am using the following script to copy data from a CSV file that is updated daily into a Mysql Database. csv_data = csv.reader(open('test.csv')) next(csv_data, None) for...

问题:

I am using the following script to copy data from a CSV file that is updated daily into a Mysql Database.

csv_data = csv.reader(open('test.csv'))
next(csv_data, None)
for row in csv_data:  
    with connection.cursor() as cursor:
         cursor.execute(("INSERT INTO test(`1` ,`2`,`3` .......) VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"), (row[0]........))

The CSV currently has over 40,000 rows and will continue to grow meaning it will take hours to do.

I know I can add a unique identifier onto the database the will stop duplicated and do INSERT IGNORE to skip over it but is there anything else I can do to speed the process?


回答1:

"INSERT IGNORE " will resolve your issue.

  • 发表于 2019-03-02 23:49
  • 阅读 ( 191 )
  • 分类:sof

条评论

请先 登录 后评论
不写代码的码农
小编

篇文章

作家榜 »

  1. 小编 文章
返回顶部
部分文章转自于网络,若有侵权请联系我们删除