{"data":{"post":{"fields":{"slug":"/quick-import-of-mysql-database-dump/","date":"2018-09-28"},"html":"

Recently I’ve faced a nagging problem in a project I was working on - I was spending too much time on importing again and again database dump that we were using for development. It was either when I was switching between branches or when I needed to reproduce some problems that were reported and I didn’t want my previous actions to have any effect.

\n

Since database we were using was quite large (almost 3 GB) plus we needed to run some time-consuming migrations after it was imported it took often over 30 minutes for environment to be ready - and that’s definitely too much time to lose. I had to figure out something that could make cleaning up my environment significantly faster.

\n

We’re using Docker Compose setup to run our project locally and for initial testing. The basic structure of docker-compose.yml file for any project based on PHP and MySQL would look along these lines:

\n
version: '3'\n\nservices:\n  # This is a container for PHP application.\n  app:\n    # ...\n    container_name: foo_app\n    depends_on:\n      - db\n\n  db:\n    image: mysql:5.7\n    # ...\n    container_name: foo_db\n    volumes:\n      # Some configuration attached using a volume.\n      - ./mysql/config.cnf:/etc/mysql/conf.d/config.cnf
\n

We don’t use named volume to keep MySQL data as we often rebuild the whole application and we don’t have a need for retaining them. But when you look at the documentation of official MySQL Docker image you’ll notice that it’s possible to mount a volume to /var/lib/mysql so that all databases are preserved.

\n

It becomes apparent that you would be able to quickly replace whole database contents if you would be able to change the content of this volume. Since this would only require some files to be copied (without building indexes, checking foreign keys, locking etc. as it is when .sql files are imported via MySQL client) it would be much faster then ordinary import.

\n

The plan is as follows:

\n
    \n
  1. Alter docker-compose.yml to mount a named data volume for MySQL service.
    2. \n
  2. Import database dump you want to preserve using MySQL client.
    3. \n
  3. Clone named data volume.
    4. \n
  4. When database needs to be cleaned up use previously cloned named data volume and clone it back.
    5. \n
\n

First following changes are required to our docker-compose.yml file:

\n
version: '3'\n\nservices:\n  # This is a container for PHP application.\n  app:\n    # ...\n    container_name: foo_app\n    depends_on:\n      - db\n\n  db:\n    image: mysql:5.7\n    # ...\n    container_name: foo_db\n    volumes:\n      # Some configuration attached using a volume.\n      - ./mysql/config.cnf:/etc/mysql/conf.d/config.cnf\n      # We add this named volume for MySQL data.\n      - foo_db_data:/var/lib/mysql\n\nvolumes:\n  # We define this volume as external - we will be preparing it manually.\n  foo_db_data:\n    external: true
\n

So let’s create this named volume now:

\n
# On host\ndocker volume create --name foo_db_data
\n

We can run this new setup and import our database with `mysql` command:

\n
# On host\ndocker-compose up -d\ndocker exec -it foo_app bash\n# Inside foo_app container\nmysql -uuser -p -hdb foo < foo.sql
\n

Following this we will create another named volume that will contain a copy of our data. We can use a simple Bash script to copy all files from foo_db_data volume to foo_db_data_clone volume - visit Guido Diepen’s blog to find out more about how it works. Basically it uses Alpine image to which both source and destination volumes are mounted and that copies all files between them. The script also creates destination volume for us. We invoke it as follows:

\n
# On host\n./docker_clone_volume.sh foo_db_data foo_db_data_clone
\n

Now we have our backup ready!

\n

Let’s say we did some work and now we want to clean up the database. We have to remove our foo_db_data volume so it can be cloned from foo_db_data_clone. In order to do it we also have to remove our foo_db container since named volumes cannot be removed if they are used by any container:

\n
# On host\ndocker-compose stop\ndocker rm foo_db\ndocker volume rm foo_db_data\n./docker_clone_volume.sh foo_db_data_clone foo_db_data\ndocker-compose up -d
\n

And that’s all we have to do! Using this solution I went down form over 30 minutes to about 30 seconds - so much more time for diving into code!

","frontmatter":{"title":"Quick import of MySQL database dump","subTitle":"How to import 3 GB database dump in under 30 seconds","postAuthor":"Mariusz Bąk (malef)","cover":"/img/docker.png"}},"tags":{"group":[{"name":"$OSTYPE","totalCount":1,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}}]},{"name":".env","totalCount":1,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}}]},{"name":".nvrmc","totalCount":1,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}}]},{"name":"AUTO_INCREMENT","totalCount":1,"edges":[{"node":{"fields":{"slug":"/is-mysql-autoincrement-really-monotonic/"}}}]},{"name":"Angular","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"Command","totalCount":1,"edges":[{"node":{"fields":{"slug":"/complex-command-handler-in-javascript/"}}}]},{"name":"DQL","totalCount":2,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}},{"node":{"fields":{"slug":"/mysql-decimal-format/"}}}]},{"name":"Data Transfer Object","totalCount":1,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}}]},{"name":"Docker","totalCount":3,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}},{"node":{"fields":{"slug":"/quick-import-of-mysql-database-dump/"}}},{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"Docker Compose","totalCount":3,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}},{"node":{"fields":{"slug":"/quick-import-of-mysql-database-dump/"}}},{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"Doctrine","totalCount":2,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}},{"node":{"fields":{"slug":"/mysql-decimal-format/"}}}]},{"name":"Doctrine NEW","totalCount":1,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}}]},{"name":"ES7","totalCount":1,"edges":[{"node":{"fields":{"slug":"/async-await-with-express/"}}}]},{"name":"Elastica","totalCount":1,"edges":[{"node":{"fields":{"slug":"/elasticsearch-script-unknown-field-source-parser-not-found/"}}}]},{"name":"Elasticsearch","totalCount":1,"edges":[{"node":{"fields":{"slug":"/elasticsearch-script-unknown-field-source-parser-not-found/"}}}]},{"name":"Elasticsearch script","totalCount":1,"edges":[{"node":{"fields":{"slug":"/elasticsearch-script-unknown-field-source-parser-not-found/"}}}]},{"name":"Express","totalCount":1,"edges":[{"node":{"fields":{"slug":"/async-await-with-express/"}}}]},{"name":"Git","totalCount":2,"edges":[{"node":{"fields":{"slug":"/git-not-detecting-renames-quick-workarounds/"}}},{"node":{"fields":{"slug":"/git-reverting-branch-removal/"}}}]},{"name":"Git renamed","totalCount":1,"edges":[{"node":{"fields":{"slug":"/git-not-detecting-renames-quick-workarounds/"}}}]},{"name":"HTTP","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-does-slack-communicate-with-spotify/"}}}]},{"name":"JavaScript","totalCount":3,"edges":[{"node":{"fields":{"slug":"/async-await-with-express/"}}},{"node":{"fields":{"slug":"/complex-command-handler-in-javascript/"}}},{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}}]},{"name":"Kafka","totalCount":1,"edges":[{"node":{"fields":{"slug":"/is-mysql-autoincrement-really-monotonic/"}}}]},{"name":"MySQL","totalCount":4,"edges":[{"node":{"fields":{"slug":"/mysql-decimal-format/"}}},{"node":{"fields":{"slug":"/is-mysql-autoincrement-really-monotonic/"}}},{"node":{"fields":{"slug":"/php-with-mysql-8/"}}},{"node":{"fields":{"slug":"/quick-import-of-mysql-database-dump/"}}}]},{"name":"MySQL 8","totalCount":1,"edges":[{"node":{"fields":{"slug":"/php-with-mysql-8/"}}}]},{"name":"Node.js","totalCount":1,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}}]},{"name":"ORM","totalCount":1,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}}]},{"name":"PDO","totalCount":1,"edges":[{"node":{"fields":{"slug":"/php-with-mysql-8/"}}}]},{"name":"PDO_MYSQL","totalCount":1,"edges":[{"node":{"fields":{"slug":"/php-with-mysql-8/"}}}]},{"name":"PHP","totalCount":6,"edges":[{"node":{"fields":{"slug":"/doctrine-new-dql-operator-and-objects/"}}},{"node":{"fields":{"slug":"/mysql-decimal-format/"}}},{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}},{"node":{"fields":{"slug":"/php-with-mysql-8/"}}},{"node":{"fields":{"slug":"/elasticsearch-script-unknown-field-source-parser-not-found/"}}},{"node":{"fields":{"slug":"/how-to-write-a-phing-target-autocomplete-bash-script/"}}}]},{"name":"Phing","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-write-a-phing-target-autocomplete-bash-script/"}}}]},{"name":"Promise","totalCount":1,"edges":[{"node":{"fields":{"slug":"/complex-command-handler-in-javascript/"}}}]},{"name":"Python","totalCount":1,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}}]},{"name":"React","totalCount":2,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}},{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"Redux","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}}]},{"name":"Redux store","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}}]},{"name":"SPA","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"Slack","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-does-slack-communicate-with-spotify/"}}}]},{"name":"Spotify","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-does-slack-communicate-with-spotify/"}}}]},{"name":"Travis","totalCount":2,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}},{"node":{"fields":{"slug":"/elasticsearch-script-unknown-field-source-parser-not-found/"}}}]},{"name":"Vue","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"XDEBUG_CONFIG","totalCount":1,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}}]},{"name":"XDebug","totalCount":1,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}}]},{"name":"ai","totalCount":1,"edges":[{"node":{"fields":{"slug":"/weekly-ai-bites-last-manual-gap-qa-testing/"}}}]},{"name":"amazon","totalCount":1,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}}]},{"name":"apple-watch","totalCount":1,"edges":[{"node":{"fields":{"slug":"/aws-cognito-without-library/"}}}]},{"name":"asp.net core","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"async","totalCount":2,"edges":[{"node":{"fields":{"slug":"/async-await-with-express/"}}},{"node":{"fields":{"slug":"/complex-command-handler-in-javascript/"}}}]},{"name":"autocomplete","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-write-a-phing-target-autocomplete-bash-script/"}}}]},{"name":"await","totalCount":2,"edges":[{"node":{"fields":{"slug":"/async-await-with-express/"}}},{"node":{"fields":{"slug":"/complex-command-handler-in-javascript/"}}}]},{"name":"aws","totalCount":3,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}},{"node":{"fields":{"slug":"/time-out-of-sync-in-aws-ec2/"}}},{"node":{"fields":{"slug":"/aws-cognito-without-library/"}}}]},{"name":"aws.cli","totalCount":1,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}}]},{"name":"bash","totalCount":2,"edges":[{"node":{"fields":{"slug":"/cross-system-xdebug-docker-compose-setup/"}}},{"node":{"fields":{"slug":"/how-to-write-a-phing-target-autocomplete-bash-script/"}}}]},{"name":"bitbucket","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-quickly-remove-merged-remote-branches/"}}}]},{"name":"blazor","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"build","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"builder pattern","totalCount":1,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}}]},{"name":"c#","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"ci","totalCount":1,"edges":[{"node":{"fields":{"slug":"/unresolved-check-from-travis-on-github-pull-request/"}}}]},{"name":"claude-code","totalCount":1,"edges":[{"node":{"fields":{"slug":"/weekly-ai-bites-last-manual-gap-qa-testing/"}}}]},{"name":"cognito","totalCount":1,"edges":[{"node":{"fields":{"slug":"/aws-cognito-without-library/"}}}]},{"name":"colors","totalCount":1,"edges":[{"node":{"fields":{"slug":"/naming-sass-color-variables/"}}}]},{"name":"continuous integration","totalCount":1,"edges":[{"node":{"fields":{"slug":"/unresolved-check-from-travis-on-github-pull-request/"}}}]},{"name":"decimal","totalCount":1,"edges":[{"node":{"fields":{"slug":"/mysql-decimal-format/"}}}]},{"name":"docker","totalCount":1,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}}]},{"name":"ec2","totalCount":1,"edges":[{"node":{"fields":{"slug":"/time-out-of-sync-in-aws-ec2/"}}}]},{"name":"environment","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"equals","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"flysystem","totalCount":1,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}}]},{"name":"front-end","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"git","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-quickly-remove-merged-remote-branches/"}}}]},{"name":"git commit","totalCount":1,"edges":[{"node":{"fields":{"slug":"/git-not-detecting-renames-quick-workarounds/"}}}]},{"name":"git reflog","totalCount":1,"edges":[{"node":{"fields":{"slug":"/git-reverting-branch-removal/"}}}]},{"name":"git status","totalCount":1,"edges":[{"node":{"fields":{"slug":"/git-not-detecting-renames-quick-workarounds/"}}}]},{"name":"github","totalCount":3,"edges":[{"node":{"fields":{"slug":"/pull-request-templates-on-github/"}}},{"node":{"fields":{"slug":"/unresolved-check-from-travis-on-github-pull-request/"}}},{"node":{"fields":{"slug":"/how-to-quickly-remove-merged-remote-branches/"}}}]},{"name":"guide","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"hashcode","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"hibernate","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"java","totalCount":2,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}},{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"java.time","totalCount":1,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}}]},{"name":"jaxb","totalCount":1,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}}]},{"name":"jdk8","totalCount":1,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}}]},{"name":"jpa","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"maven","totalCount":1,"edges":[{"node":{"fields":{"slug":"/make-jaxb-great-again/"}}}]},{"name":"middleware","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}}]},{"name":"mono","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"multi-stage","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"naming","totalCount":1,"edges":[{"node":{"fields":{"slug":"/naming-sass-color-variables/"}}}]},{"name":"networking","totalCount":1,"edges":[{"node":{"fields":{"slug":"/time-out-of-sync-in-aws-ec2/"}}}]},{"name":"ntp","totalCount":1,"edges":[{"node":{"fields":{"slug":"/time-out-of-sync-in-aws-ec2/"}}}]},{"name":"nvm","totalCount":1,"edges":[{"node":{"fields":{"slug":"/travis-problematic-python-and-node/"}}}]},{"name":"persistence","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"port 4381","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-does-slack-communicate-with-spotify/"}}}]},{"name":"pr","totalCount":1,"edges":[{"node":{"fields":{"slug":"/pull-request-templates-on-github/"}}}]},{"name":"production","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"project","totalCount":1,"edges":[{"node":{"fields":{"slug":"/pull-request-templates-on-github/"}}}]},{"name":"pull","totalCount":1,"edges":[{"node":{"fields":{"slug":"/pull-request-templates-on-github/"}}}]},{"name":"qa","totalCount":1,"edges":[{"node":{"fields":{"slug":"/weekly-ai-bites-last-manual-gap-qa-testing/"}}}]},{"name":"razor","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"reducer","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}}]},{"name":"regex","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-write-a-phing-target-autocomplete-bash-script/"}}}]},{"name":"repo","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-quickly-remove-merged-remote-branches/"}}}]},{"name":"repository","totalCount":1,"edges":[{"node":{"fields":{"slug":"/how-to-quickly-remove-merged-remote-branches/"}}}]},{"name":"s3","totalCount":1,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}}]},{"name":"sass","totalCount":1,"edges":[{"node":{"fields":{"slug":"/naming-sass-color-variables/"}}}]},{"name":"setup","totalCount":1,"edges":[{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"skills","totalCount":1,"edges":[{"node":{"fields":{"slug":"/weekly-ai-bites-last-manual-gap-qa-testing/"}}}]},{"name":"styles","totalCount":1,"edges":[{"node":{"fields":{"slug":"/naming-sass-color-variables/"}}}]},{"name":"symfony","totalCount":1,"edges":[{"node":{"fields":{"slug":"/minio-as-s3-replacement-in-development-and-beyond/"}}}]},{"name":"templates","totalCount":1,"edges":[{"node":{"fields":{"slug":"/pull-request-templates-on-github/"}}}]},{"name":"test-automation","totalCount":1,"edges":[{"node":{"fields":{"slug":"/weekly-ai-bites-last-manual-gap-qa-testing/"}}}]},{"name":"transaction","totalCount":1,"edges":[{"node":{"fields":{"slug":"/is-mysql-autoincrement-really-monotonic/"}}}]},{"name":"travis","totalCount":1,"edges":[{"node":{"fields":{"slug":"/unresolved-check-from-travis-on-github-pull-request/"}}}]},{"name":"tutorial","totalCount":2,"edges":[{"node":{"fields":{"slug":"/how-to-implement-redux-in-react/"}}},{"node":{"fields":{"slug":"/Dev-and-prod-ready-Docker-setup-for-SPA-app/"}}}]},{"name":"ubuntu","totalCount":1,"edges":[{"node":{"fields":{"slug":"/time-out-of-sync-in-aws-ec2/"}}}]},{"name":"unsupported","totalCount":1,"edges":[{"node":{"fields":{"slug":"/aws-cognito-without-library/"}}}]},{"name":"uuid","totalCount":1,"edges":[{"node":{"fields":{"slug":"/jpa-and-uuid/"}}}]},{"name":"variables","totalCount":1,"edges":[{"node":{"fields":{"slug":"/naming-sass-color-variables/"}}}]},{"name":"wasm","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]},{"name":"watchos","totalCount":1,"edges":[{"node":{"fields":{"slug":"/aws-cognito-without-library/"}}}]},{"name":"web","totalCount":1,"edges":[{"node":{"fields":{"slug":"/net-in-my-browser/"}}}]}]},"author":{"id":"37bda159-ff45-50c9-aeba-18326b7ca66b","childMarkdownRemark":{"html":"

XSolve We are Agile Software House focused on PHP/Symfony, #JavaScript, #Java and #Mobile (iOS, Android, Windows Phone) #Inc5000 European company in 2018.\n
\nxsolve.software

"}},"footnote":{"id":"b4474c9b-7be6-5f34-866a-cdfffa885bce","childMarkdownRemark":{"html":""}},"site":{"siteMetadata":{"facebook":{"appId":""}}}},"pageContext":{"slug":"/quick-import-of-mysql-database-dump/"}}