~$ cd /mnt/d/data/Linux_projects/Final_projects/C7
Download Scripts
MySQL Backup & Restore
Backups
Full Backups
A full backup captures the entire database at a specific point in time. It includes all the data, schema, and configurations necessary to restore the database to that state. Full backups serve as the foundation for PITR and are typically performed periodically (e.g., daily or weekly) to ensure that there is a recent baseline from which to start the recovery process.
Incremental Backups
Incremental backups capture only the changes made to the database since the last backup (full or incremental). This approach reduces the amount of data that needs to be backed up, saving time and storage space. In MySQL, binary logs are crucial for incremental backups as they record all database changes.
Binary Logs
Binary logs in MySQL are essential for PITR. They record all database changes (e.g., INSERT, UPDATE, DELETE statements) in a binary format. These logs enable the replay of changes during recovery, allowing you to restore the database to any specific point in time by applying the changes recorded after the last full backup.
How Binary Logs Work
1. Enable Binary Logging: Binary logging must be enabled on the MySQL server. This can be done by adding the log_bin directive to the MySQL configuration file (my.cnf) and restarting the MySQL server.
2. Recording Changes: Once binary logging is enabled, MySQL starts recording all changes to the database in binary log files. Each log file is given a sequential number (e.g., mysql-bin.000001, mysql-bin.000002).
3. Managing Log Files: As the number of binary log files increases, it’s important to manage them to prevent disk space issues. MySQL provides options to purge old binary logs that are no longer needed.
4. Point-in-Time Recovery: To perform a point-in-time recovery, you first restore the database from the most recent full backup. Then, you apply the changes recorded in the binary logs to bring the database to the desired state.
Point-in-Time B&R
PITR relies on both full backups and incremental backups to provide a comprehensive recovery solution. These two types of backups work together to ensure that you can restore your database to any specific point in time.
Significance of (PITR)
Point-in-Time Recovery (PITR) allows you to restore a database to a specific moment, often before an error or data corruption occurred. This capability is critical for minimizing data loss and maintaining data integrity, especially in environments where continuous availability and data accuracy are paramount.
- PITR involves a combination of full backups and incremental backups (or binary logs in MySQL), enabling administrators to revert the database to any desired point in time.
- This method provides flexibility and precision in recovery operations, ensuring that only the necessary data changes are applied during the restoration process.
PITR - Example 1
Let us explore a scenario where you are tasked with taking a point in time backup of the MySQL server that houses a database named world. Within this database, you will encounter tables like cities, countries and languages, each holding data related to cities, countries and languages spoken worldwide.This exercise will build upon the world database created in the previous exercise: Backup and Restore Using MySQL.
- Say you have a full logical backup of your whole database in your last mysqldump file as of yesterday evening. However, several changes may have been made (including data loss) since then.
- Using point-in-time backup and restore, you can get each and every change that occurred since then, so that even after your last logical backup you have a record of all new transactions.
- Point-in-time backup is the set of binary log files generated subsequent to a logical backup operation of a database. The binary log files contain events that describe database changes such as table creation operations or changes to table data.
- To restore a database to a point-in-time, you will be using binary log files containing changes of a database for a time interval along with the last logical backup of the database.
To install MySQL go to Data/MySQL page
Data
The World database used in this lab comes from the following source: https://dev.mysql.com/doc/world-setup/en/
Linux in WSL
Since I’ll be using Linux in WSL I’ll have an extra word or two in addition to the usual linux commands to access my local directory
Download the required SQL script file in the linux terminal by executing the command below to save all the scripts to our local machine:
- I created a directory for the files first
- Downloaded 3 back to back
//cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0231EN-SkillsNetwork/datasets/World/world_mysql_script.sql -P C7
$ wget https:
//cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0231EN-SkillsNetwork/datasets/World/world_mysql_update_A.sql -P C7
$ wget https:
//cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0231EN-SkillsNetwork/datasets/World/world_mysql_update_B.sql -P C7
$ wget https:
# OUTPUT
--2012-09-30 17:18:20-- https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0231EN-SkillsNetwork/datasets/World/world_mysql_update_B.sql
-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 198.23.119.245
Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|198.23.119.245|:443... connected.
Connecting to cf200 OK
HTTP request sent, awaiting response... 3497 (3.4K) [application/x-sql]
Length: /world_mysql_update_B.sql’ Saving to: ‘C7
Verify Download
~$ ls
world_mysql_script.sql world_mysql_update_A.sql world_mysql_update_B.sql
Start MySQL
$ sudo systemctl start mysql
Login to MySQL
/mnt/d/data/Linux_projects/Final_projects/C7$ sudo mysql -u root -p
$:
Enter password:
# RESPONSE
with ; or \g.
Welcome to the MySQL monitor. Commands end id is 8
Your MySQL connection 8.0.39-0ubuntu0.22.04.1 (Ubuntu)
Server version:
2000, 2024, Oracle and/or its affiliates.
Copyright (c)
is a registered trademark of Oracle Corporation and/or its
Oracle
affiliates. Other names may be trademarks of their respective
owners.'help;' or '\h' for help. Type '\c' to clear the current input statement.
Type
> mysql
Create db
Use db
Execute first Script file
- To create the first db from the backup script file
world_mysql_script.sql
- run the last line of code
- It will take a few seconds to complete the creation of the db
>create database world;
mysql>use world;
mysql>source world_mysql_script.sql; mysql
Show Dbs
> SHOW DATABASES;
mysql+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| world |
+--------------------+
5 rows in set (0.13 sec)
Show Tables
> SHOW TABLES;
mysql+-----------------+
| Tables_in_world |
+-----------------+
| city |
| country |
| countrylanguage |
+-----------------+
3 rows in set (0.01 sec)
View all records for CAN
> SELECT * FROM city WHERE countrycode='CAN';
mysqlset (0.00 sec) Empty
Restore Again
- As you see it appears that the first restore using the first script didn’t cover the data that was added later
- If you remember this could be the intermittent backup which is information added since the complete backup
- So let’s ALTER the db by executing the second script file
> source world_mysql_update_A.sql;
mysql0 rows affected (0.00 sec)
Query OK,
1 row affected (0.00 sec)
Query OK,
1 row affected (0.01 sec)....
Query OK,
# View all records again
> SELECT * FROM countrylanguage WHERE countrycode='CAN';
mysql# we see that the languages were added (I guess I should've run the same code before as well, but take my word for it this data was not there)
Close Connection
> \q
mysql
Bye~:/mnt/d/data/Linux_projects/Final_projects/C7$
Dump Full Backup
Stop MySQL
Before you backup you must STOP MySQL
sudo systemctl stop mysql
Full Logical Backup
Next, create a full logical backup of the current state of your whole world database.
- Use the command below in the terminal
- Redirect the output to > filename
--user=root --password --flush-logs --delete-master-logs --databases world > world_mysql_full_backup.sql
sudo mysqldump for yashaya:cccc
[sudo] password
# File is created and saved
# For in the cloud use I had to run this code to backup the countrylanguage TABLE ONLY of the world db
--host=mysql --port=3306 --user=root --password world countrylanguage > world_countrylanguage_mysql_backup.sql mysqldump
View Content of Backup file
If you want to inspect the content of the file for informational purposes use
cat world_countrylanguage_mysql_backup.sql
Login to MySQL
~$ sudo mysql -u root -p
Enter password:
# RESPONSE
with ; or \g.
Welcome to the MySQL monitor. Commands end id is 10
Your MySQL connection 8.0.39-0ubuntu0.22.04.1 (Ubuntu)
Server version:
2000, 2024, Oracle and/or its affiliates.
Copyright (c)
is a registered trademark of Oracle Corporation and/or its
Oracle
affiliates. Other names may be trademarks of their respective
owners.
'help;' or '\h' for help. Type '\c' to clear the current input statement.
Type
> mysql
Repeat Steps above
> SHOW TABLES;
mysql+-----------------+
| Tables_in_world |
+-----------------+
| city |
| country |
| countrylanguage |
+-----------------+
3 rows in set (0.00 sec)
# Review data for CAN
> SELECT * FROM city WHERE countrycode='CAN';
mysqlset (0.00 sec) # An empty set again Empty
Intermittent Backup
Restore from Intermittent
- Let’s pretend the last script: ~update_B.sql is an intermittent backup since the dump was created
- Run the script for it
> source world_mysql_update_B.sql;
mysql
# Review data for CAN
> SELECT * FROM city WHERE countrycode='CAN';
mysql+------+------------------+-------------+------------------+------------+
| ID | Name | CountryCode | District | Population |
+------+------------------+-------------+------------------+------------+
| 1810 | Montréal | CAN | Québec | 1016376 |
| 1811 | Calgary | CAN | Alberta | 768082 |
| 1812 | Toronto | CAN | Ontario | 688275 |
| 1813 | North York | CAN | Ontario | 622632 |
| 1814 | Winnipeg | CAN | Manitoba | 618477 |
| 1815 | Edmonton | CAN | Alberta | 616306 |
| 1816 | Mississauga | CAN | Ontario | 608072 |
| 1817 | Scarborough | CAN | Ontario | 594501 |
| 1818 | Vancouver | CAN | British Colombia | 514008 |
| 1819 | Etobicoke | CAN | Ontario | 348845 |
| 1820 | London | CAN | Ontario | 339917 |
| 1821 | Hamilton | CAN | Ontario | 335614 |
| 1822 | Ottawa | CAN | Ontario | 335277 |
| 1823 | Laval | CAN | Québec | 330393 |
| 1824 | Surrey | CAN | British Colombia | 304477 |
| 1825 | Brampton | CAN | Ontario | 296711 |
| 1826 | Windsor | CAN | Ontario | 207588 |
| 1827 | Saskatoon | CAN | Saskatchewan | 193647 |
| 1828 | Kitchener | CAN | Ontario | 189959 |
| 1829 | Markham | CAN | Ontario | 189098 |
| 1830 | Regina | CAN | Saskatchewan | 180400 |
| 1831 | Burnaby | CAN | British Colombia | 179209 |
| 1832 | Québec | CAN | Québec | 167264 |
| 1833 | York | CAN | Ontario | 154980 |
| 1834 | Richmond | CAN | British Colombia | 148867 |
| 1835 | Vaughan | CAN | Ontario | 147889 |
| 1836 | Burlington | CAN | Ontario | 145150 |
| 1837 | Oshawa | CAN | Ontario | 140173 |
| 1838 | Oakville | CAN | Ontario | 139192 |
| 1839 | Saint Catharines | CAN | Ontario | 136216 |
| 1840 | Longueuil | CAN | Québec | 127977 |
| 1841 | Richmond Hill | CAN | Ontario | 116428 |
| 1842 | Thunder Bay | CAN | Ontario | 115913 |
| 1843 | Nepean | CAN | Ontario | 115100 |
| 1844 | Cape Breton | CAN | Nova Scotia | 114733 |
| 1845 | East York | CAN | Ontario | 114034 |
| 1846 | Halifax | CAN | Nova Scotia | 113910 |
| 1847 | Cambridge | CAN | Ontario | 109186 |
| 1848 | Gloucester | CAN | Ontario | 107314 |
| 1849 | Abbotsford | CAN | British Colombia | 105403 |
| 1850 | Guelph | CAN | Ontario | 103593 |
| 1851 | Saint John´s | CAN | Newfoundland | 101936 |
| 1852 | Coquitlam | CAN | British Colombia | 101820 |
| 1853 | Saanich | CAN | British Colombia | 101388 |
| 1854 | Gatineau | CAN | Québec | 100702 |
| 1855 | Delta | CAN | British Colombia | 95411 |
| 1856 | Sudbury | CAN | Ontario | 92686 |
| 1857 | Kelowna | CAN | British Colombia | 89442 |
| 1858 | Barrie | CAN | Ontario | 89269 |
+------+------------------+-------------+------------------+------------+
49 rows in set (0.00 sec)
Quit MySQL
> \q
mysql Bye
Simulate Crash
We’ll create a scenario where a database crash will be conducted intentionally which will result a significant loss of your world database files.
- To simulate a database crash and create a scenario for recover stop the MySQL service
- Remove the database files
Stop MySQL
sudo systemctl stop mysql
Here’s what each part of the command does:
systemctl: This is a command-line utility used to manage systemd, which is a system and service manager for Linux operating systems. It allows users to start, stop, enable, disable, and manage services and other units.
stop: This subcommand of systemctl instructs systemd to stop the specified service. In this case, mysql refers to the MySQL service
Remove Db
- Remove the db so we can show how the recovery part of B & R
- MySQL by default stores the dbs in: /var/lib/mysql/
- The
rm -rf
command removes the world directory and all of its contents recursively and forcefully. This includes all the data files, subdirectories, and any other files related to the world database within the MySQL data directory. - After removing the dv, try to retrieve records from any table (let’s try the city table) from the database
- Should get an error because the db no longer exists
-rf /var/lib/mysql/world
sudo rm
# Verify deletion
--user=root --password --execute="SELECT * FROM world.city;"
$ sudo mysql
Enter password:
# RESPONSE
2013 (HY000) at line 1: Lost connection to MySQL server during query ERROR
Restore DB
Let’s show how to restore the database after we had made a backup earlier
- So we have a backup file created in the Dump section above
- We also have an intermittent backup the last script: ~update_B.sql which we are going to pretend was done after the full logical backup
Display Binary Logs
--user=root --password --execute="SHOW BINARY LOGS;"
$ sudo mysql
Enter password:
+---------------+-----------+-----------+
| Log_name | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000004 | 15805 | No |
| binlog.000005 | 157 | No |
| binlog.000006 | 157 | No |
+---------------+-----------+-----------+
Combine Binary Logs
- Combine all binary log files together into a single file
- Here’s what each part of the command does:
Read Binary Log Files: The
mysqlbinlog
utility reads the specified binary log files (binlog.000003 and binlog.000004).Convert to SQL Statements:
mysqlbinlog
converts the binary format of the log files into readable SQL statements. These statements represent the database changes recorded in the logs.Redirect Output: The SQL statements are redirected and saved into a file named
logfile.sql
.
/var/lib/mysql/binlog.000004 /var/lib/mysql/binlog.000005 /var/lib/mysql/binlog.000006 > logfile.sql $ sudo mysqlbinlog
PITR
Now we are ready to perform Point-In-Time-Recovery.
Restore Full Backup
- First restore the full logical backup of your whole world database you created earlier in this example using the command below in the terminal
- Enter your MySQL service session password from the MySQL service session tab if necessary
- Note the < arrow where we are using the ~full_backup.sql file as the source
/C7$ sudo mysql --user=root --password < world_mysql_full_backup.sql
Enter password:
Verify Data
To verify if you have the updates from the update script world_mysql_update_B.sql, retrieve all the Canada (countrycode=’CAN’) related records from the city table
--user=root --password --execute="SELECT * FROM world.city WHERE countrycode='CAN';"
sudo mysql
# OUTPUT
set empty
Run Logfile
We obviously took some steps after the full backup and were all tracked in the log files that were combined into one. So let’s apply the steps in the logfile to the db to fully restore it to the exact point in time
--user=root --password < logfile.sql sudo mysql
Verify Data
Now that the full backup as well as the logfile has been executed, let’s verify the output and compare it to the output above, prior to the simulated crash test.
--user=root --password --execute="SELECT * FROM world.city WHERE countrycode='CAN';"
C7$ sudo mysql
Enter password:
+------+------------------+-------------+------------------+------------+
| ID | Name | CountryCode | District | Population |
+------+------------------+-------------+------------------+------------+
| 1810 | Montréal | CAN | Québec | 1016376 |
| 1811 | Calgary | CAN | Alberta | 768082 |
| 1812 | Toronto | CAN | Ontario | 688275 |
| 1813 | North York | CAN | Ontario | 622632 |
| 1814 | Winnipeg | CAN | Manitoba | 618477 |
| 1815 | Edmonton | CAN | Alberta | 616306 |
| 1816 | Mississauga | CAN | Ontario | 608072 |
| 1817 | Scarborough | CAN | Ontario | 594501 |
| 1818 | Vancouver | CAN | British Colombia | 514008 |
| 1819 | Etobicoke | CAN | Ontario | 348845 |
| 1820 | London | CAN | Ontario | 339917 |
| 1821 | Hamilton | CAN | Ontario | 335614 |
| 1822 | Ottawa | CAN | Ontario | 335277 |
| 1823 | Laval | CAN | Québec | 330393 |
| 1824 | Surrey | CAN | British Colombia | 304477 |
| 1825 | Brampton | CAN | Ontario | 296711 |
| 1826 | Windsor | CAN | Ontario | 207588 |
| 1827 | Saskatoon | CAN | Saskatchewan | 193647 |
| 1828 | Kitchener | CAN | Ontario | 189959 |
| 1829 | Markham | CAN | Ontario | 189098 |
| 1830 | Regina | CAN | Saskatchewan | 180400 |
| 1831 | Burnaby | CAN | British Colombia | 179209 |
| 1832 | Québec | CAN | Québec | 167264 |
| 1833 | York | CAN | Ontario | 154980 |
| 1834 | Richmond | CAN | British Colombia | 148867 |
| 1835 | Vaughan | CAN | Ontario | 147889 |
| 1836 | Burlington | CAN | Ontario | 145150 |
| 1837 | Oshawa | CAN | Ontario | 140173 |
| 1838 | Oakville | CAN | Ontario | 139192 |
| 1839 | Saint Catharines | CAN | Ontario | 136216 |
| 1840 | Longueuil | CAN | Québec | 127977 |
| 1841 | Richmond Hill | CAN | Ontario | 116428 |
| 1842 | Thunder Bay | CAN | Ontario | 115913 |
| 1843 | Nepean | CAN | Ontario | 115100 |
| 1844 | Cape Breton | CAN | Nova Scotia | 114733 |
| 1845 | East York | CAN | Ontario | 114034 |
| 1846 | Halifax | CAN | Nova Scotia | 113910 |
| 1847 | Cambridge | CAN | Ontario | 109186 |
| 1848 | Gloucester | CAN | Ontario | 107314 |
| 1849 | Abbotsford | CAN | British Colombia | 105403 |
| 1850 | Guelph | CAN | Ontario | 103593 |
| 1851 | Saint John´s | CAN | Newfoundland | 101936 |
| 1852 | Coquitlam | CAN | British Colombia | 101820 |
| 1853 | Saanich | CAN | British Colombia | 101388 |
| 1854 | Gatineau | CAN | Québec | 100702 |
| 1855 | Delta | CAN | British Colombia | 95411 |
| 1856 | Sudbury | CAN | Ontario | 92686 |
| 1857 | Kelowna | CAN | British Colombia | 89442 |
| 1858 | Barrie | CAN | Ontario | 89269 |
+------+------------------+-------------+------------------+------------+
Backup Example 2
Data
Db script is located online and can be gotten via. I already have that locally from above.
//cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0231EN-SkillsNetwork/datasets/World/world_mysql_script.sql wget https:
Start MySQL
Reminder: Here’s what each part of the command does:
- sudo: This prefix grants temporary administrative privileges to the command that follows it. It allows the user to execute the subsequent command with superuser (root) privileges.
- systemctl: This is a command-line utility used to manage systemd, which is a system and service manager for Linux operating systems. It allows users to start, stop, enable, disable, and manage services and other units.
- start: This subcommand of systemctl instructs systemd to start the specified service. In this case, mysql refers to the MySQL service.
Login to MySQL
Here’s what each part of the command does:
- mysql: This is the command used to launch the MySQL command-line interface.
- -u root: This option specifies the username to use when connecting to the MySQL server. In this case, it specifies that the root user should be used.
- -p: This option prompts the user to enter the password for the specified MySQL user (in this case, the root user). After entering the password, if it’s correct, the user gains access to the MySQL CLI with administrative privileges.
$ sudo systemctl start mysqlfor yashaya:
[sudo] password
# Login to MySQL
-u root -p
$ sudo mysql
Enter password:
# RESPONSE
with ; or \g.
Welcome to the MySQL monitor. Commands end id is 13
Your MySQL connection 8.0.39-0ubuntu0.22.04.1 (Ubuntu)
Server version: 2000, 2024, Oracle and/or its affiliates.
Copyright (c) is a registered trademark of Oracle Corporation and/or its
Oracle
affiliates. Other names may be trademarks of their respective
owners.'help;' or '\h' for help. Type '\c' to clear the current input statement.
Type
> mysql
Create db
Use db
Execute Script
Execute the script to complete the creation of the db. The script holds the data for the tables: cities, countries, languages
;
create database world;
use world; source world_mysql_script.sql
List Dbs
> SHOW DATABASES;
mysql+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| world |
+--------------------+
5 rows in set (0.13 sec)
Stop MySQL
Before executing a backup you MUST stop MySQL. Here we’ll first quit it, then stop it
# ---------- QUIT
> \q
mysql
Bye
# --------- STOP
~$ sudo systemctl stop mysql
Backup
Complete backup of the db, which will recursively copy all the directories and subs to the specified > directory
Here’s what each part of the command does:
- cp: This is the command used to copy files and directories.
- -R: This option is used to specify a recursive copy, meaning it will copy the specified directory and all of its contents, including subdirectories and files.
- /var/lib/mysql: MySQL data files, including databases, tables, and other related files are present in this directory.
- Note this will copy all the data in the directory not just one database
- /path/to/backup_directory: This is the destination directory where the MySQL data directory will be copied. You should replace “/path/to/backup_directory” with the directory which you want the backup to be stored.
-R /var/lib/mysql /path/to/backup_directory sudo cp
-R /var/lib/mysql /mnt/d/data/Linux_projects/Final_projects/C7 sudo cp
Verify
-l /mnt/d/data/Linux_projects/Final_projects/C7
$ sudo ls 5120
total -rwxrwxrwx 1 yashaya yashaya 74134 Oct 1 09:10 logfile.sql
1 yashaya yashaya 512 Oct 1 10:33 mysql
drwxrwxrwx -rwxrwxrwx 1 yashaya yashaya 241056 Sep 30 18:01 world_mysql_full_backup.sql
-rwxrwxrwx 1 yashaya yashaya 389702 Sep 6 2022 world_mysql_script.sql
-rwxrwxrwx 1 yashaya yashaya 1035 Sep 6 2022 world_mysql_update_A.sql
-rwxrwxrwx 1 yashaya yashaya 3497 Sep 6 2022 world_mysql_update_B.sql
Rename Old Directory
Now that we have a copy of the entire /mysql directory we can rename the previous one on the default path: to mysql_old
/var/lib/mysql /var/lib/mysql_old sudo mv
Copy Backup to Default
Now we can recursively copy the entire mysql directory that we backed up to /mnt/d/data/Linux_projects/Final_projects/C7 back to the default location to replace the corrupt one.
Ensure there is sufficient storage space before performing the copying
-R /mnt/d/data/Linux_projects/Final_projects/C7/mysql /var/lib/mysql sudo cp
Set Ownership of Directory
-R mysql:mysql /var/lib/mysql sudo chown
Here’s what each part of the command does:
- chown: Stands for change ownership, a command used to change the ownership of files or directories in Linux.
- -R: Specifies a recursive operation,including subdirectories and files.
- mysql:mysql: Specifies the new owner and group for the directory and its contents. In this case, it sets the user to “mysql” and the group to “mysql”.
- /var/lib/mysql: Specifies the directory whose ownership will be changed. This is typically the MySQL data directory.
Once completed you can restart the MySQL service using the command: sudo systemctl start mysql