AWS / Azure / GCP Cloud | Database | Data Analytics | LakeHouse | Machine Learning | Shrenik Parekh

Migrate MySQL User from MySQL 5.6 to 8.0.21: If you are migrating MySQL from 5.6 to 8.0.21 new env. (Not in place upgrade) then this is one of the  option to migrate mysql users.  $ mysql -uroot -p -S/mysql/mysql.sock --skip-column-names --execute \ "SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';') FROM mysql.user" | tee step1 $ mysql -uroot -p -S/mysql/mtsprd/data/mtsprd.sock  --skip-column-names < step1 | tee step2 $ sed -e "s/ IDENTIFIED BY PASSWORD '.*'//" -e "s/$/;/" < step2 | tee step3 Note: Interesting use of function CONCAT, QUOTE, Linux utility tee, and sed makes it easy to generate script.  Generate dump file from MySQL 5.6.30: $ mysqldump -uroot -p --set-gtid-purged=OFF -S /mysql/mysql.sock --databases <db1> \<db2> --events --triggers --routines --force --ignore-table=<table1> \ --ignore-table=<table2> > /mysql/dump.sql Note:  --force, -f - Ignore all errors; ...

Python Script to connect MySQL Database on Linux : I explored two options to connect MySQL database from Python script: 01. mysql.connector - Explore more about mysql.connector / Python 02, MySQLdb module - Explore more about MySQLdb mysql.cnnector: Download mysql.connector from MySQL Community Download Install mysql.connector on Linux: Using pip: # pip install mysql-connector-python Using yum: # yum update mysql-community-release # yum install mysql-connector-python Explore mysql.connector connection examples  

Docker: Docker is an engine by Docker. Explore slides . Runs on host OS as well as guest OS, bare metal, cloud. Image: Packages software code, run-time, system tools, system libraries and settings. Images become containers when they run on Docker Engine. Container:   Standard unit of software. Isolate software from its environment Software run same on Windows and Linux. Could be portable anywhere. Abstraction at the app layer that packages code and all dependencies together. Runs application quickly from one computing env. to another env. Do not require an OS per application, as it share the machine’s OS system kernel Applications are safer in containers Multiple containers can run on the same machine Each container runs as an isolated processes in user space. Require less space than VM. Virtual Machines: VMs are an abstraction of physical hardware. Turns one server into many servers. The hypervisor allows multiple VMs to run on a single machine. Each V...

Kubernetes directory structure and process on Control Plane / Master: Kubelet environment file - /var/lib/kubelet/kubeadm-flags.env Kubelet configuration file - /var/lib/kubelet/config.yaml Certificate directory folder - /etc/kubernetes/pki Kube config directory - /etc/kubernetes admin config file - /etc/kubernetes/admin.conf kube config file - /etc/kubernetes/kubelet.conf control manager config file - /etc/kubernetes/controller-manager.conf scheduler config file - /etc/kubernetes/scheduler.conf Manifest directory - /etc/kubernetes/manifests Cluster store - /etc/kubernetes/manifests /etc/kubernetes/manifests/etcd.yaml  /etc/kubernetes/manifests/kube-apiserver.yaml  /etc/kubernetes/manifests/kube-controller-manager.yaml  /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/pki: Has following key, crt files and one directory for etcd apiserver.crt apiserver.key ca.crt  front-proxy-ca.crt front-proxy-client.key apiserver-etcd-client.crt  apiserver-kub...

Install Kubernetes Control Plane (master) and Worker Node: Pre-requisite for Kubernetes Control Plane and Worker Nodes / minion: Disable selinux: Edit /etc/selinux/config file, and set SELINUX-disabled as follows, and bounce the server. SELINUX=disabled #getenfoce Disabled Disable firewall: #systemctl stop firewalld #systemctl disable firewalld #stystemctl status firewalld Active: inactive (dead) Set bridge-nf-call-iptables contents: echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables Oterwise you will get following error [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1 Disable swap: #swapoff -a Otherwise you will get following error [ERROR Swap]: running with swap on is not supported. Please disable swap Configure Repository on control plane and worker node: Execute following command to create file /etc/repos.d/kubernetes.repo and add content for kubernetes, which will help to set na...

Create MySQL custom image: MySQL custom image can be created using Dockerfile: Create custom .sql script which could have DDL, DML, Meta Data: /mysql/docker_compose1/sql_scripts.sql cd to directory /mysql/docker_compose1/: Create Dockerfile: Include following into the file Dockerfile FROM mysql COPY ./sql_scripts.sql/ /docker-entrypoint-initdb.d/ Create own MySQL image: #docker build -t docker_compose1 .

MySQL Custom Docker Container | Docker Compose: Here we are going to explore how to customize, MySQL Docker container using custom parameters in file, using volume, and Docker compose: Create parameter file my.cnf locally, create Docker file which has mount volume (directory) configuration, bring up container using "docker-compose up -d", connect database and make sure everything is as expected. Create directory for persistent data volume: Create directory /mysql/mysql8020_data/docker_compose1/data_1 to store MySQL docker container persistent data. Create my.cnf file at /mysql/docker_compose_1/conf.d/: [mysqld] default_authentication-plugin=mysql_native_password server_id=7777 port=7399 Note: You might be wondering how local conf.d director's my.cnf file is read by docker container. Local my.cnf file is mapped to container /etc/mysql/conf.d directory which can be access by following command. File my.cnf is not mapped, directory conf.d is mapped to /etc/m...