how to shrink disk on qcow2 image (with an xfs filesystem)

abstract

This explains how to shrink (or change the size of the disk) of a qcow2 image.

qcow2 images are used in VMware virtual machines.
I am converting everything to vdi to test with virtualbox, so I can test directly in my computer.

Basically, you can't directly shrink a qcow2 image (using xfs). The actual process is to make another image (with a different disk size), copying all the existing contents from the original one.

For simplicity, this image is a GNU/Linux system (centos), with a grub2 bootloader.

I'm doing all these commands on a Debian 11 system.

steps overall

1. convert qcow2 to vdi

2. create new disk for new disk size, and another to locally save the backup (temp)

3. attach the new and the old image disks (vdi) to the VM

4. create new filesystems, mount and backup/restore the old image files

5. copy mbr to new system

6. update `/boot/grub2` and `/etc/fstab` for the new disk's uuid, and remove grub2 admin password (in case your original image had the grub2 admin password set)

7. convert vdi to qcow2

pre-requirements

1. the qcow2 image (can download from openstack)

2. virtualbox with a working GNU/Linux OS (to run from a live CD ISO image is OK too)

steps with commands in detail

# 1. convert qcow2 to vdi
# (get the image to your local machine first)
qemu-img convert -O vdi some_image.qcow2 some_image_backup.vdi

# 2. create new disk for new disk size, and another to locally save the backup (temp)
# use virtualbox's GUI for this
# create new_disk.vdi. And temp_disk.vdi if you are running a live CD. Using your local disk is OK too.

# 3. attach the new and the old image image disks (vdi) to the VM
# use virtualbox's GUI for this. It's done in the "storage" tab

# 4. create new filesystems, mount and backup/restore the old image files
cd /mnt
mkdir old_random_image
mkdir new_random_image
mkdir temp_disk

fdisk -l /dev/sda
n
ENTER bunch of times
w

# same for /dev/sdb
mkfs.ext4 /dev/sda1
mkfs.xfs /dev/sdb1

mount /dev/sda1 temp_disk
mount /dev/sdb1 new_random_image
mount /dev/sdc1 old_random_image
cd temp_disk

apt-get update
apt-get install xfsdump parted
xfsdump -f random_image.dump /mnt/old_random_image
# use random names for job and disk labels like: foo, foo2, bar, etc

xfsrestore -f random_image.dump /mnt/new_random_image

# use df -h and ls to confirm the filesystems seem to match

# add boot permission
# check old image
parted /dev/sdc
p
Ctrl-D

# add boot flag to partition
parted /dev/sdb
p
toggle 1
boot
p
Ctrl-D

# 5. copy mbr to new system

# backup old mbr 
dd if=/dev/sdc of=orig_mbr bs=512 count=1

# copy MBR to new disk (booting part only, without the partition table definition)
dd if=orig_mbr of=/dev/sdb bs=446 count=1

# 6. update `/boot/grub2` and `/etc/fstab` for the new disk's uuid, and remove grub2 admin password
cp -r /boot/grub2  /boot/grub2_bk
cd /boot/grub2
rm user.cfg grub.cfg.*.rpmsave

# check the new disk's uuid
ls -lh /dev/disk/by-uuid/

# replace the old uuid with the new one, and remove password requiring config
vi grub.cfg

################################################################################
# (here is a diff for above)

78,87d80
< ### BEGIN /etc/grub.d/01_users ###
< if [ -f ${prefix}/user.cfg ]; then
<   source ${prefix}/user.cfg
<   if [ -n "${GRUB2_PASSWORD}" ]; then
<     set superusers="root"
<     export superusers
<     password_pbkdf2 root ${GRUB2_PASSWORD}
<   fi
< fi
< ### END /etc/grub.d/01_users ###
90c83
< menuentry 'CentOS Linux' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-XXXX-OLD-DISK-UID' {
---
> menuentry 'CentOS Linux' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-XXXX-NEW-DISK-UID' {
98c91
< 	  search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  OLD-DISK-UID
---
> 	  search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1' NEW-DISK-UID 
100c93
< 	  search --no-floppy --fs-uuid --set=root OLD-DISK-UID
---
> 	  search --no-floppy --fs-uuid --set=root NEW-DISK-UID
102c95
< 	linux16 /boot/vmlinuz-XXX.x86_64 root=UUID=OLD-DISK-UID ro console=tty0 crashkernel=auto console=ttyS0,115200 audit=1 LANG=en_US.UTF-8
---
> 	linux16 /boot/vmlinuz-XXX.x86_64 root=UUID=NEW-DISK-UID ro console=tty0 crashkernel=auto console=ttyS0,115200 audit=1 LANG=en_US.UTF-8
105c98
# ...
################################################################################

# do the same for fstab
vi /etc/fstab
# should look like this:
# UUID=XXXXXXXXXXXXXXXXXXX  /                       xfs     defaults        0 0
# (this has the new disk's UUID)

# restart VM unmounting all disks but the new image
# and check it boots OK (probably can't get a prompt due to network settings, but you should
# get no kernel error, or booting errors like "no disk/booter found" etc)

# 7. convert vdi to qcow2
qemu-img convert -f vdi -O qcow2 ~/virtualbox_vms/disks/random_image10G.vdi some_image_10G.qcow2

# 8. upload image to openstack
openstack image create --disk-format qcow2 --container-format bare  --file image/some_image_10G.qcow2 img-name-in-openstack

booting cisco iosxrv in virtualbox

abstract

Cisco has some virtual images of their iosxr system (iosxrv), which aren't very easy to use.

This is how I got iosxrv to work on virtualbox.

The trickiest parts were that the specs fixed/imported by image are pretty large, so you have to scale them down to make it work. And that virtualbox apparently can't find the interface to do ssh forwarding so you can actually connect to the device (is my GUI client buggy?... idk, but found it in the xml file you're not supposed to edit manually, but that works if you do anyway).

overall steps

1. import ova image

2. change image specs/configs

3. boot image 

4. (on X) create default user

5. (on X) setup ssh server and interface

6. confirm you can ssh with a localhost high port

Prerequisites

1. virtualbox

2. image files (ova files)

3. an X working system (connect from a GNU/Linux system on your client end, or install a windows client)

I did this on my debian server, on both a physical machine, and a VM, worked just fine.

steps

# ssh to the VM with the -X option, so you can run X programs from the server
ssh -X YOUR_SERVERS_IP_ADDR

# sample for 6.5.2. Please replace VM/image name as needed
# import image

VBoxManage import xrv9k-fullk9-x.vrr.vga-6.5.2.ova

# change name
VBoxManage modifyvm com.cisco.ios-xrv9000 --name iosxrv6.5.2

# change default options that make VM unbootable
VBoxManage modifyvm iosxrv6.5.2 --vtxux on
VBoxManage modifyvm iosxrv6.5.2 --ostype Linux_64
VBoxManage modifyvm iosxrv6.5.2 --cpus 6
VBoxManage modifyvm iosxrv6.5.2 --memory 7096
VBoxManage modifyvm iosxrv6.5.2 --vram 10
VBoxManage modifyvm iosxrv6.5.2 --longmode on
VBoxManage modifyvm iosxrv6.5.2 --graphicscontroller vmsvga

# add ssh port forwarding settings from virtualbox
VBoxManage modifyvm iosxrv6.5.2 --nic6 nat

# Change local port ("5801" here) and connection name (ssh_652) per VM.
# The rest stays the same
VBoxManage modifyvm iosxrv6.5.2 --natpf6 ssh_652,tcp,127.0.0.1,5801,10.0.7.15,22

# check mac address
grep "Adapter slot=\"5\"" ~/ ~/VirtualBox\ VMs/iosxr\ 6.5.2/*box # replace dir as necessary

#    <Adapter slot="5" enabled="true" MACAddress="XXXX" cable="true" type="XXXX">
# we'll use this to confirm on the VM side later that the right interface
# is being addressed

# boot once with X on
# it will be super slow (like 10m?), but it should boot. And ask you for your
# new root username/password

VBoxManage startvm iosxrv6.5.2

##################
# on the X window
##################

# create the new user following the prompt's instructions

# name the device, and setup the ssh server
conf t
hostname iosxr652
commit
exit

crypto key generate rsa
conf t
ssh server v2
line default transport input ssh
commit
exit

# setup interface for ssh forwarding
show int MgmtEth0/RP0/CPU0/0

# confirm mac address is the right one for our port forward
# you should get somewhere around the top a string representing the mac address
# check it matches the string got in the above step "check mac address"
# if not, to check which MAC address matches the mgmt interface, and do the NAT
# ssh port settings to that one instead

conf t
interface MgmtEth0/RP0/CPU0/0
 ipv4 address dhcp
 no shutdown
commit
exit
exit

# confirm interface is up and has an IP address assigned
show int MgmtEth0/RP0/CPU0/0

# we're assuming the IP address is 10.0.7.15
# if not, change NAT ssh forwarding settings accordingly

########################
# outside the X window
# (on your regular terminal)
########################

# (on the host running virtualbox)
ssh random_admin@localhost -p 5801 # replace username as necessary

# you should be able to login
# click the close button on the X window with the VM, and choose "shutdown VM"

# start VM again, with no X window
VBoxManage startvm --type headless iosxrv6.5.2

# after 10m or so, confirm you can ssh again
ssh random_admin@localhost -p 5801

# that's it!

# to stop:
# VBoxManage controlvm iosxrv6.5.2 poweroff 

moving lots of files with BOX

I had to move like 300K files in BOX, so just sharing a bit of how I did it and how to check the progress.

Basically, get:

1. BOX drive (no gnu/linux version available, so mounting the "folder" on a gnu/linux VM as a shared dir, and working from there)

2. A gnu/linux VM (I use virtualbox) with the necessary kernel stuff to mount shared dirs (thing to mount vboxfs)

I guess you could have done something similar in windows or mac too just writing a program directly in those platforms. But I hate both OSs, so, no.

You just write the program to rename/move the files however you want, and run it as with any other dir.

The catchy part is that box is super slow (for me at least), so even when in your disk all files would have been updated fairly fast, in the background, the thing takes forever to finish the sync... so if you check in the web version, many of the files would still be in the old location/will have their old filenames.

There should be a file called Box-{version?}.log in:

c:/Users/$USER/AppData/Local/Box/Box/logs/

There you should be able to see logs in the format:

DATE TIME ID? INFO    LocalExecutor-2      box_fs_sync_api       Move item on box.  XXXX

Telling you what it's doing.
I also mounted this in my VM and checked the logs with tail -f.

As of the time of this post, there is apparently no way to tell the sync progress besides this... And you basically will have to grep the log and compare with the list of files done with whatever you have in order to have an idea of the total percentage.

Not the coolest system.

Until the sync is complete, your box drive icon will show "Box Drive is updating your files" on mouse over. And it takes really super long... But maybe it's because of the many requests I sent, so my account was limited or something. It's like 1~3 requests a second based on the logs above.

If you're luckier than me, it should take under 3 days. If not, I hope you have a quiet computer.

installing gnu/linux in DELL G5 5090

So, first time ever not buying a used computer or server. Wanting to enjoy 4K and everything, I bought Dell's G5 5090, obviously to use with GNU/Linux.

First impressions

1. the blue thingy is super annoying. Why is it so damn bright... I guess it must be a gaming/visual thing that makes no sense for engineers. (I was able to disable the LED from the bios later)

2. besides that, it looks awesome, lots of USB slots, a nice video card, HDMI interface, and lots of PCI slots to add more stuff if I need to.

After opening it

1. apparently it only has 2 HDDs slots... more would be nicer. I'll just use my external stands I guess.

2. holy sh*t the NVME disk is tiny

Installing GNU/linux

1. The first issue is that by default, it won't let you access the BIOS... no matter how many times you press the F2 button. It immediately boots windows, and gets you that incredibly ugly and hard to use "compulsory" setup screen.
   For god's sake, windows is annoying.
   So, we have reset the CMOS so we can enter the BIOS setup screen, and choose another disk to boot (and disable the UEFI "secure" boot thing). Let's do that...
2. Resetting the CMOS is pretty easy actually. You just open the cover, and move the jumper from the PASSWD slot to the CMOS_CLEAR or something slot. Here is a reference document: https://www.dell.com/support/article/ja-jp/sln284985/how-to-perform-a-bios-or-cmos-reset-and-clear-the-nvram-on-dell-systems?lang=en#Shortcut_2 
3. now we can see the bios screen, yay. You will need to:
1. disable UEFI secure boot (so you access your GNU/Linux install disk prob)
2. change the SATA controller from intel RAID to AHCI (so you can actually use your NVME disk from linux)
4. that's basically it. Now you can just insert your USB stick with the GNU/Linux install (I recommend Debian 10, whatever you choose, just no f*cking ubuntu ok?), and it should boot on it by default. If that doesn't work for you, access the BIOS again, go to the boot order screen, and make sure your linux disk is at the top.
   Partition notes: If you want to boot from your NVME disk, remember to create an EFI on the partition guide. Otherwise you'll have to boot from an external disk just for the boot loader.
   
   

Warning

One more really annoying thing, is that apparently windows rewrites the bios settings once you boot it. So if you by mistake boot in windows, the next time you start your computer you won't be able to access the BIOS again. So you'll have to do the CMOS reset once again... really really really get rid of that piece of sh*t OS.

Message for Dell

Please sell it to me cheaper, and without windows. That's a real win-win situation.

I also didn't need the mouse and keyboard... which is not even in a keymap I can/like to use.

workaround for MySQL server has gone away error with flask

I had this issue when my flask simple app kept failing the next day it was started.
It kept giving the error "MySQL server has gone away".

I tried a couple of things like, starting the mysql session for every request instead of for each flask process (app-wide). But apparently flask doesn't like that.
At least not when using flaskext.mysql's MySQL.

I think I didn't have much luck trying to change the timeout on flask either, as this module just didn't let me.

I could have changed the module to use mysql, but I didn't have much time to finish the damn thing, so in the end I just set a cron job to access the mysql select query using page every 10 min or so. Ugly, but simple workaround.

It was like
curl -k https://localhost/some_app/index.html

every 10m.

flask app returns select with old data

I had this issue where my flask app kept giving me different results (doing a select query).

And it only happened on production, not in the development one.
Apparently due to the fact that flask has a couple of fork processes running, each with their own mysql session.
Even when you commit after an insert/update query, you still need for some reason to do a commit before a select in order to see the latest results.

The following is a sample code with some basic functions in order to do select and update queries.
I just call validate_sql_conn()  for pretty much everything I need to do in mysql, so it's always committing itself.

from flask import Flask, render_template, url_for, request, send_from_directory
from flaskext.mysql import MySQL
import yaml

app = Flask(__name__, template_folder='views')

def get_mysql_conf(mysql_conf_file="some_app_name_conf/mysql_conf.yml"):
    "gets the conf to use for the mysql connection"
    with open(mysql_conf_file, 'r') as f:
        content = f.read()
    yaml_content = yaml.load(content, Loader=yaml.FullLoader)
    return yaml_content

mysql = MySQL()
mysql_conf = get_mysql_conf()
app.config['MYSQL_DATABASE_USER'] = mysql_conf['mysql_database_user']
app.config['MYSQL_DATABASE_PASSWORD'] = mysql_conf['mysql_database_password']
app.config['MYSQL_DATABASE_DB'] = mysql_conf['mysql_database_db']
app.config['MYSQL_DATABASE_HOST'] = mysql_conf['mysql_database_host']
mysql.init_app(app)

conn = mysql.connect()
cursor = conn.cursor()


def validate_sql_conn():
    """
    uses global connection variable (conn) and recreates 
    it if seems to have dropped
    """
    global conn
    global cursor
    try:
        cursor.execute("show tables")
    except:
        conn = mysql.connect()
        cursor = conn.cursor()
    conn.commit()
    return True
        

def run_sql_and_get_results(sql_query, cursor=cursor):
    validate_sql_conn()
    cursor.execute(sql_query)
    data = cursor.fetchall()
    return data


def run_sql_and_commit(sql_query, cursor=cursor, conn=conn):
    validate_sql_conn()
    cursor.execute(sql_query)
    data = cursor.fetchall()
    commit_data = conn.commit()
    return commit_data

gnus filtering html body mail

If you love emacs, you probably use gnus, which is in my opinion the
best mailer ever.
You can manage your complete filter in emacs lisp, and there are so many
ways to filter stuff.

The most complex and useful filtering method is using
"nnmail-split-fancy".

Which as I'm showing in this example, can be set as something like this:

(setq nnmail-split-fancy
      '(|
 ("subject" ".*/var/log/messages.*" "server_syslogs")
 (any "some_sender" "some_annoying_sender")
 ;; to me, not mailing lists
 (any ".*MY_NAME.*"
      (|
       (from ".*some_from_field.*" "some_company")
       (from "some_mailer_that_needs_body_filtering"
      (| 
       (: split-on-body ".*Assignee: MY_NAME.*" "your_service_me_assigned")
       (: split-on-body ".*Reporter: MY_NAME.*" "your_service_me_requested")
       "your_crappy_service")))
      "me_somewhere"
      ))
      "misc")

This would split stuff by subject first (having /var/log/messages in the subject), then see if the header contains "some_sender" and put the mail in the mail folder "some_annoying_sender". If neither of those matched, then would look for MY_NAME in the mail header, and split the logic even further, to split from the "from" header, and even on the mail body. It can be any function, but this "split-on-body" one is a common one I guess, since it comes with the documentation and all.
This is the split-on-body function:

(defun split-on-body (regexp-to-search group-to-split)
  (save-excursion
    (save-restriction
      (widen)
      (goto-char (point-min))
      (when (re-search-forward regexp-to-search nil t)
 group-to-split))))

And it works great. But you get issues when filtering HTML mail... which gnus kind of usually parses someway (when you read it on gnus, it'll depend on the value of the "mm-text-html-renderer" variable).
The thing is, when the mail is being filtered, the text apparently is all in pure text, and that text kind of differs from the actual raw mail body, or what you see parsed in the mail view.
So, in short, you'll need to get the correct regexp, which you can't see anywhere but in the filter function itself...
So, here is a filter function for that purpose. It's pretty much the same as the above, but you get the prin1 text for the complete mail text in you *Messages* buffer.

;; debug version
(defun split-on-body (regexp-to-search group-to-split)
  "debug version"
    (save-excursion
      (save-restriction
 (widen)
 (goto-char (point-min))
 (message "searching for regexp in split on body")
 (let ((search-results (re-search-forward regexp-to-search nil t))
       (complete-buffer (prin1 (buffer-substring (point-min) (point-max))))
       )
   (message (concat  "regexp to search for = " regexp-to-search))
   (message (concat "search results =" (format "%S" search-results))))
 (goto-char (point-min))
 (when (re-search-forward regexp-to-search nil t)
   group-to-split))))

For some reason I didn't think of this until like 10 years after
starting using gnus...