workaround for MySQL server has gone away error with flask

I had this issue when my flask simple app kept failing the next day it was started.
It kept giving the error "MySQL server has gone away".

I tried a couple of things like, starting the mysql session for every request instead of for each flask process (app-wide). But apparently flask doesn't like that.
At least not when using flaskext.mysql's MySQL.

I think I didn't have much luck trying to change the timeout on flask either, as this module just didn't let me.

I could have changed the module to use mysql, but I didn't have much time to finish the damn thing, so in the end I just set a cron job to access the mysql select query using page every 10 min or so. Ugly, but simple workaround.

It was like
curl -k https://localhost/some_app/index.html

every 10m.

flask app returns select with old data

I had this issue where my flask app kept giving me different results (doing a select query).

And it only happened on production, not in the development one.
Apparently due to the fact that flask has a couple of fork processes running, each with their own mysql session.
Even when you commit after an insert/update query, you still need for some reason to do a commit before a select in order to see the latest results.

The following is a sample code with some basic functions in order to do select and update queries.
I just call validate_sql_conn()  for pretty much everything I need to do in mysql, so it's always committing itself.

from flask import Flask, render_template, url_for, request, send_from_directory
from flaskext.mysql import MySQL
import yaml

app = Flask(__name__, template_folder='views')

def get_mysql_conf(mysql_conf_file="some_app_name_conf/mysql_conf.yml"):
    "gets the conf to use for the mysql connection"
    with open(mysql_conf_file, 'r') as f:
        content = f.read()
    yaml_content = yaml.load(content, Loader=yaml.FullLoader)
    return yaml_content

mysql = MySQL()
mysql_conf = get_mysql_conf()
app.config['MYSQL_DATABASE_USER'] = mysql_conf['mysql_database_user']
app.config['MYSQL_DATABASE_PASSWORD'] = mysql_conf['mysql_database_password']
app.config['MYSQL_DATABASE_DB'] = mysql_conf['mysql_database_db']
app.config['MYSQL_DATABASE_HOST'] = mysql_conf['mysql_database_host']
mysql.init_app(app)

conn = mysql.connect()
cursor = conn.cursor()


def validate_sql_conn():
    """
    uses global connection variable (conn) and recreates 
    it if seems to have dropped
    """
    global conn
    global cursor
    try:
        cursor.execute("show tables")
    except:
        conn = mysql.connect()
        cursor = conn.cursor()
    conn.commit()
    return True
        

def run_sql_and_get_results(sql_query, cursor=cursor):
    validate_sql_conn()
    cursor.execute(sql_query)
    data = cursor.fetchall()
    return data


def run_sql_and_commit(sql_query, cursor=cursor, conn=conn):
    validate_sql_conn()
    cursor.execute(sql_query)
    data = cursor.fetchall()
    commit_data = conn.commit()
    return commit_data

gnus filtering html body mail

If you love emacs, you probably use gnus, which is in my opinion the
best mailer ever.
You can manage your complete filter in emacs lisp, and there are so many
ways to filter stuff.

The most complex and useful filtering method is using
"nnmail-split-fancy".

Which as I'm showing in this example, can be set as something like this:

(setq nnmail-split-fancy
      '(|
 ("subject" ".*/var/log/messages.*" "server_syslogs")
 (any "some_sender" "some_annoying_sender")
 ;; to me, not mailing lists
 (any ".*MY_NAME.*"
      (|
       (from ".*some_from_field.*" "some_company")
       (from "some_mailer_that_needs_body_filtering"
      (| 
       (: split-on-body ".*Assignee: MY_NAME.*" "your_service_me_assigned")
       (: split-on-body ".*Reporter: MY_NAME.*" "your_service_me_requested")
       "your_crappy_service")))
      "me_somewhere"
      ))
      "misc")

This would split stuff by subject first (having /var/log/messages in the subject), then see if the header contains "some_sender" and put the mail in the mail folder "some_annoying_sender". If neither of those matched, then would look for MY_NAME in the mail header, and split the logic even further, to split from the "from" header, and even on the mail body. It can be any function, but this "split-on-body" one is a common one I guess, since it comes with the documentation and all.
This is the split-on-body function:

(defun split-on-body (regexp-to-search group-to-split)
  (save-excursion
    (save-restriction
      (widen)
      (goto-char (point-min))
      (when (re-search-forward regexp-to-search nil t)
 group-to-split))))

And it works great. But you get issues when filtering HTML mail... which gnus kind of usually parses someway (when you read it on gnus, it'll depend on the value of the "mm-text-html-renderer" variable).
The thing is, when the mail is being filtered, the text apparently is all in pure text, and that text kind of differs from the actual raw mail body, or what you see parsed in the mail view.
So, in short, you'll need to get the correct regexp, which you can't see anywhere but in the filter function itself...
So, here is a filter function for that purpose. It's pretty much the same as the above, but you get the prin1 text for the complete mail text in you *Messages* buffer.

;; debug version
(defun split-on-body (regexp-to-search group-to-split)
  "debug version"
    (save-excursion
      (save-restriction
 (widen)
 (goto-char (point-min))
 (message "searching for regexp in split on body")
 (let ((search-results (re-search-forward regexp-to-search nil t))
       (complete-buffer (prin1 (buffer-substring (point-min) (point-max))))
       )
   (message (concat  "regexp to search for = " regexp-to-search))
   (message (concat "search results =" (format "%S" search-results))))
 (goto-char (point-min))
 (when (re-search-forward regexp-to-search nil t)
   group-to-split))))

For some reason I didn't think of this until like 10 years after
starting using gnus...

rsyslog change output log file based on hostname

I want to put all the logs for hostnames starting with XXX in
/var/log/XXX.log.

And I'd also like to be able to read the file without having to do sudo
everytime...

Here is how:

in /etc/rsyslog.conf:

# remember to open your udp port to receive logs from other servers

$ModLoad imudp
$UDPServerRun 514
#### GLOBAL DIRECTIVES ####

# change umask so the default one doesn't mess with your filecreatemode permissions
$umask 0000

then create a file in /etc/rsyslog.d/50-my-XX-logs.conf:

#(any number is fine)

# this makes the file readable by anyone
$FileCreateMode 0644
:HOSTNAME, startswith, "XXX" /var/log/UHN2.log
# and stop any further filtering with the next line
& stop

This made it work for me. You can check more on the filters and
conditionals with rsyslog.conf(5) manual page.

use youtube-dl with sites that need credentials/cookies

You might want to download something with youtube-dl, and get the following error:

ERROR: XXXX requires authentication. You may want to use --cookies.


Quick way to get the cookies and do the download:
1. get the chromium/chrome extension Editthiscookie
https://chrome.google.com/webstore/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg/related?hl=en

2. on editthiscookie's settings, go to options -> "Choose the preferred
export format for cookies" -> "Netscape HTTP Cookie File"

3. on the site you're trying to download the video, click on
editthiscookie's extension, export

4. paste/yank your clipboard's contents into a file

5. do youtube-dl --cookies exported_cookies_file https://the-vid-you-wanted-to-download

fix emacs package-list's "Failed to verify signature..."

Everytime I called M-x package-list-packages, I got

package--check-signature: Failed to verify signature
some_package.el.sig: ("No public key for XXX")

I'm not really sure what effect this had, but it was annoying.


Fix

1. in a random buffer, do

(setq package-check-signature nil) ;; press C-x C-e here

2. do M-x package-list-packages
search for gnu-elpa (complete name: gnu-elpa-keyring-update)
and install that

3. then just restart emacs or do
(setq package-check-signature 'allow-unsigned) ;; press C-x C-e here


The next time you run package-list-packages, the warning should be gone.

emacs as rest client (API testing interface, like postman)

I love being able to use emacs as a REST API testing interface, kinda like postman, but in emacs.

Here are two ways to do this:

- restclient-mode http://emacsrocks.com/e15.html
- walkman https://github.com/abrochard/walkman

In short

Restclient should be your thing 90% of the time.
walkman prob 10%. But walkman works inside org-mode, so if you like org,
maybe can be 100% your thing.

In long

Restclient's thing even has a cute video and everything. So,
noob-friendly.
Resclient is probably the de facto standard right now to use in this
kind of thing though.
It's awesome, you can easily use variables, which can be plain elisp,
and it uses emacs' internal http request libraries for HTTP interaction.
The output is json-prettified and all, and requests are done async, very
nice.

The only thing, is that those libraries appear to be a bit buggy... and
sometimes won't work as expected.
I have this thing, that for this specific site, I always get a 404 or
500 status code.
But when I generate the curl request from the exact same block, it works
(curl works, elisp's request library apparently doesn't like my
headers ?).

So for that API, I use walkman inside org-mode, which uses pure curl,
and that works perfectly.

I like walkman a lot, as I love org-mode.
But the variable definitions must done in elisp, and you have to quote
them to use them... which is not very readable (and elisp code doesn't
look too good in org mode).

Samples

restclient

# -*- mode: restclient; -*-
# block1 gets fsf's site
GET https://www.fsf.org

# press C-c C-c somewhere in this block to send the request
# block2 POSTing stuff with a payload
POST https://your-fav-api/v1/resource
YOUR-HEADERS-HERE
{your payload
can use multiple lines
}

# block3 ...

walkman

* get a page
GET https://www.fsf.org

# you do C-c C-RET here to send the request

* post something
  POST https://your-favorite-api/v1/resource
  - HEADER1
  - HEADER2
  { PAYLOAD-HERE }

That's about it. They're both pretty intuitive, and the documentation is
clear enough to let you do whatever you want within like 5mins of
looking for it.

I'd usually want to test/use the API interface in a different file. So I
guess I'd be using restclient mainly, linking it from my "task" file in
org-mode. Mostly because the variables, and output are more readable
(highlighted and everything).
And when that fails, use walkman.

But that's just me.

playing dvds in debian10

Steps
1. add the contrib (and probably non-free) repo to your sources list
2. install the libdvd-pkg package, and a player
3. run the libdvdcss library installer
4. (might be necessary to) set the region for your dvd device
5. try playing it

From top to bottom
if necessary, change your sources list

$ cat /etc/apt/sources.list
deb http://debian-mirror.sakura.ne.jp/debian/ buster main non-free contrib
deb-src http://debian-mirror.sakura.ne.jp/debian/ buster main non-free contrib

...

(you need to have the contrib repo as above)

if you did any changes, do "sudo apt-get update" to fetch the new repo data.

install the libdvd-pkg

$ sudo apt-get install libdvd-pkg

or better

$ sudo apt-get install libdvd-pkg libdvdcss-dev libdvdcss2 libdvdcss2-dbgsym libdvdnav4 libdvdread4 lsdvd regionset mpv lsdvd

then run the configure script, to compile the libdvdcss library

$ sudo dpkg-reconfigure libdvd-pkg

I don't remember if you needed to do this the first time, or just if your dvd wouldn't play... but try inserting a dvd, and do

$ sudo regionset

Some commercial DVDs won't play unless you have the "correct" dvd region for the region they were supposed to be sold to. Capitalism can be scary.

So, you can select a region, or just leave it as it is.

More info at /usr/share/doc/regionset/README

Finally, let's try playing something

$ mpv dvd://1

And you can check your dvd contents with

$ lsdvd

I'm sure you can also click around with your favorite desktop environment and find a way to play your dvd as well.