Wednesday, April 4, 2018

Macross news in RSS

What I did

An RSS newsfeed you can use with your favorite RSS reader to checkout Macross related news (it's all in Japanese though... based on the site Macross Portal (News))

RSS feeds



Source

https://github.com/rikijpn/macross-news-rss/


A bit about RSS

I love RSS.
I read lots of RSS feeds with emacs gnus, together with my mail at work, and it's just so much better than having to go to each website to look for news and stuffs. Due to my Free Software beliefs I don't use SNS/microblogging services very often.
You could even use it for twitter and stuffs, but I'd rather hope someday there will be a world where our main SNS services will be decentralized as diaspora... sorry, that's another topic.

A bit about this code

I don't know why I even bothered using that lxml library... it'd have saved me so much time to just write my own xml-like class. But in the long term I guess lxml is a lot better to maintain real (large) xml.
RSS isn't really made of a super complex XML, but a rather small, with a lot of commonly used fields defined one.

The simple version is my starting one, which is good enough for small things. But this site usually has some pics I end up wanting to see, so clicking on the "link" part very often. I'd stand that for RSS made by the official site, but since it's my whole thing, I decided to add a "full" version, with the complete page as the "contents" of each news, so I very rarely need to actually access the original site anymore, yay.

I put the scripts on cron to run daily, and to do the sync with the web server.

The hardest part was knowing how to write RSS... since apparently there are lots of things that aren't defined-per-se, but are common used like that, etc.
I ended up mimicking some site's RSS feed. Which gave me the most important hints I needed:
1. You need to use the "" tags instead of "" for CDATA (html code) including source.
2. CDATA let's you put HTML! Copying sections from web pages becomes a lot easier that way...
3. You need to declare the "encoded" Namespace (some kind of XML terminology I still don't really understand, but thanks to the lxml library I had to read a lot of in order to just make those little content:encoded tags!)

That's pretty much it.
All the rest you can just read in w3schools. They even have an XML namespaces section!

For the simple version you don't even need more than what w3schools has in their tutorials.


The code is a bit messy (especially the "full version" one). But basically I have one class to fetch the data, and another to make an RSS(XML) object.
So I fetch the main news titles page, and make an RSS feed out of it, voila.

In the "full version" one, as I'm fetching not only the titles, but the contents of the pages behind those titles, I'm saving the data every time, so only new titles' sub pages are fetched and not all pages every time.
Plus I added a description fetching class pretty similar to the one for the titles, but that gets the main HTML part I want to have as contents, and after checking the new news have no contents yet, add them there.


That's it

Well, I hope I never forget about another concert or new Valkyrie sales after this. You're welcome to use the RSS if you're a macross fan too.
I'm kinda regretting not having used lisp to write this thing since that would have probably been faster... but I guess I should be able to write some python too and all.

Thursday, October 19, 2017

LINE timeline video download

If you live in Japan you'll probably be using LINE to chat with everyone.

Here's how you can download movies from your timeline (your android won't let you without rooting it...).

You'll need:
1. chrome or chromium
2. the chrome line app? (maybe not needed)
3. some html/javascript knowledge?

This is what you do:
1. Login to your LINE account using the chrome LINE app
2. Click on the timeline icon on the app, this will take you to somewhere like https://timeline.line.me
3. Login to the timeline screeen
4. Do Ctrl-Shift-J to get the javascript console

5-a. You can inspect the DOM html code until the vid you want is highlighted, and click until you find the "video" html tag, then just copy paste the src attribute's value (an mp4 vid url)

5-b You can also just get all the vids in the page by typing the javascript:

  document.getElementsByTagName('video')

 if it's only one, then

  document.getElementsByTagName('video')[0].src

should give you the url for the vid, you just copy paste this

6. paste in browser's omnibar and start downloading!


Saturday, August 19, 2017

downloading torrents in GNU/linux (magnet URIs)

Torrents have changed so much...

Nowadays they're greatly distributed in magnet:XXX-looking URIs.
Check more about that in http://www.vuze.com/about-torrents/magnet-links.
I guess it's better than getting lots of torrent files.

Anyway, to be able to open and use these files, here's my favorite torrent client and how to use it.
It's utorrent!

http://www.utorrent.com/downloads/linux

1. Get the utorrent server (it's a server because you can actually access it from the internet too, or your LAN, but you can just use it on localhost as a normal client).
2. cd to wherever the file "utserver" is, and run "./utserver -settingspath &"
3. login to http://localhost:8080/ (user = admin, blank password by default)
4. click the chain looking icon to add a random torrent URL and you're done.

For non GUI related stuffs I liked aria2c, but it stopped working... lots of data are now being encrypted and everything, kinda going out of the original scope of a bittorrent client, which has some good and bad sides.

Anyway, have fun with utorrent.

Monday, August 7, 2017

common lisp getting drakma working from default install

Basic clisp setup to get drakma working


drakma is a lisp html client.

I'm using GNU CLISP 2.49.

If you just try to install it with the default asdf and quicklisp installed under it on debian 8 it won't work...

Anyway,
1. Install asdf's latest version https://common-lisp.net/project/asdf/ (I have 3.1.4) 
This means just downloading the source, and doing make mostly.
2. load your new asdf every time by adding it to your ~/clisprc.lisp
something like
(load "/yourhomedir/bin/lisp/clisp/asdf/build/asdf.lisp")
3. Install quicklisp https://www.quicklisp.org/beta/#installation
This is mostly downloading the install file, and doing 
(load "/yourhomedir/bin/lisp/clisp/install_ql/quicklisp.lisp")
(ql:add-to-init-file)

4. Then you're ready to install drakma
(ql:quickload :drakma) (drakma:http-request "https://www.debian.org/" :connection-timeout 'nil)
(in common lisp you need the :connection-timeout 'nil thing or it won't work... that's not very nice)

And you should be getting something the requested page on stdout:

...
...!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd
...


debian change display output interface from vga to dvi

Updating your X output interface


I have an nvidia video card, and I'm finally switching from VGA to DVI... I didn't even know I had that port till a few days ago.
I'm not adding a new display for dual/triple display, but just switching the cables for the same one and only display I have.

1. Plug in the DVI cable to your monitor and video card.
2. When you do the xrandr command, you should be seeing something like:
$ xrandr
Screen 0: minimum 8 x 8, current 1440 x 900, maximum 16384 x 16384
DVI-I-0 disconnected primary (normal left inverted right x axis y axis)
VGA-0 connected 1440x900+0+0 (normal left inverted right x axis y axis) 410mm x 260mm
   1440x900      59.89*+  74.98  
   1400x1050     74.87  
   1280x1024     75.02    60.02  
   1152x864      75.00  
   1024x768      75.03    70.07    66.00    60.00  
   800x600       75.00    72.19    60.32    56.25  
   640x480       75.00    72.81    65.99    59.94  
DVI-I-1 connected (normal left inverted right x axis y axis)
   1440x900      59.89 +  74.98  
   1400x1050     74.87  
   1280x1024     75.02    60.02  
   1152x864      75.00  
   1024x768      75.03    70.07    66.00    60.00  
   800x600       75.00    72.19    60.32    56.25  
   640x480       75.00    72.81    65.99    59.94  
HDMI-0 disconnected (normal left inverted right x axis y axis)


3. Here you see the VGA-0 (your current display), and DVI-I-1 (your new one) are connected.
The only thing to do next is:

xrandr --output DVI-I-1 --auto


4. Your monitor should blink for a moment. Then you're probably ready to go! Disconnect your VGA cable, and if you still have something on your screen, it's working, yay.

5. Disconnect your old VGA cable, and give it a good burial.

Sunday, June 4, 2017

fixing B-frames ('packed B-frames') vids

mplayer warning

[mpeg4 @ 0xb7411fe0]Video uses a non-standard and wasteful way to store B-frames ('packed B-frames'). Consider using a tool like VirtualDub or avidemux to fix it.

You'd get this warning, and video wouldn't be playable (it'd be super slow in the video stream, while the audio is OK).

Suggested fix

ffmpeg -i input_vid.avi -codec copy -bsf:v mpeg4_unpack_bframes   output_vid.avi

(didn't work, still was super slow)

Work-around

You'll need: avidemux, ffmpeg

Steps
1. convert file to avi (mine were in ogm format)

ffmpeg -i $INPUT_FILE -codec copy $OUTPUT_FILE

2. separate audio

ffmpeg -i $INPUT_FILE  -vn $OUTPUT_FILE

3. open the avi file on avidemux -> choose audio -> disable sound tracks

4. merge audio and video in one final vid

ffmpeg -i $VID_FILE -i $AUDIO_FILE -codec copy -shortest $OUTPUT_FILE

You can then just remove all remaining files created during the process and just the one on step 4. It'll still show the warning, but the video will be watchable (it was for me at least).

Sunday, November 15, 2015

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) fix on debian 8

I'm using:

Debian GNU/Linux 8
Python 2.7.9
openssl 1.0.1k-3


Using python (pip) to access https stuff, I got the error
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/search.py", line 40, in run
pypi_hits = self.search(query, index_url)
File "/usr/lib/python2.7/dist-packages/pip/commands/search.py", line 54, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1591, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
File "/usr/lib/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 812, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1212, in connect
server_hostname=server_hostname)
File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 566, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)


Quick fix (can or cannot work... and not to be permanent as not
verifying the certificates is not very smart)
In your code, do:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context


Permanent solution


rmdir /usr/local/share/ca-certificates (was blank)
ln -s /usr/share/ca-certificates/mozilla /usr/local/share/ca-certificates
sudo apt-get install ca-certificates # in case you don't have it already
sudo update-ca-certificates

# the new files will be sym-linked here:
/etc/ssl/certs

That fixed it for me at least.

Sociable