Refactor the Life http://kunxi.org Yet another code monkey blog. Tue, 18 Nov 2008 06:59:54 +0000 http://wordpress.org/?v=2.6.1 en 191470http://www.feedburner.comThis is an XML content feed. It is intended to be viewed in a newsreader or syndicated to another site, subject to copyright and fair use. Poor man’s NAS http://feeds.kunxi.org/~r/rtl/~3/456860956/ http://kunxi.org/archives/2008/11/poor-mans-nas/#comments Sat, 15 Nov 2008 02:42:26 +0000 bookstack http://kunxi.org/?p=315 ShareThis

]]> A Network Attached Storage(NAS) has been in my wanted list for quite a long time, thanks to Live Search Cashback program to make it happen: a Western Digital MyBook World Edition(500GB). More information about the hardware specification:

  • ARM926EJ-Sid(wb) [41069265] revision 5 (ARMv5TEJ) 99.73 MHz
  • Memory: 32M
  • VIA Networking Velocity Family Gigabit Ethernet
  • WD5000AAVS-0 500G HD

I believe 100MHz ARM CPU is powerful enough to drive this tiny box, but the limited capacity of memory cripples it as a lame duck. The sustainable file write(85G using lftp mirror) rate is approximately 3.8MB/s. It hardly qualifies any service beyond file server. Now, it is time to hack.

Jailbreak and SSH

The first thing to do is to create a user in the web interface of MyBook as root with null password is banned for security reason. Log on with admin and 123456, create a user JOE and setup the password for later use.

Run the script discussed in the wiki, and ssh with JOE. Now you can su to root with blank password, 0wned!

User management

MyBook takes a very intricate way to manage users:

All Samba users are granted shell access, but unix password sync = yes is not set, the /etc/shadow and /var/private/smbpasswd are updated individually by a Perl script via the web interface. The only reasonable explanation is the minimized Samba lacks PAM support.

All user names are capitalized. I assume this is a brutal force approach to address the difference between Samba and Linux native accounts: Windows user name is case insensitive, while Linux is case-sensitive.

As the password scrambled in /etc/shadow, it is easier to add/delete/update users via the web interface, then fine-tune the corresponding files. The user administration executives are hidden in /usr/www/nbin.

Share with Samba

The default exported directory is /share/internal/PUBLIC, the permission of the directory is set as rwsr-sr-x, and the owner is www-data, YMMV. So any file/directory created will be owned by www-data. If you are unhappy with the name, you may add a user, e.g joe as discussed before, then add joe to www-data group:

# /etc/group, YMMV
www-data:x:33:share

remember to change the default mask in /etc/smb.conf:

create mask = 0775
directory mask = 0775

Package management

Though I am a big fan of Gentoo, it is a little bit paranoid to build everything from scratch. A precompiled package management, like Optware makes more sense. Check out this tutorial for bootstrapping.

The essential packages for daily administration imho are screen, lftp.

Feature requests

There are some itchy miss features, if you happen to know a solution or hint, please drop me a message in the comment:

Access Anywhere No mionet, just SSH. If you are a perfectionist, consider to port this Delphi application to MyBook to host MyBook in your preferred domain.

Download Manager A web front-end to listen to download requests from Firefox/IE plugins, then delegate it to wget backend with cookie support. A more aggressive approach may support megaupload happy hour.

]]> http://kunxi.org/archives/2008/11/poor-mans-nas/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F11%2Fpoor-mans-nas%2Fhttp://kunxi.org/archives/2008/11/poor-mans-nas/ Rewrite WordPress and ZenPhoto for Nginx http://feeds.kunxi.org/~r/rtl/~3/449633469/ http://kunxi.org/archives/2008/10/rewrite-wordpress-and-zenphoto-for-nginx/#comments Wed, 15 Oct 2008 15:08:47 +0000 bookstack http://kunxi.org/?p=300 ShareThis

]]> Nginx also supports URL rewrite, not compatible to Apache’s mod_rewrite, but more intuitive and more powerful imho. The only problem is most applications, WordPress and ZenPhoto for this specific case do include the mod_rewrite code snippet and/or may update the .htaccess for your convenience.

Thanks to the Slicehost community, the port of mod_rewrite rules perfectly covers WordPress and SuperCache. Here are some minor modification to craft for more general usage:

# the blog dir, aka where index.php is
set $blog_dir ”;
# the wordpress dir where all wp-* stays
set $wordpress_dir ‘/wordpress’;
include wordpress.rewrite;

In nginx.conf, define wordpress_dir and blog_dir, these two variables are equivalent to WordPress address (URL) and Blog address (URL) stripped off the host information. Then we can replace the hard-coded /blog path by using $wordpress_dir or $blog_dir:

2d1
<
26c25
< set $supercache_file /blog/wp-content/cache/supercache/$http_host/$1index.html;
---
> set $supercache_file $wordpress_dir/wp-content/cache/supercache/$http_host/$1index.html;
36c35
< rewrite . /blog/index.php last;
---
> rewrite ^(.*)$ $blog_dir/index.php?q=$1 last;

Here is my zenphoto.rewrite, it seems sivel’s more concise. Either of these should work.

]]> http://kunxi.org/archives/2008/10/rewrite-wordpress-and-zenphoto-for-nginx/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F10%2Frewrite-wordpress-and-zenphoto-for-nginx%2Fhttp://kunxi.org/archives/2008/10/rewrite-wordpress-and-zenphoto-for-nginx/ Parse HTML file with BeautifulSoup http://feeds.kunxi.org/~r/rtl/~3/419237797/ http://kunxi.org/archives/2008/10/parse-html-file-with-beautifulsoup/#comments Mon, 13 Oct 2008 06:20:35 +0000 bookstack http://kunxi.org/?p=307 ShareThis

]]> In the last post, regular expression is used to fetch the specific information. To access the structured information, BeautifulSoap BeautifulSoup is preferred for its simplicity and convenient API:

  • You may override the fromEncoding in the constructor, this is very useful for non-roman, non-standard web pages.
  • Versatile find/findAll on tag, attributes.
  • Developer-friendly syntactic sugar, the Tag implements the interface of string, list, dict and callable function, so there are many ways to access the data as you wish. The drawback of this approach is the typo is only caught in the run time instead of compilation time.
  • Easy to deploy, only one BeautifulSoup.py file.

Something I don’t like:

  • No XPath support, more efforts are needed to port from JavaScript.
  • The API does not support stream, or file object. Laziness is always cherished for pipelining.
  • Why BeautifulSoup? I have made typo as soap more than ten times.

Here is the home-brewed script to wage through the Dvbbs thread to find the corresponding messages： elevator.py

]]> http://kunxi.org/archives/2008/10/parse-html-file-with-beautifulsoup/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F10%2Fparse-html-file-with-beautifulsoup%2Fhttp://kunxi.org/archives/2008/10/parse-html-file-with-beautifulsoup/ WYS is not always WYG in python.re http://feeds.kunxi.org/~r/rtl/~3/419169128/ http://kunxi.org/archives/2008/10/wys-is-not-always-wyg-in-pythonre/#comments Fri, 03 Oct 2008 06:22:36 +0000 bookstack http://kunxi.org/?p=301 ShareThis

]]> After almost two month hard work, I finally check-in the feature, and tonight I decided to relax on some leisure python programming:

This side project is quite trivial, fetch the HTML content, search the keywords in the thread, and build links table of contents for navigation. The only intrigue highlight that make this post worthy your 5 minute is that the language of the page is Chinese, and it is encoded in GB2312.

Long story short, I am trying to search the total number of posts in the thread using this regular expression:

pattern = re.compile(‘(?<=<b class="page">总数 )(?P<total>\d+)’)

The first catch is I have to declare the code page used for the source code, as python interpreter complains:

SyntaxError: Non-ASCII character ‘\xe6′ in file ./elevator.py on line 17, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

OK, I will stick to UTF-8, so add this declaration in the second line:
# -*- coding: utf-8 -*-

It does not work. And the dumped content of the page is totally messy. Oops, we forget to decode the content to Unicode, use codec to wrap the urlopened handle:

gb = codecs.lookup(‘gb2312′)
    # load the page
    content = gb.streamreader(urllib.urlopen(url)).read()

And don’t forget to add either Unicode prefix or re.Unicode flag to the pattern.

pattern = re.compile(‘(?<=<b class="page">总数 )(?P<total>\d+)’, re.UNICODE)

Still no luck, but it works in the python console with the same pattern, faked data, and also works if we change a little bit:

pattern = re.compile(‘(?<=<b class="page">.{2} )(?P<total>\d+)’, re.UNICODE)

Looks like the trouble maker is the non-Latin characters: 总数. Let’s play a little bit in the pdb console:

(Pdb) ‘总数’
‘\xe6\x80\xbb\xe6\x95\xb0′
(Pdb) ‘总数’.decode(‘utf8′)
u‘\u603b\u6570′

And it works finally with the hard-coded Unicode character:

pattern = re.compile(‘(?<=<b class="page">\u603b\u6570 )(?P<total>\d+)’, re.UNICODE)

We can use the decode method to avoid the ugly Unicode string for better readability:

pattern = re.compile(‘(?<=<b class="page">总数 )(?P<total>\d+)’decode(‘utf-8′), re.UNICODE)

And a note is recorded that the decoded codec MUST be consistent to the code page declaration.

Some speculations based upon the observation:

  • re.UNICODE does not enforce the Unicode mode, it just redefine the escaped characters like: \b, \w etc.
  • The pattern and string in Unicode implicitly invokes the Unicode mode. That explains why some pattern works in Python console only. Both of them are encoded in UTF-8, so re really runs in 8bit!
  • Python interpreter will not translate the literal string even though the code page is specified.

Please leave your insight in the comments. Thanks

UPDATE:
Thanks for all the comments first. Seems that I have a typo when testing the pattern with Unicode prefix. Here are the test cases:

patterns = [
    re.compile(‘(?<=<b class="page">总数 )(?P<total>\d+)</b>’.decode(‘utf8′), re.UNICODE),
    re.compile(ur‘(?<=<b class="page">总数 )(?P<total>\d+)</b>’, re.UNICODE),
    re.compile(u‘(?<=<b class="page">总数 )(?P<total>\d+)</b>’, re.UNICODE),
    re.compile(‘(?<=<b class="page">总数 )(?P<total>\d+)</b>’, re.UNICODE),
    ]

print [ pattern.search(s) for pattern in patterns ]

The output is:

[<_sre.SRE_Match object at 0xb7c18260>, <_sre.SRE_Match object at 0xb7c18360>, <_sre.SRE_Match object at 0xb7c183a0>, None]

Download test.py

]]> http://kunxi.org/archives/2008/10/wys-is-not-always-wyg-in-pythonre/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F10%2Fwys-is-not-always-wyg-in-pythonre%2Fhttp://kunxi.org/archives/2008/10/wys-is-not-always-wyg-in-pythonre/ HOWTO: Serve virtual host with Nginx http://feeds.kunxi.org/~r/rtl/~3/386117326/ http://kunxi.org/archives/2008/09/howto-server-virtual-host-with-nginx/#comments Sat, 06 Sep 2008 20:44:44 +0000 bookstack http://kunxi.org/?p=290 ShareThis

]]> As the limited memory budget, and I plan to host multiple website in the VPS, I decided to take a less versatile, but lightweight Apache alternative, the Nginx made by the polar bear.

There is no RPM in the repository I have enlisted, so let’s fallback the old-school way:

# remove the blocking glibc-dummy-centos-4 package, then get the toolchain:
yum remove glibc-dummy-centos-4
yum -y install gcc openssl-devel
# Now it is time to build the nginx:
./configure –prefix=/opt/nginx –with-http_ssl_module –with-http_stub_status_module
make && sudo make install

Then get the PHP with FastCGI support, and the lighttpd-fastcgi for the fastcgi loader.

yum install php-cli php-mysql lighttpd-fastcgi

Here is the nginx.conf that server.

user nobody;
worker_processes 2;
pid logs/nginx.pid;
error_log logs/error.log;

events {
worker_connections 2048;
use epoll;
}

http {
include mime.types;
include fastcgi_params;
default_type application/octet-stream;

log_format main ‘$remote_addr - $remote_user [$time_local] $request ‘
‘”$status” $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for”‘;

access_log logs/access.log main;
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;

client_header_buffer_size 1k;
large_client_header_buffers 4 4k;

gzip on;
gzip_min_length 1100;
gzip_buffers 4 8k;
gzip_types text/plain;
gzip_static on;

output_buffers 1 32k;
postpone_output 1460;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 75 20;

server {
#REVIEW: how to redirect https? using re?
server_name www.kunxi.org;
rewrite ^(.*) http://kunxi.org$1 permanent;
}

server {
# kunxi’s gallery
listen 80;
server_name gallery.kunxi.org;
root /home/webadmin/$host;
access_log logs/$host.log main;

location / {
root /home/webadmin/$host;
index index.html index.htm index.php;
include zenphoto.rewrite;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
}

# deny access to .htaccess files.
location ~ /\.ht {
deny all;
}
}

server {
# kunxi’s sites
listen 80;
server_name kunxi.org *.kunxi.org;
root /home/webadmin/$host;

error_page 404 $document_root/404.html;
error_page 500 502 503 504 $document_root/50x.html;
access_log logs/$host.log main;

location / {
root /home/webadmin/$host;
index index.html index.htm index.php;

# the blog dir, aka where index.php is
set $blog_dir ”;
# the wordpress dir where all wp-* stays
set $wordpress_dir ‘/wordpress’;

include wordpress.rewrite;
}

# rewrite the /files/
#
location /files/ {
alias /home/webadmin/static.kunxi.org/;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
}

# deny access to .htaccess files.
#
location ~ /\.ht {
deny all;
}
}
}

Some highlights of the configuration:
Rewrite www.kunxi.org to kunxi.org, yes, we support no-www!

server {
#REVIEW: how to redirect https? using re?
server_name www.kunxi.org;
rewrite ^(.*) http://kunxi.org$1 permanent;
}

And this wildcards will cover all sub-domains powered by PHP:

server_name kunxi.org *.kunxi.org;
root /home/webadmin/$host;
… ….

Home-brewed nginx and fastcgi init scripts to make it works after the reboot:

chkconfig –add nginx
chkconfig –add fcgi-php
service start nginx
service start fcgi-php

Tips and Traps:
Nginx supports 0 downtime upgrade, so if your nginx.conf is wrong, the server would ignore it and suck up the complain. Make sure stop the nginx service and start nginx during debugging rewrite rules.

fcgi-php seems to have problem to parse localhost, so I use 127.0.0.1 instead.

The rewrite rule for WordPress and ZenPhoto are explained here and here. (TODO).

]]> http://kunxi.org/archives/2008/09/howto-server-virtual-host-with-nginx/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F09%2Fhowto-server-virtual-host-with-nginx%2Fhttp://kunxi.org/archives/2008/09/howto-server-virtual-host-with-nginx/ Move to the new VPS, Yummy http://feeds.kunxi.org/~r/rtl/~3/385637426/ http://kunxi.org/archives/2008/09/vps-warm-up/#comments Thu, 04 Sep 2008 08:35:41 +0000 bookstack http://www.kunxi.org/?p=284 ShareThis

]]> I finally ride the trend of VPS, yes, I just bought Economy VPS COS4×64 with 512M RAM, 200G HD and 350G bandwidth. The host is visualized via virtuozzo, and loaded with CentOS 4.4.

The package management of RPM is quite lacking, so the first thing is to get YUM working:

rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/yum-2.4.3-4.el4.centos.noarch.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/python-elementtree-1.2.6-5.el4.centos.x86_64.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/python-sqlite-1.1.7-1.2.1.x86_64.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/sqlite-3.3.6-2.x86_64.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/python-sqlite-1.1.7-1.2.1.x86_64.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/python-urlgrabber-2.9.8-2.noarch.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/yum-metadata-parser-1.0-8.el4.centos.x86_64.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/x86_64/CentOS/RPMS/yum-2.4.3-4.el4.centos.noarch.rpm
yum update

Since I have been spoiled by Gentoo’s portage, the upgrade of CentOS makes little sense to me: wipe off the disk, then reinstall the OS. The upgrade from CentOS 4 to CentOS is quite hair-rising, and I was scared off by the tech support. So I would stick to CentOS 4 right now.

The next step is to configure the repository:

wget -P /etc/yum.repos.d http://jpackage.org/jpackage.repo
wget -P /etc/yum.repos.d http://centos.karan.org/kbsingh-CentOS-Extras.repo
wget -P /etc/yum.repos.d http://centos.karan.org/kbsingh-CentOS-Misc.repo
wget -P /etc/yum.repos.d http://www.sipfoundry.org/pub/sipX/3.2/sipx-centos.repo
wget -P /etc/yum.repos.d http://dev.centos.org/centos/4/CentOS-Testing.repo

And manually add the following repos as well

# /etc/yum.repos.d/dag.repo
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
enabled=1

# /etc/yum.repos.d/utterramblings.repo
[utterramblings]
name=Jason’s Utter Ramblings Repo
baseurl=http://www.jasonlitka.com/media/EL$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=http://www.jasonlitka.com/media/RPM-GPG-KEY-jlitka

[dries]
name=Extra Fedora rpms dries - $releasever - $basearch
baseurl=http://ftp.belnet.be/packages/dries.ulyssis.org/redhat/el4/en/i386/dries/RPMS
gpgcheck=1
enabled=1

[atrpms]
name=Fedora Core $releasever - $basearch - ATrpms
baseurl=http://dl.atrpms.net/fc$releasever-$basearch/atrpms/stable
gpgkey=http://ATrpms.net/RPM-GPG-KEY.atrpms
gpgcheck=1enabled=1

Import the GPG keys as well:

rpm –import http://centos.karan.org/RPM-GPG-KEY-karan.org.txt
rpm –import http://dries.ulyssis.org/rpm/RPM-GPG-KEY.dries.txt
rpm –import http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt
rpm –import http://ATrpms.net/RPM-GPG-KEY.atrpms

And accelerate YUM by cache the meta data locally:

yum makecache

With YUM, it is much easier to start the engine.

]]> http://kunxi.org/archives/2008/09/vps-warm-up/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F09%2Fvps-warm-up%2Fhttp://kunxi.org/archives/2008/09/vps-warm-up/ Reverse mashup http://feeds.kunxi.org/~r/rtl/~3/385272893/ http://kunxi.org/archives/2008/09/reverse-mashup/#comments Wed, 03 Sep 2008 08:40:11 +0000 bookstack http://www.kunxi.org/?p=279 ShareThis

]]> I barely remember when was the last time I updated Gelman. This project starts as an exercise to practice my Django + dojo skills, and also to manage my eBook collection, I were stuck in UI design which I suck and the authentication/registration which I takes little interest in.

What if we take a different approach to avoid the reinvention of the wheel? Let’s take an online eBook management Web2.0 application, for example LibraryThing, to manage the Read, Reading, Wish to Read list, if an electronic copy is available, it will be inserted into the LibraryThing search/detail page, and we can add it to Reading list for quick access later. We just mashup our web service into LibraryThing!

This application can be decomposed to two parts:

  • A Web service to CRUD eBook collection hosted in home-brew server
  • Firefox Add-on or GreaseMonkey user script for mashup
  • A web service client to automate importing books
  • Find your favorite app, then mash it up!

A more portable solution to glue the pieces is to leave the mashup in the server side, using a proxy server to insert all the mashup data.

As all comments, tags, reading history has been hosted in the cloud, it is essential to do something against the raining day:

  • Big name may endorse the availability. I would seriously consider Shelfari only after it is acquired by Amazon
  • Open API is the key. This is the trend, the web site could not lock on the user by closing itself, the users may cold feet in the first place. Douban, the leader in the Chinese market, did an excellent job to open its platform.

I am working on an prototype of the RESTful web service using Django, stay in tune.

]]> http://kunxi.org/archives/2008/09/reverse-mashup/feed/ http://api.feedburner.com/awareness/1.0/GetItemData?uri=rtl&itemurl=http%3A%2F%2Fkunxi.org%2Farchives%2F2008%2F09%2Freverse-mashup%2Fhttp://kunxi.org/archives/2008/09/reverse-mashup/ http://api.feedburner.com/awareness/1.0/GetFeedData?uri=rtl