Friday, 9 May 2014

Strange problem on KVM VPS



This one for the X-Files.

I have been helping a client migrate his web server from one VPS to another because he needed more disk space. His previous VPS is based on Xen, and the migration is to a KVM-based VPS.

Everything was smooth-going at first. I installed Debian 7 x64, all the usual packages: nginx, php, mariadb etc. Easy. Configured and tested the setup. Easy.

Then I migrated his database and web server over and asked him to do some business-side testing. That was when we hit an unusual snag. His PayPal IPN wouldn't work.

At first, I traced the logs, thinking it was some kind of PHP code incompatibility. Nothing. The code will break at this point:

   $fp = fsockopen('ssl://www.paypal.com', 443, $eno, $estr, 30);

and never return.

I was stumped. So I wrote a simple test program that retrieves the HTTP and the HTTPS version of google.com. Lo and behold, the HTTPS fetch breaks right at that point with:

    Illegal instruction

I looked up what that means and found that it is just another name for the good ol' core dump. Shocked, I ran the same code on the Xen VPS (running the same Debian 7 and PHP version) and the code ran OK.

A number of searches did not turn up anything useful on this problem. It doesn't seem anyone else has encountered this. Having run out of ideas, I reported the details back to my client and suggested that he migrate to a Xen-based VPS instead.

I hope one day I will find out what this was all about...

Bluetooth headsets: Samsung HM1700 vs Plantonics M25

I had been using a Samsung HM1700 headset for over a year now. Recently, the Plantonics M25 was on sale. It looks a lot smaller, lighter and sleeker than the Samsung HM1700, it supports A2DP, it has long battery life, so I bought one to have a try.

The thing about Bluetooth headset is that you can never tell how well it works for you from the specs. You have to live with it for a week or two before you can tell if the two of you are compatible, sorta like a roommate or girlfriend. So I lived with the M25 for a week.

Samsung HM1700

Pros:
  • Long battery life
  • Great reception
Cons:
  • Bigger and heavier
  • Ear loop is not as comfortable
Plantonics M25

Pros:
  • Long battery life
  • Sleeker, smaller and lighter
  • Ear loop is fitting and comfortable
Cons:
  • Too many A2DP dropouts for my liking. By that, I mean the A2DP stream will drop out for a second or two being picking up again. This happens when I turn or bend my body, pick up a tool etc. It happens with the HM1700 too, but much less frequently.
So it seems like I will be sticking with the HM1700 for now and returning the M25. Let's hope something better comes along later at the same price point.

Friday, 14 February 2014

Bitcoin - Figures on transaction cost

Here are some figures on transaction costs in cross-border transactions.

Currently the de-facto reference currency in cross-border transactions is the USD.

A recent purchase of USD186 using the credit card resulted in the following line items in AUD:
  • AUD (from USD) - $202.00
  • Currency conversion assessment fee: $0.40
  • International transaction fee: $3.00
  • Cross border assessment fee: $1.60
Ignoring the spread used in the currency conversion, the transaction cost is about 2.5%. The actual transaction cost taking the spread into account is probably about 3+%

A recent purchase of USD700 using PayPal incurred a fee of $23 (buyer or seller pays). The transaction cost is a whooping 3.3%. I happened to have some USD in my account, so I didn't need to perform currency conversion. PayPal's spread in forex rate is notoriously poor compared to other avenues.

A purchase of 1.45BTC (approximately USD1085 at that time) incurred a transaction fee of 0.0002BTC, which is negligible. Speed rivals that of PayPal. Again, I happened to have some BTC lying around, otherwise conversion from fiat to BTC will incur additional charges, typically in the 2% range.

So if BTC (or some other crypto-currency) replaces USD as the de-facto reference currency for transaction, it should drastically lower the transaction cost. However, the price volatility of BTC is one factor that prevents this from happening. BTC needs to be relatively stable against other currencies before it will be widely adopted for cross-border transactions.


Friday, 7 February 2014

Bitcoin - Micropayment system for the Internet?

I have been dabbling with crypto-currencies, specifically Bitcoin and Litecoin, over the past 12 months. Everything from mining to e-commerce to currency exchange. I think I "get" the magic of the whole idea now.

For better or for worse, crypto-currencies drastically reduces the friction of money flow. You probably won't be able to find another means of moving money at a lower transaction cost, especially when small amounts are involved.

In addition, the speed of transfer is excellent for a distributed system. With Bitcoin, it takes a few minutes. With Litecoin, it takes a couple of seconds. The low transaction fee and fast transfer speed respect no geographical boundaries. It applies whether you are moving money within the same country, or between two countries on opposite ends of the Earth. Which probably freaked more than a few legislators out.

Plus this is a distributed system with no central authority to slow things down or screw things up. PayPal is fast, but try talking to the people who have had their PayPal accounts frozen at a whim and had to jump through hoops to unfreeze their account, sometimes without success. Also, PayPal's transaction fees are notoriously high (Visa/Mastercard's fees are even higher) and they have to impose random rules required by the powers-that-be of the countries they operate it. For example, in certain countries, PayPal now charges a fee for transferring money between two individuals regardless of amount, even though PayPal started out as an easy way for people to email small sums of money to one another!

It is therefore clear that crypto-currencies like Bitcoin have unique features to offer over existing systems, so it is not entirely frivolous. It has the potential to solve problems such as micropayment on the Internet, which until now has not taken off due to the high transaction cost and lack of standard, so we pay for content with our privacy via online ads. If one of the crypto-currency becomes de-facto standard, it should be quite straightforward to have a browser that will automatically tip the author when content is accessed, made worthwhile due to the low transaction fee.

Which is not to say there aren't teething issues to be resolved. The idea of keeping all the transactions that ever existed in a monolithic blockchain makes it huge, and it is unclear whether such a system can accommodate micro-transactions of this scale without also raising the transaction fee. But there are ideas such as making the blockchain itself distribtued, which should eventually make its way into the mainstream.

There is also currently a heated debate between inflationary versus deflationary crypto-currency. Quite a few crypto-currency are intentionally deflationary, and standard economic theory tells us that people will tend to hoard deflationary currencies, which is probably what is driving the massive speculative activity behind Bitcoin right now. For a crypto-currency to be used instead of hoarded, it probably needs to be mildly inflationary. And some kind of clever algorithm to dynamically adjust the rate of inflation based on economic activity would be nice!

Still, like other P2P technologies, I think the cat is out of the bag and there is no turning back the clock. The concept of distributed crypto-currency is here to stay and it will only evolve at the speed of software. Eventually, I think most, if not all, of the kinks will be ironed out, one or more standards will emerge, and there will be widespread adoption by netizens. The next few years will be interesting to watch in this space...

Thursday, 15 August 2013

110GB+ worth of photos/videos on a microSD memory card #2

In my previous post, I talked about wanting to write a script to see how far I can squeeze 110GB worth of media (photos and videos).

I wrote the script the quick-and-dirty way, in PHP. When people think of PHP, they think of web pages, but PHP can equally be used for writing your everyday scripts, just like Perl or Python.

The script resizes all photos to a maximum width/height of 1024. I found this provides maximum size reduction for the photos without sacrificing the viewing quality in full photo mode on my tablet (screen resolution: 1280x800). Obviously it will affect the quality when zooming in, but I rarely do that when flipping through the photos. Since I haven't got a tablet with retina display yet, I am not sure how that will fare, but for this exercise, I am sacrificing quality for size.

For videos, I use handbrake-cli to transcode all of them to MP4 with a maximum width of 640. Again, this reduces the size drastically, improves compatibility (since all Android tablets/smartphones I have come across so far can play MP4 with no problems) and looks OK quality-wise on my tablet.

The script takes a source directory and replicates the folder structure in a target directory. It transcodes all files with certain extensions (configurable in config.php) and places them in the corresponding target subdirectories. If the file has already been transcoded, it will not be transcoded again, so it will be relatively quick to update the target directory after adding more files to the source directory.

The script took 1.5 days to transcode my 110GB collection (bear in mind this is a older, slower AMD Turion II laptop). The result was only 29GB and fits easily into a 32GB microSD card.

I popped the microSD card into my tablet and viewed the files using QuickPic. The result was awesome.Check it out in the video below:



Note that this video was produced with the microSD card inserted into an al-cheapo single-core tablet bought for the kids during a sale. I was pleasantly surprised by how quick and smooth it was. Needless to say, on my own 10" quad-core tablet, the result was even better.

QuickPic is a better app than Gallery for this purpose because:

1. It allows you to set "Include Folders" to include only the media files from the microSD card

2. It lets you set the 'Explorer View" to browse through the folders/subfolders in a hierarchical manner.

3. It has "Stack", "Grid" and "List" views for different visual summaries of the folder content.

4. It is quick and has tons of other options.

The result of the transcoding operation exceeded all my expectations. I have a snapshot of 110GB worth of media on a tiny memory card, ready to be popped into any tablet or PC to view. At only 29GB, there's lots of room for growth, especially on a 64GB microSD card (which no doubt all tablets will support in the future, even cheapo ones).

If you think you might find this useful, you can download the script below. It includes the PHP and Handbrake-CLI binaries so it should be ready to run under Windows. It shouldn't be too difficult to adapt the script to work under other platforms as well.

After download, unzip into another directory and edit config.php. You only need to change $srcdir and $tgtdir. Then bring up a command prompt and run transcode.


Once you verify that everything works correctly, you may want to schedule it to run periodically via the Task Scheduler.

Download: Batch Media Transcoder V1.0 (ZIP 25.2MB)

Download: Batch Media Transcoder V1.0 (7-ZIP 17.1MB)

110GB+ worth of photos/videos on a microSD memory card #1

We have a family NAS where we keep all kinds of stuff. Well, actually it's an old laptop that functions as a server, with 3 x 3TB external USB HDDs attached. I run all the standard mirroring and backup on the laptop, and one of the HDD is shared over the network for everyone to dump their stuff into.

Like most family these days, we take photos and videos on a number of different devices. The smart phones, digital camera, DSLR etc. Even the kids have their own cheap digital compacts. And the devices have evolved over the years. The photos have mostly stayed JPEG, with the resolution increasing steadily. The videos, however, are a different story. Starting from the V8 (which we converted to XVID AVIs) to a Sony (which produced MPEG-1 files) to a Casio (which gave us H264/MOV files) to the current-gen Sony (MTS format), there's an incredible number of video formats we have to deal with. On top of that, we do scans of old photos and slides, the kids' artwork etc. These are all dumped into the family NAS.

To avoid being locked into a particular platform, we resisted moving to any software that relies on an underlying database for organizing the photos. Instead we created folders (2010, 2011, 2012 etc) and subfolders (eg. 20100825 Adam's Birthday Party) within the underlying filesystem to organize the photos. With the subfolders, the photos and videos are renamed to "YYYYMMDD-hhmmss" format. jhead is a great help batch-renaming of photos. For video files, the strategy varies. And not all filles can be renamed this way (eg. the scanned photos). Though crude, this has mostly worked. It is also quite flexible when it comes to backup and batch processing.

For viewing the photos, we use XnView.


IrfanView works as well, but it requires more tinkering to get it working with the different video file formats that we have. XnView works pretty much out of the box with the video files.


These two apps have a number of advantages going for them:

1. They are both freeware with lots of plugin support.

2. They work at the filesystem level, and we can easily expand/collapse the folders/subfolders to view our media archive. Some apps require you to import the media that you want to view.

3. They display both photos and videos, both at the thumbnail and individual media level. Most media viewing apps support only photos.

4. They are portable, meaning they can be easily copied/moved from machine to machine without re-installation and executed with their settings intact.

The downside is that both apps are not particularly sexy, but they do get the work done quickly and effectively.

One problem I face is that the photo/video archive on the NAS is standing at 110GB and growing steadily. I want to be able to view this archive on my Android tablet, which is a very natural medium for perusing all the photos and videos that we have accumulated. But fitting 110GB of files into the tablet is next to impossible currently (the largest microSD card is 64GB), and even if I could, the tablet won't be able to deal with all the video formats in the archive (the 1080p MTS files are particularly problematic).

So I decided to write a script to transcode all the files in the archive to a smaller size to see how much I can squeeze it down without ruining the viewing experience on the tablet. This will be the topic of my next post.


Wednesday, 31 July 2013

Fast Diet - Hunger Pangs

When I am doing the Fast Diet, I frequently experience long periods of calm after a brief period of hunger pangs. It is as if the body has gone into hibernation, and I am left well alone to carry on with my daily routine. This is most apparent the morning after the diet day, when I could usually skip breakfast and have a late lunch.

I am reminded of what my diving instructor told me during training many years ago, that feelings of suffocation after a brief period under water is not due to lack of oxygen, but due to the buildup of carbon dioxide in the blood stream. This means you are not going to die (yet), and with knowledge you should  try to suppress the panic, remain calm and find a way out of the current situation to surface.

Hunger is mostly the same thing. It makes us uncomfortable and compels us to eat to rid ourselves of the discomfort. Worse, we have come to associate a lot of other feelings (both positive and negative) with eating. So we eat to make ourselves feel better, not necessarily when we are hungry. And this slowly builds up over the years in the world we live in today, where food is available in abundence 24x7.

I think occasional calories restriction (which is essentially what the Fast Diet is about) serves to remind ourselves that it is OK to feel hungry, that you don't have to eat at the first sign of hunger. It also serves to calibrate the "gauges" in our bodies and I think that's where the health benefits are derived.