Migrating F-Spot to Darktable

F-Spot was my favorite photo management program for the past years. In connection with GIMP as photo editor it was a good team. Again and again I came accross Windows users using Lightroom which seemed to make everything so easy. After some other photo editing session I reached the point where the limit was reached. I had to change something! But what? Switch to Windows? Nogo! After searching for alternative solutions I cam accross darktable. This piece of software seemed to be exactly what I was searching for. After some test it was clear: I need to move from F-Spot to Darktable.

The main problem: There is no import function for migrating the photo library from F-Spot to Darktable. At the moment I have like 20k photos in my database and all are well tagged. I would never throw that work away! After some searching I realized that there is no solution out there. I had to create a solution for myself.

I knew that F-Spot stores it’s information about images in an SQLite database. It should be no problem to extract those information. After a short chat at the #darktable channel in freenode IRC network I knew that darktable reads the *.xmp files during import of the images. Seems to be a straight forward solution: Write a script which reads the photos and tags from the F-Spot SQLite database and create an .xmp file for each image. Then import the images to Darktable. And be happy.

The plan was a good one. It worked like a charm. Don’t know if some others might trap into this situation but I decided to release the script. Here it is:

# encoding: utf-8
# Copyright (C) 2012 Lars Michelsen <lm@larsmichelsen.com>,
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307,
# GNU General Public License: http://www.gnu.org/licenses/gpl-2.0.txt
# Report bugs to: lm@larsmichelsen.com
import os, sys, urllib2
fspot_db = '%s/.config/f-spot/photos.db' % os.getenv('HOME')
opt_verbose = '-v' in sys.argv or '--verbose' in sys.argv
opt_help    = '-h' in sys.argv or '--help' in sys.argv
def help():
 This script helps migrating your photo library from F-Spot to Darktable.
 It extracts assigned tags from the F-Spot SQLite database and creates
 .xmp files for each image with the tag information. The script cares
 about and preserves the tag hierarchy structure.
 The script must be executed as user which F-Spot DB should be migrated.
 The script will query the SQLite database of F-Spot and extract all tags
 and hierarchical tag information from the database for images which do
 exist on your harddrive and do not have an associated xmp file yet. It
 will write those information in a basic xmp file.
 This script has been developed to be executed only once for a photo
 library. The cleanest way is to execute it before you import your images
 into Darktable. This way you can ensure all the new tags are really loaded
 correctly into Darktable.
 During development I removed the images again and again from the Darktable
 database and also removed all *.xmp files in my photo folders to have a
 clean start. After cleaning up those things I ran fspot2darktable, then
 started Darktable and imported all the fotos.
 After executing the script you can import single images or the whole
 photo library to Darktable. You should see your tag definitions in
 Darktable now.
 The script has been developed with
   - F-Spot 0.8.2
   - Darktable 1.0
 Please report bugs to <lm@larsmichelsen.com>.
def err(s):
    sys.stderr.write('%s\n' % s)
def log(s):
    sys.stdout.write('%s\n' % s)
def verbose(s):
    if opt_verbose:
        sys.stdout.write('%s\n' % s)
if opt_help:
    import sqlite3
except Exception, e:
    err('Unable to import sqlite3 module (%s)' % e)
if not os.path.exists(fspot_db):
    err('The F-Spot database at %s does not exist.' % fspot_db)
conn = sqlite3.connect(fspot_db)
cur  = conn.cursor()
# Loop all images, get all tags for each image
cur.execute('SELECT id, base_uri, filename FROM photos')
num_files = 0
num_not_existing = 0
num_xmp_existing = 0
num_created = 0
for id, base_uri, filename in cur:
    num_files += 1
    # F-Spot URLs are url encoded. Decode them here. There seem to be
    # some encoding mixups possible. Damn. Try simple utf-8 then latin-1
    # vs utf-8. This works for me but might not for others... let me know
    # if you got a better way solving this
    path = urllib2.unquote(base_uri.replace('file://', ''))
        path = path.decode('utf8')
    except UnicodeEncodeError:
        path = path.encode('latin-1').decode('utf-8')
    path += '/' + filename
    xmp_path = path + '.xmp'
    if not os.path.exists(path):
        verbose('Skipping non existant image (%s)' % path)
        num_not_existing += 1
    if os.path.exists(xmp_path):
        verbose('Skipping because of existing XMP file (%s)' % path)
        num_xmp_existing += 1
    # Walks the tag categories upwards to find all the parent tags to
    # build a list of parent tags. This will be used later to build
    # hierarchical tags in the XMP files instead simple tags
    def parent_tags(category_id):
        cur = conn.cursor()
            'SELECT id, name, is_category, category_id '
            'FROM tags WHERE id=\'%d\'' % category_id
        tag_id, tag, is_category, category_id = cur.fetchone()
        parent_tags_list = []
        if category_id != 0:
            parent_tags_list += parent_tags(category_id)
        return parent_tags_list + [ tag ]
    hierarchical_tags = []
    simple_tags       = []
    cur2 = conn.cursor()
        'SELECT tag_id, name, is_category, category_id '
        'FROM tags, photo_tags WHERE photo_id=\'%d\' AND id=tag_id' % id)
    for tag_id, tag, is_category, category_id in cur2:
        if category_id:
            hierarchical_tags.append('|'.join(parent_tags(category_id) + [ tag ]))
    def xml_fmt(tags):
        return ''.join([ '      <rdf:li>%s</rdf:li>' % \
                        t.encode('utf-8') for t in tags ])
    # Now really create the xmp file
    file(xmp_path, 'w').write('''<?xpacket begin="<feff>" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
<?xpacket end="w"?>''' % (xml_fmt(hierarchical_tags),
    num_created += 1
log('FINISHED! - Summary:')
log(' %-10s images in total' % num_files)
log(' %-10s images do not exist' % num_not_existing)
log(' %-10s images already have xmp files' % num_xmp_existing)
log(' %-10s created xmp files' % num_created)

Simply copy the script to your system, name it for example fspot2darktable, make it executable (chmod +x fspot2darktable) and execute it with ./fspot2darktable -h. The help output should give you enough information about how the script is working and what it is doing. To let the script do it’s work execute it without parameters like this: ./fspot2darktable.

Filed under: Open Source

Install linux check_mk_agent on QNAP 495 NAS

The QNAP 495 NAS is running a custom linux operating system provided by QNAP.

First we need make the xinetd available on the qnap device. It is possible to install an xinetd package using optware. This is a quick walk through the installation steps:

If you do not have installed optware yet you need to install it first. Go to the webinterface of the device and go to the page “Application servers -> qpkg plugins”. There install the correct package of optware. You need to download it first from the qnap website.

After installing optware you can use the ipkg command to install third party packages. If you need to use a proxy to connect to the web you need to configure your proxy e.g. in /etc/wgetrc or /root/.wgetrc using these statements:

http_proxy = http://proxy.domain.com:3128
use_proxy = on

Now you can execute the command:

ipkg install xinetd
configure xinetd

Now you can copy the linux check_mk_agent from e.g. from the Check_MK git repository (http://git.mathias-kettner.de/git/?p=check_mk.git;a=tree;f=agents;hb=HEAD) to your qnap device. A good path to store it is /opt/bin/check_mk_agent. You also need to make it executable.

Now you have to configure the xinted. You can use the sample xinetd.conf from the git repository linked above. But don’t forget to change the paths to the check_mk_agent script.

There are two modifications needed in the /opt/bin/check_mk_agent script to make the agent work:

  • Change the shebang from #!/bin/bash to #!/bin/sh
  • Change the df command to
    df -k | sed 1d | awk '{print $1" ext4 "$2" "$3" "$4" "$5" "$6}'

If you start the xinetd now you should be able to connect to the agent from your Nagios host.

Filed under: Nagios

Mixed release notes (NagVis 1.6.4, nagios_downtime.vbs 0.8.3)

It’s been a while since my last post. I have so many different projects running that I had no time to write about them. So you have to be content with this short note.

I just released a new version of the windows nagios downtime script (nagios_downtime.vbs). It has mainly been powered by a couple of changes submitted by Olaf Morgenstern (thanks!). You can download the latest version directly from the git repository.

Wow. I just wanted to write some short notes about the latest NagVis release but then I realized NagVis 1.6.4 has been announced a week ago. So no big news here. But maybe a good hint for the ones who did not notice yet.

I think that’s it for the moment …

Filed under: Nagios

NagVis 1.6 released

I must admit, I am a little late with these “news”. But in case you did not realize it yet: NagVis 1.6 has been released as new stable release of NagVis. It replaces the version 1.5.10.


Photo autocopy from SD card

Copying photos from the SD card of my camera is a very common task. And always the same pattern. Take the card out of the camera, put it into the sd card reader, wait for the automount popup, browse to the picture directory, select all images, cut all images, browse to my photo directory, create a new folder, name the folder, move to the folder, paste the pictures into it, wait for the finished job.

Sound boring? Sure, it is!

Now, how to make this one a little smarter? Maybe scripting some of those steps is a cool idea. So how would this work in a perfect world?


Fixing “show desktop” shortcut (hotkey) in Ubuntu 11.10.

After updating my ubuntu 10.04 installation I had to recognize that the shortcut for minimizing all windows to show the desktop has changed (It seem to be + + d after update). I don’t like that combination + d would be way better. So how to fix this?

Solving this is a real quickie:

Launch gconf-editor and browse to /apps/metacity/global_keybindings

Change the value of show_desktop from d to d.

Press enter, close gconf-editor and enjoy the results of pressing d ;-).

Filed under: Open Source

Ubuntu 11.10: A lot of problems …

I decided to update my notebook from Ubuntu 11.04 to the newest release Ubuntu 11.10. I did not expect any big problems since I had no real problem with the last updates. But it was different this time. The update went smooth, some questions about changed config files but no real problems.

One positive point: The login screen looks nice.

After the first login:

WTF, Unity desktop manager only? grml! I don’t like the unity stuff. I just want to have my console, browser, mail program open and switch between them fast. I don’t need that blingblinghoverfadesmoothscrollpop stuff. I tried the gnome fallback mode but it seems like a real fallback mode which appears like a stripped of version of the old gnome desktop. Also not very comfortable. So I’ll try unity now for some days but will switch to another window manager, maybe xfce.

The second and more annoying thing: When connecting to my home wlan the router crashes and refuses connections on WLAN/LAN. Yay! The new denial of service tool: Ubuntu 11.10. I have a buffalo router with OpenWRT firmware installed. The router is capable of broadband wifi (N). This seems to be a part of the problem. After disabling the N features in the intel driver of my lenovo w500 (03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300) it works as expected. For details take a look here.

Why are the folder (sort, threading, …) settings of my thunderbird gone? I have no clue.

That is the score after three hours using the new version …

Filed under: Open Source

Livestatus Slave 1.1 released

After some feedback from users I made several changes to the livestatus slave code and released it as version 1.1.

Here is the list of changes:

  • FIX: Fixed config typo for TCP sockets
  • Query types and tables are now configurable using the config option queryTypes and queryTables
  • Added optional support for JSONP requests
  • It is now possible to run live.php from the command line as follows: live.php "GET services\n"

It can be downloaded here for directly from the git.

Filed under: Nagios

Nagios is dead? No, it is not!

There is not much progress in Nagios development – this is visible since a couple of years. This is what it looks like when checking out the Nagios Core sources, mailing lists and so on. This could lead us to the question: Is there something happening under the surface which we don’t see? Maybe one day a brand new Nagios Core 4.x is released as big bang and from one day to the other everything changes? Well, I don’t think so.

But why do we care about the development of the Nagios Core so much? Do we really miss important features there? Maybe we have the wrong view on these things.


Simply amazing: Running Linux in JavaScript

I am really, really, really impressed: JSLinux