How-To

Fixing Dropbox’s Camera Upload Organization Mess – EXIFmover

First, I have to note that I love Dropbox. I’ve been using the service almost since the beginning to synchronize my work across multiple desktops, notebooks, and mobile devices. There’s really no alternative at this point that rivals Dropbox in my mind.

That said, the state of their camera upload organization is abysmal. This is a recent new emphasis for Dropbox, automatically uploading all the photos taken on a mobile device or onboard an SD card inserted into a notebook. The problem is that all of these photos are dumped into the same /Dropbox/Camera Uploads folder, so you end up with a huge list of photos. That’s not so bad, except that in my case I’m dealing with sample photos taken on multiple devices for testing results in a completely unwieldy directory with so many files that it often causes Finder on OS X or Explorer on Windows to stall for seconds upon opening.

Screen Shot 2012-12-15 at 3.06.23 PMThis isn’t that well organized…

I’ve pleaded my case for a simple feature to be added – per device folder creation. This didn’t make it into the last update, and about a week or so ago I wrote a script to do exactly that. It parses the EXIF data, and sorts photos into a folder with name /[type]+[model]. The end result looks like this:

Screen Shot 2012-12-15 at 2.28.35 PMFinally, some per-device sorting

It’s called EXIFmover and I stuck it up on Github as a Gist since it’s pretty simple. I use Python 2.7 personally, so that’s what this is tailored for. I’ve tested on OS X, Windows, and Ubuntu. Python’s OS interface is fairly platform agnostic. The one prerequesite is EXIF-Py for parsing the EXIF metadata from photos, which can be obtained from that project’s Github page.

Stick both EXIFmover.py and EXIF.py in the Camera Uploads directory, and run. As more files are uploaded, this can be run again and photos will be sorted once more.

Finally, some organization.

How to Switch from Bars to Numerical Signal Level on iOS without Jailbreaking

I’ve made a habit of only ever looking at signal level in numerics on iOS since the iPhone 3GS days. This has paid off a few times in the past (notably the iPhone 4 antenna situation) but in general just gives a better perspective for network conditions. I regularly post screenshots with both cellular and WiFi in numerics instead of the default bars, and on iOS this is pretty simple to make happen, at least to the cellular signal level indicator, without jailbreaking.

To do this the basic workflow is to enter FieldTest.app, then force quit the application. When FieldTest launches, it changes a .plist file for springboard which controls whether numerics are being shown for cellular signal. This is exactly the file that SBSettings tweaks if you’re toggling “Numeric GSM” and “Numeric WiFi.” I should note that these settings also stay around across iOS restores. Anyhow a lot of people have been asking on Twitter lately for some reason how to make this happen, so I thought I’d write it up.

  
Dialer for FieldTest (left), FieldTest.app (right)

Anyhow to show cellular signal without using SBSettings:

  1. Launch FieldTest.app by going into the dialer and dialing *3001#12345#*
  2. Hold down standby/lock like you’re going to turn the phone off
  3. Release standby/lock after the power off slider appears, then hold home (this is force quit on iOS – it’s impressive so few people know it)
  4. Boom, you have numerics instead of bars

This can now be tapped to switch back and forth. Launching FieldTest again and quitting will restore the file however, so every time you quit this will have to be a force quit to preserve the setting without jailbreaking or restoring an iOS backup with the plist file set how you want it.

I should also note that on LTE this number is RSRP (Reference Signal Received Power) in units of dBm. On WCDMA this is RSCP (Received Signal Code Power) in dBm, and on CDMA2000 1x/EVDO this is RSSI I believe (or EC, I haven’t ever carried a CDMA2000 iPhone for an apreciable amount of time). On WCDMA and 1x/EVDO values between -50 and -113 dBm are typical, with -50 being at cell center and -113 dBm being at cell edge. On LTE because the iDevice is showing RSRP, values between -75 and -120 dBm are typical, with RSRP showing ~20 dB lower than the analogous RSCP/WCDMA-land signal if you’re trying to compare.

Update/Note: In iOS 7 the signal bars were changed to dots, but the trick still works. Although switching into numerals by force quitting Field Test still works, sometimes it takes a few tries before it works. I’m not sure why that is, but keep force quitting or try quitting and coming back and force quitting again and eventually you’ll switch over.

Setting SSD instead of HDD Icon on OS X

It’s been an awful long time since I’ve blogged anything on my personal site. I’ve been working hard and doing lots of smartphone reviews for AnandTech lately, and just haven’t found the time. That said, I do have a rather substantial backlog of ideas that I’ve been meaning to post. The first and quickest of which is how I swapped my HDD icon on OS X 10.6.4 Snow Leopard on my MacBook Pro from the rather generic HDD icon to a more fitting SSD icon.

To begin with, I acquired an OCZ Vertex 2 120 GB SSD and installed it in my 2010 MacBook Pro. I had been waiting for a while to do this move, particularly because I find the optical drive of marginal use and decided to try an OptiBay adapter which would let me use the optical drive’s space as an other 2.5″ bay for whatever I wanted. More on that later, but I moved the MBP’s rather generic 5400 RPM 500 GB traditional platter drive into this optibay space, and stuck the 120 GB Vertex 2 where the HDD was.

Just get to how you did it already!

Cloning the drive onto the SSD was easy, everything works fine, but I was left with that generic and somewhat ugly mechanical hard disk drive icon that OS X has stuck with since forever.

First, I set out to create an appropriate SSD icon. Getting a Vertex 2 image was pretty simple, and I cleaned it up, cropped it, and added the appropriate clipping mask/alpha channel (depending what name suits you) for transparency. It’s a nice big PNG here:

Vertex 2 SSD by OCZ

The next step was changing the icon. It used to be trivially simple in OS 9 to do this, but things got a bit more complicated in OS X. To do so, I found a program named (aptly enough) “Set Icon” you can get here.

Set Icon for OS X

Pretty straightforward. Select the drive, drag an icon in, and click set icon.

Now you’ll have something like this:

Mac SSD

Looks pretty good alongside the old icon, if I may say so myself: Pretty simple, right? Enjoy!

App Store: My Must-Have Applications

iPhone Apps

The other day, one of my Twitter followers asked if I could post a list of iPhone applications I have installed that are useful. Right now, there are quite a few (145 icons by my count).  I’ll share a gallery of all of them, and post a list with links to the ones I really like or use a lot.

It’s a definite goal to reach the installed application limit, and admittedly the organization of just a bunch of tiles on a grid is already stretched thin. I originally did a better job organizing applications by page such that similar tasks or groups were all consolidated. For example, games are all on one page, utilities are on another, e.t.c., but it’s fallen apart lately.

The irony is that there isn’t an app you can install that will tell you what other apps are installed because of sandboxing reasons and App Store restrictions. Oh, Apple…

Favorite Apps

Some of my favorite and most used applications are:

  • Speedtest.net – This is the iPhone version of Ookla’s speedtest.net. It used to be absolutely positively horrible. I mean not just totally false – but boldfaced staring you in the face wrong. They’ve improved it a ton in recent updates, and it now supports exporting to CSV as I’ve mentioned before, including all the geospatial, test results, and other relevant data. Makes analysis possible for end users, not just them.
  • Xtreme Speedtest – Before Ookla got off their collective arses and made the speedtest.net application work, this was my favorite. I’m ashamed to admit I ran it as much as I did. Lately there haven’t been any updates or any love for even the paid “pro” version.
  • Jaadu RDP – Hands down the best remote desktop application. It’s also the most expensive at $24.99, which is annoying, but it truly does work the best. Integration is just extremely smooth and well executed. There’s also Jaadu VNC.
  • Mark the Spot – This should come preinstalled on every single iPhone. If you’re on AT&T, this is your best friend. It’s both a way to report bad connectivity and vent when coverage sucks too.
  • BeeJive IM – All around best IM application. It was one of the first to really leverage push notifications well, and keeps you logged in for as long as you’d like. It’s a brave new world being logged into IM on the phone all the time, but if you want it, this is what’s awesome.
  • Gass Cubby – Keeps track of gas mileage. It does an awesome job, and is fast and easy enough that I do it every time. It even syncs back up to the cloud for backups and storage, or if you have multiple drivers on a car. The graphical visualization and ability to correlate fuel economy changes with service is what really makes it stand out.
  • iStat – Although the real beauty of this application is that it ties into the dashboard widget and server daemons of its namesake, it also works great as a simple resource monitor and informational view. There’s more info about everything here.
  • SpaceTime – This is the absolute best computer algebra system for the iPhone. It’s that simple. There’s 3D plotting, derivatives, integrals, and just about everything else you can get from a Ti-89. I still like my 89, but this is the next best thing.
  • Pi Cubed – Another really good mathematical tool, this one finally leverages the full capacitive touch screen of the platform. There isn’t a virtual keyboard or buttons, but rather a more intuitive interface with pretty print that’s better. I really like that it can export to PDF and LaTeX dynamically.
  • iSynth and Seadragon - These are both awesome Microsoft Research Labs applications that exist on the iPhone. The former is a photosynth viewer created by a software intern as an independent project, and it works surprisingly well. The Seadragon viewer lacks the photosynth code and just displays images using the same sort of algorithm.
  • iRa Pro and IP Vision – If you have a network camera that has MJPEG streaming outputs, these should be on your phone. No excuses. iRa Pro is an $899 application (last I checked, the most expensive in the App Store) but delivers absolutely unparalleled integration with the big enterprise camera setups including PTZ and up to 6 camera streams at a time. If you don’t have a fancy enterprise setup, IP Vision lets you view one MJPEG stream and 2 stills at a time, which is totally adequate for most everything. It’s what I end up using most of the time, and is considerably cheaper at $7.99 – there’s also a more expensive version that works with PTZ cameras.
  • Pocket Universe – This was the first augmented reality application, and for its purpose, the implementation is superb. It’s designed to be an aide for amateur astronomers trying to find a particular celestial body of note. It uses compass and accelerometer data to point you in the right direction, toward finding it.
  • JotNot – It’s a document scanner using your iPhone’s camera. The beauty here is that it removes the distortion based on edge detection, works for large documents, posters, books, and other rectangular, er, media. The other awesome part is that it tightly integrates for output over email, evernnote, WebDAV/iDisk, Google Docs, Dropbox, and Box.net. Either PDF or JPEG output with optional OCR to boot! I use this one a lot.

There are a ton of others that are installed, but these are the ones that really stick out to me as being relatively undiscovered. If you’ve got others you think are useful or related, I’d love to hear about ‘em in the comments section!

My Notetaking Workflow

I didn’t have much time this year to follow TED (In fact, when I first sat down to write this, it was still going on). To be honest, I usually watch the videos a few months afterward, once they’re all finally uploaded and the hype has died down. It’s easy to get caught up in how much certain talks are plugged compared to others, especially with how much live information leaks out over twitter.

But I did break that trend this year a bit. I noticed an intriguing project by Robert Scoble on a blog post of his involving taking photos of notes by the attendees and posting them to flickr. Intrigued, I expected to be wowed by the different creative and thoughtful methods employed which I could use myself for note-taking.

Imagine my disappointment, then, when what I saw that most attendees were either using their iPhones or BlackBerries, scraps of paper, nonstandard spiral bound notebooks, or just generally chaotic methods for taking notes. I mean, aside from the now-famous mind-mapping note girl (photo here; I can’t look at it again because it makes my brain hurt and my teeth start gnashing), there really wasn’t anything TED-level-inspiring.

Numerical Breakdown

Let’s just break it down for a second:

  • 34 pictures in the set
    • Mobile devices: 9 – 26.5%
      • iPhones: 7 – 20.6%
      • BlackBerry: 2 – 5.9%
    • Paper: 25 – 73.5%
      • Notebooks (spiral or bound): 14 – 41.2%
      • Mini Notebooks (or similarly sized): 6 – 17.6%
      • Program/Scraps: 4 – 14.7%
      • PowerPoint Handouts (Bill Gates): 1 – 2.9%

Generally, I abhor excel plots, but this does a good job communicating my point:

Notes Breakdown

But that’s not all; of the iPhone note photos, virtually every single one used the built-in notes application. Yeah, the notes application that ships with the iPhone which lacks just about everything imaginable.

A typical AT&T "Mark The Spot" report

No Evernote love? No Google Documents love? That’s certainly surprising. Yet these attendees consider themselves shakers and movers? Definitely avant-garde? Perhaps ahead of the curve at adoption of new tech? Sorry, virtually every one of you was thoroughly beaten by mind-map girl entirely by default, entirely because of her uniqueness factor. Even more surprising, the journalists in the photo set aren’t even using Steno pads.

With the exception of Bill Gates (who obviously is using PowerPoint handouts for his presentation), there’s really no excuse.

Granted, this could entirely just be bad sampling on Scoble’s part. Whatever the case, it’s a unique opportunity to segue into how much I love the way I take notes.

OneNote – The best kept secret for organizing everything

Ok, those words aren’t entirely my own, but they’re the truth. Microsoft OneNote 2007 (and its predecessor) aren’t just about notes, they’re about collecting, organizing, searching, and making accessible just about anything and everything. You don’t need a tablet, and it isn’t just about text. I think it’s pretty fair to say that OneNote is almost the best kept secret and most undiscovered part of Office 2007.

Relatively Typical Notes View

My freshman year of college, I decided that I wanted to try using it for all of my notes. At the time, I was intrigued by the notion of using a Samsung Q1 Ultra V, a UMPC, due to its tiny form factor and long battery life. That worked, but I’ve since moved on to a Latitude XT in favor of its active digitizer and capacitive multitouch screen. Regardless, I’ve used OneNote for virtually all my notes since, and it has numerous advantages over paper:

Search Results

Search Results

  1. My notes are searchable, entirely. Not just text in its native form either, but handwritten text from the tablet, images (it searches the images), and audio.
  2. I don’t have to carry around spiral bound notebooks that are heavy, or waste money on dead trees (hey, this is one aspect of my life that actually is green).
  3. I can annotate and take notes directly atop PDFs, PowerPoints, or whatever materials are being studied without having to print them beforehand. This is extremely useful as I can get anything into notes by printing it to OneNote.
  4. My notes can be (and are) backed up regularly. That’s something you can’t really do with paper notes, short of making copies or scanning every day.
  5. I can keep every year’s worth of notes in one place. Obviously, that’s a ton of stuff 3 years in. I think you’d be hard pressed to carry around your spiral bound notebooks every day.
  6. I can organize with sections, tabs, notebooks, and pages. The analogues to a notebook are obvious, but there are other things as well that make a lot more sense in the context of digital notes.
  7. Something which always comes in handy is being able to instantly send my notes to other people; I can make PDFs of pages, sections, or entire notebooks.
  8. Everything lives in one place: text notes, powerpoints, images, clips of webpages, even file.

I honestly can’t see how it’d be possible to take notes electronically without OneNote at this point. Granted, there are a lot of other alternatives, but I find that they either have gamestopping flaws or are otherwise unwieldy:

  • Microsoft Word
    • I see this one a lot in classes, and don’t even know where to start. Word is great as a word processing tool, but that’s about all. Sure, you can take notes, but they won’t be searchable (which is a huge drawback for me), and ultimately you’re constrained by this page-by-page model that lies at its core. Combining graphics with notes is possible, but hard. OneNote is almost like Word without pages.
    • How the heck are you supposed to take equations down quickly in Word? If you’ve used the equation editor, you know what a lesson in frustration it is.
  • Google Docs
    • I think using google documents makes a lot of sense, especially given the online nature, but it seems just as difficult to manage with lots of media. Of course, the fact that you can access it anywhere is a huge plus.
  • \LaTeXe/LyX
    • A while back on Slashdot I read a great article I could relate to about taking notes in class for science and engineering. It discussed/asked what the optimal computerized note-taking suite was given an emphasis on entering equations. Of course, \LaTeXe came up, along with its GUI-wrapped similar cousin LyX. I’m a big big fan of \LaTeXe, especially for documents and other things, but I can’t see it being practical or fast enough for taking notes every day. Granted, there are people out there (like some of my crazier friends) that are faster at typing the equations than writing them, but I find myself being able to write faster.
    • You run into the same problems that Word has here; you’re stuck managing files for each set of notes.

I’ve been meaning to try Evernote, and have heard great things about integration across virtually every platform. It seems like the way to go, and if I’d definitely like to try it out.

I guess the point that I’m trying to make is that there are so many better solutions than just using pen and paper or the default notes application that ships with most smartphones. Even though those are what you might grab for at first, you’re setting yourself up to be locked into two methods that leave much to be desired.

Zoneminder MJPEG and Shared Memory Settings

If you’ve read my big post on the Zoneminder configuration I have at home, you’ll notice that I favored capture of JPEG stills over using MJPEG during initial configuration.

At the time, the reason was simple; I couldn’t make MJPEG work. I’ve now succeed in doing so, and understand why it didn’t work the first time.

Memory Settings

I remembered reading something in the Zoneminder documentation about a shared memory setting resulting in capture at higher resolutions failing. Originally, when I first encountered the problem I decided that it was simply me getting something wrong with the path to the .mjpeg streams on the cameras, since I was more familiar with capture of jpeg stills from prior scripting.

However, I stumbled across some documentation here from another tinkerer, which also pointed to the memory sharing issue.

The problem is that the buffer of frames (usually between 50 and 100 for the camera) must be contained in memory for processing. If the size of the image:

buffer\, size\times image\, width\times image\, height\times3_{for\,24\, bits}+overhead

Exceeds this shared memory maximum, you’ll run into errors or see the camera status go to yellow/orange instead of green. (It can get pretty confusing trying to troubleshoot based on those status colors unless you’re checking the logs… /doh)

In fact, the problem I was seeing was likely directly as a result of the large capture image size of my Axis 207mW, as they cite it directly:

Note that with Megapixel cameras like the Axis 207mw becoming cheaper and more attractive, the above memory settings are not adequate. To get Zoneminder working with a full 1280×1024 resolution camera in full colour, increase 134217728 to, for example, 268424446

/facepalm. I really wish I had come across this the first time around. Either way, you’re going to ultimately run into this problem with either higher framerate connections, color, or higher resolutions.

I followed the tips, here, but doubled them since the machine I’m running ZM has a pretty good chunk of memory available.

The process is simple. You’re going to have to edit /etc/sysctl.conf to include the following somewhere:

# Memory modifications for ZoneMinder (kernel.shmall = 32 MB, kernel.shmmax = 512 MB)
kernel.shmall = 33554432
kernel.shmmax = 536870912

Now, apply the settings with

sysctl -p

Which forces a reload of that file. Next, you can check that the memory parameters have been changed:

brian@brian-desktop:~$ cat /proc/sys/kernel/shmall
33554432
brian@brian-desktop:~$ cat /proc/sys/kernel/shmmax
536870912

Which is successful. You can also check it with ipcs -l. Now, reboot ZoneMinder and you shouldn’t have any problems.

Motion JPEG Time!

Having made these changes, I was ready to finally explore whether MJPEG works! I went ahead and decided to use the MJPEG streams from my two respective types of cameras in place of the static video links. These are:

Linksys WVC54GCA: http://YOURIPADDY/img/video.mjpeg

Axis 207mW: http://YOURIPADDY/axis-cgi/mjpg/video.cgi?resolution=640×480&clock=0&date=0&text=0

I also discovered (by reading the manual) that there’s a handy utility on the Axis config page (under Live Video Config -> HTML Examples -> Motion JPEG) which generates the proper URL based on a handy configuration tool where you can select size, compression, and other options:

Handy config page

The idle load on the system has increased, as expected, but that’s partly from me raising the FPS limit to 10 which seems reasonable, and enabling continual recording with motion detection (mocord).

Nice, 10 FPS! (still a gaping hole though...)

I’m making a lot of tweaks as I get ready to transition everything onto a VM on a faster computer with much more disk space (on the order of 8 TB). If you’re interested in reading more about the Linux kernel shared memory settings, I found some good documentation:

  • IBM/RedHat settings: here
  • Configuring shared memory: here

Photosynthing everything

These past couple of days, I’ve finally gotten some time to work on the tremendous backlog of photos that I have sitting around from a number of trips. Among those pictures are sets of photos in the hundreds destined for photosynth. A number of my friends have expressed interest in what the software is, what it does, how it works, and how to take photos best suited for processing. I think now is a great opportunity to go over the basics.

What Photosynth Does

First of all, what Photosynth does is create a 3D point cloud model/representation of an object or scene from a set of photos. Depending on the scene complexity, the number of photos might be in the tens, or hundreds for sufficiently complicated scenes. It all depends on the model and how much time you have on your hands.

Perhaps the best way to explain it, is to see it. The following is a synth of the Pantheon that I recently finished processing, constructed from photos taken by my brother and I from a D80 and D90:

How it does it

The software uses feature extraction to identify textures in parts of each image that are similar, then tries to fit each corresponding from each image together to create a perspective-correct view. The process is extremely computationally intensive, but only needs to be done on the initial set of images to determine position and location. The beauty, of course, is that this process requires no human input for reconstructing the scene; it’s entirely computationally derived.

I won’t claim to be the most qualified to talk about it, but it does use feature extraction and some fancy fitting to work. An important note is that the software works based on unique features in texture, not necessarily on structure. This is why synths with lots of unique patterns turn out extremely well, while others don’t.

Creation

Creating the actual Synth is actually the easiest step; just create an account, install the software, add your photos, and go.

The real work in that process is creating proper tags, descriptions, and then adding geotagging data from photos, or later on in the web interface. Doing so is a great way to get your synth recognized.

How to take the best shots

There’s a great how-to on the official Photosynth website that goes over how to take pictures optimally, but I’d like to share some of my own.

If I’m taking a photo of a single object, something like this column, for example, I’ll try to stay equal distance away from the object, and take photos in steady progression around the subject.

The important thing to keep in mind is that although Photosynth can extrapolate the point cloud from features, it still cannot extrapolate images that you haven’t given it. Simply put, if you want to get the nice scrubber bar to circle around an object, you’ll need to take the requisite photos to make it. I find that pacing steadily around while taking photos at regular intervals is the best way.

  1. Take photos of subjects from a variety of angles. If you can, from every angle possible in an equal manner.
  2. Take photos from a single perspective pointing in multiple directions. I find that spinning around taking photos from each corner of the room works marvelously well; even though you look slightly special in the process.
  3. The most important thing to keep in mind is that quantity is generally on your side; so long as there is variety in the shots, but overlap as well.
  4. Choose subjects that have a variety of textures and features. Things like the Sistine Chapel synth really well because unique texture is everywhere, while cars generally don’t because of their solid color.
  5. Take wide angle shots of the entire room from the four corners first, then one from the center. Afterwards, take photos of objects/features close up. These are things that people will want to focus on when viewing later; a good example are pictures on the wall in a museum or specific fresco sections on a large wall.

Shameless plug

Some of my favorite Photosynth creations are:

  • Piazza San Marco: Link
  • Sistine Chapel: Link
  • Artemision Bronze: Link
  • Salpointe Graduation 2009: Link
  • Replica of Equestrian Statue of Marcus Aurelius: Link
  • Library of Celsus: Link
  • Pantheon: Link

For equal comparison, here are some that didn’t turn out so well:

  • Parthenon: Link
  • Olympic Grounds: Link
  • Port of Dubrovnik: Link

Scanning Books – My Take

Why Scan Books?

With the prevalence of eBook readers like the Nook, Kindle, Spring Design Alex and others, comes the necessity of building and maintaining a vast digital library. There are more resources online than one can easily list for both purchasing (and downloading) books in a suite of electronic formats, from PDF to DJVU, but what if you already own a book of the traditional dead-tree sort? What if you aren’t willing to purchase it again just for the convenience and ease of reading it on your brand new eBook reader?

Scanning becomes your only option.

I’ll be honest, the process isn’t easy, quick or glamorous. But it beats spending a day craning over your flatbed scanner or cutting the spine out of your expensive book to feed it through an equally expensive loose-leaf scanner (speaking of which, what the heck is up with how expensive they are?!). If the book is sufficiently expensive, it becomes an economical prospect quickly given the few hours required from start to finish.

I’m not going to address the legal/ethical/moral considerations. You could argue that making a PDF copy for yourself constitutes Fair Use, but the law being what it is, who the heck knows? Regardless, just exercise some moral introspection and decide for yourself.

Equipment

  • A relatively decent Digital SLR with wide to normal focal length lens
  • Large sheet of black construction paper
  • Tripod/Monopod and a way to hold the camera
  • Snapter or other image processing software
  • Adobe Acrobat/other PDF creating utility
  • 2-4 hours of your time, depending on the book complexity

The specific equipment I use is:

  1. Nikon D80 with Tamron 17-50 f/2.8 lens
  2. Nikon SB600 flash (optional)
  3. Nikon remote shutter release (IR)
  4. Large piece of black construction paper from Michael’s
  5. Monopod, table, and a copy of my CRC Handbook (more on that weird combo later)
  6. Snapter for processing images
  7. Adobe Acrobat 9 Pro for making PDF and OCR

Software

I’ve already mentioned Snapter twice, and although they’re commercial software (with a very generous 15 day free trial that gives you all the functionality of the real book), don’t let that fool you. I’ve had a lot of success with their software just because of how easy and functional it’s been in my experience. So much so that I went ahead and got the paid version.

That said, there are a few open source alternatives that do a pretty good job and are worth mentioning:

Scan Tailor is pretty good, has a nice GUI, and is very active. Unpaper doesn’t have a GUI but offers a lot for a command line tool. There’s always the advantage both OSS solutions offer that you can either code/propose functionality changes in the software itself with the active developers.

Another relevant article with tips is from /. , which posted ironically the week after I had already embarked on and discovered the ins and outs of scanning with a digital camera myself.

Setup

My setup is simple: I mount the camera on the monopod, stick it on the table, and balance it there with my trusty CRC handbook and some other heavy books.

You might be wondering why I didn’t just use a tripod. The reason is that it’s a much more challenging prospect to carefully both tilt the tripod and balance it so the camera is completely perpendicular to the book’s surface. For the best photo quality, one needs the book to be as close to coplanar with the camera sensor as possible. It makes sense, otherwise we’ll have a more challenging time getting the book totally in focus (depth of field will come into play), and have a harder time flattening the book in software.

I generally tape the black paper down to the floor, snap photos of the cover and back cover, and then tape those down as well. More on positioning later.

The whole thing looks like the following:

Doing this in my office

I have the flash set to bounce from the ceiling, just because in practice this yields the most readable photos. I also use all the light I can from the room itself.

A difficult consideration is that sometimes the print/copy itself has glare. This seems a lot more common with newer books than older ones; it’s almost like the print has a layer of varnish atop it. Just make sure you preview a few images and can actually read the copy.

Positioning the book is the tricky part; it’s difficult to balance between filling the frame with the book (so you have good resolution), and leaving enough space at the edges so that your software can do edge detection. Leave too little space around, and you’ll have a nightmarish time trying to field flatten. Leave too much, and you’ll be throwing away a ton of your image. Even worse, if you don’t tape the book down, it will gradually creep out of the frame.

Another big consideration is rotation. I’ve discovered that Snapter doesn’t really account that well for material that has even subtle rotation. You end up with slight skew in the resulting images. It isn’t a big problem, but rotation will immediately cause you headaches.

I usually go for something like this:

Pretty good for taken with my 3GS

You could zoom in a bit more in this case if you wanted; in practice you’ll discover for yourself what works best.

I set the camera to use a relatively big F/# (in this case F/5.6) so there’s as much depth of field as possible. You want the whole book in focus.

Now just snap away

This is the grueling part, capture images of every page. Snag a friend or something as having two people makes this process go much faster. One can turn the page and crease stubborn ones into place, and the other can trigger the shutter with the remote and make sure the book isn’t creeping out of the frame.

I find this can take anywhere between a half hour to much longer, depending on how much trouble the book gives you. The most challenging parts are the very beginning and the end. At these points, the pages have the most curve to them, sometimes sticking up. This is where sometimes creasing them down or using some tape on the stubborn ones can make or break your day.

Eventually, you’ll have a directory full of images somewhere you need processed.

Processing

Snapter-ing?

At this point, you can use whatever tool suits your fancy, but if you’re using Snapter, read on.

Click Book, grab all your photos, and go make yourself a drink as you wait for it to do initial edge detection and processing on images. Nothing is being changed, it’s just generating the initial traces around the book it finds.

After this is done comes the only other bothersome part. It’s very worthwhile to manually go through each page and make sure you’re happy with the edge detection. Frequently, pages that have black or dark color at the edge cause headaches. Drag the handles around until they match closer. This can be grueling, but it’s important.

Click Input, change the background color to black (since we’re using a black piece of paper, or at least I did). Under Output, I also generally turn cropping each page off since I’d rather deal with a spread. Grayscale output will save on space later, and I keep the DPI the same since I’ll compress and downsample later in Acrobat. Now, you can click process and have yourself another drink.

Processing Images

After this is done, you can preview the results on the right. If everything is right, click Save and wait a little longer.

Saving flattened images

Now you should have a directory full of images waiting to be made into a PDF.

Adobe Acrobat

You can use whatever you’d like to make the PDF from the resulting JPEGs, however, I’ve had luck just using Acrobat.

Click Create -> Merge Files into a Single PDF, and then grab all those images you have.

Combine them, and you should now have a huge PDF. Save it, but you aren’t done yet. At this point, I generally take a look at the PDF Optimizer under the Advanced tab, and click Audit Space Usage. Yeah, it should be pretty huge.

I cheated; this is actually after compression

If you absolutely need color, just skip this. If your book is black and white, converting is going to save you a ton of space.

To convert pages to grayscale, under Advanced click Print Production -> Convert Colors. Check “Convert Colors to Output Intent” and select “Gray Gamma 1.8.” I usually then exclude the front and back covers from the page range, unless you don’t care about that pretty color you’ll be missing out on.

This will take a while...

This process also will take some time. Adobe is multithreaded, but still doesn’t use all my 8 logical cores on my i7 920. Just be patient.

After this finishes, you should now see a dramatic difference under the space audit report for Images. There might be a lot of document overhead, however. Don’t worry, this is normal.

At this point, it usually makes the most sense to do some OCR if you want, just to make the document searchable. Document -> OCR Text Recognition -> Recognize Text Using OCR does the trick.

Adobe's OCR works quite well

Click Edit and select Searchable Image (Exact). This won’t resize your images or do compression; we’ll do that later. Now, wait a long time while it consumes CPU cycles and hopefully makes your document so much more powerful and useful.

After this finishes, you’re ready to do some compression and hopefully make your document small enough to not be an embarrassment, you storage hog, you. I usually downsample to around 300 DPI, leave monochromatic images alone (since we don’t have any), and opt for JPEG2000. Check everything in the Discard Objects, Discard User Data, and Clean Up tabs.

JPEG 2000 is amazing

Click Ok, and now be prepared to wait the longest you have yet. Even on my rig, this takes an hour or two.

Check the space audit once more, and you should now have a reasonable sized, fully searchable, readable PDF, ready for your enjoyment.

My ZoneMinder Configuration

Why Home Security?

In recent months, home security and monitoring has become a matter of increasing concern across the country. Whether the reason is local downturn due to a spike in crime or just peace of mind, the price and difficulty of setting up an enterprise-level security system at home is lower than ever.

That said, the variety of hardware, open and closed source monitoring software, and configuration options makes it a bit daunting to jump right into. I’ve worked and experimented with a number of configurations and finally settled on one that I think works best (at least for my needs).

Camera Hardware

Camera 1 – Linksys WVC54GCA

Wireless-G Internet Home Monitoring Camera

I originally started out with just one Linksys WVC54GCA. It’s a 640×480, wired/wireless 802.11b/g network camera with built in web server for stills and video, and some simple motion detection and alert functionality. The reason for its choice was simple; price. It’s Linksys’ primary network camera offering, and you can find it as of this writing for $89 at newegg. In addition, there’s a newer camera with 802.11n, the WVC80N.

However, it isn’t perfect. To quote the cons of my newegg review:

Cons: Wireless range isn’t excellent; I have a very powerful wireless AP with a 12 dBi omnidirectional antenna and a 6 dBi directional antenna, and I had to reposition it so the camera could send video back at a decent bitrate (around 2 megabits is where it sits).

An important thing to note is that the latest .24 firmware breaks WPA/WPA2 support. Mine shipped with .24 and I had to downgrade back to .22 for it to work. A bit disappointing, but hopefully future firmware will fix this glaring problem. The linksys forums have the link to a custom built .22 (oddly enough with german language selected by default, but don’t worry, all the menus are still english).

Motion detection isn’t perfect, sometimes false positives will get annoying. I have sensitivity set all the way down and still get a few random videos of nothing going on.

More recently, I discovered that the software (despite being open source and a *nix derivative) locks up after anywhere between 6-24 hours when the camera is connected wirelessly. This is fixable (in a haphazard sort of way) by calling an internal page that reboots the camera every 3 hours through a schedule in my Tomato router:

Tomato Scheduler Screenshot

Thus far, this has proven a robust fix and makes the cameras entirely usable. I’ve notified Linksys and even had a chat online with a higher level tech that passed my findings on to a firmware engineer. They’ve recently released an update which purports to fix stability issues:

Version v1.1.00 build 02, Jun 15, 2008
- Support of Setup Wizard is temporarily disabled to address security issue
- Fix security issues
- Fix Camera stability issues

I have yet to fully test it. As an aside, the cameras are actually embedded x86 inside, sporting an AMD Geode SC1100 processor, 32? MB of SDRAM (2x TSOP marked PSC A2V28S40CTP), and Ralink 802.11b/g/a(?) chipset (RT2561T) as pictured.

AMD Geode SC1100

Camera visible, other SoC

RaLink MiniPCI card

Image quality is a little above average but nothing wonderful due to the relatively tiny plastic fixed focus lens system. Low light sensitivity is ok, but nothing stellar; you still need moonlight or ambient street lighting to get usable results at night. If you don’t mind those caveats, you’ve basically got the beginnings of a very robust (and cheap) network camera.

Night versus day performance

There are a number of relevant pages that are undocumented on the camera itself:

Reboot: http://USER:PASS@ADDRESS/adm/reboot.cgi

MJPEG stream: http://USER:PASS@ADDRESS/img/mjpeg.jpg

JPEG still: http://USER:PASS@ADDRESS/img/snapshot.cgi?size=3 (3- 640×480, 2- 320×240, 1-  160×120)

The options offered in the camera’s internal setup pages aren’t very robust, but offer just enough for you to do almost everything you’d want to.

Image Settings

Camera 2 – AXIS 207W and MW

After acquiring another Linksys camera for myself (and another 3 for the parents), trudging my way through the reboot issue, and reasonable but not stellar image quality, I decided I was ready for something more. Axis seems to have very good support, choice, and performance, in addition to heaps more customization and options for the camera itself. The catch? Price.

I decided to start off with Axis’ cheapest offering, the 207-series of network cameras. I managed to snag an Axis 207MW that had been used just once at a trade show that was being sold as used on eBay, and my dad went ahead and just purchased outright a 207W from Newegg. The distinction between the 207W and MW is that the 207MW has a 1.3 megapixel camera supporting resolutions of up to 1280×1024, whereas the 207W is just 640×480. They’re both 802.11b/g so you can move them throughout the house, and have almost identical setup and configuration pages. Of course, like the Linksys WVC54GCA, there’s optional ethernet support as well. Virtually all the other features are the same between the 207W and 207MW.

Axis 207MW (the 207W looks identical)

As of this writing, the 207MW is $328 at Amazon, and the 207W is $286 at Amazon.

Right off the bat, you can tell this camera is much different. It’s got an actual glass lens system, focusing ring, and a compact form factor with a longer cable. In addition, the antenna is external and swivels and snaps out so you can position it however suits getting the best signal. There are a variety of status LEDs on the back that make troubleshooting limited wireless connectivity simpler. The front clear ring is actually a large lightguide for four LEDs that can be either green or amber depending on the status of the camera. These can be disabled as well.

Day versus night performance on the 207MW

Image quality is also much better on the Axis 207MW than the Linksys. Originally, I had a WVC54GCA mounted where the Axis is now inside the garage. The Axis is both much more stable, and also gets better wireless reception outside in an otherwise difficult to reach trouble spot.

Among other things, the Axis offers many more configuration options within its internal administrative pages, as well as (if you’re interested in running it) many more options for built in motion detection. One of the more important things I’ve come across is the ability to change exposure prioritization so the otherwise very well lit driveway doesn’t come out a homogeneous white from pixels saturating as often. This kind of exposure prioritization can be done on the Axis, but not on the Linksys as shown:

Exposure settings

There are just a wealth of options that really make the Axis shine over the cheaper Linksys if you delve deeper. I could write pages about the differences that the extra nearly $200 makes (if you can afford it). Both cameras offer the ability to upload images and 5-10 second video clips of motion detection events to an FTP share, or attach them to an email. Detailing the differences between the two (and the ultimate shortcomings of both) is another article in and of itself.

At the end of the day, I found motion detection somewhat unreliable on both the Axis and Linksys; either I wound up with far too many motion event video clips or nearly nothing. Even worse, downloading and then watching hundreds if not thousands of false positives a grueling task. If you’re a basic user or just interested in having a camera for temporary purposes while you’re away on a trip, perhaps just the in-camera features are enough. However, if you’re looking for something more robust for a number of permanent cameras with much better motion detection, keep reading. At the end of the day, I use both types of camera just as inputs for ZoneMinder as you’ll see later on.

ZoneMinder Setup

ZoneMinder is a GPL’d, LAMP-based web tool for managing and monitoring virtually every kind of possible video source. Its supported sources span everything from cheap USB Logitech webcams, to network security cameras with built in webservers (like the two I’ve covered), to traditional video sources through a video capture card. Their documentation is a bit overly complicated (you can get to their supported hardware list here), and at the end of the day you’re going to either need to have local linux driver support (and a path to video like you’d expect for a webcam/TV tuner), or a path for JPEG, MJPEG, or another kind of MPEG4 stream.

The aim of ZoneMinder is to do all motion detection, video archiving, and image processing in one centralized place; simplifying use and making it easier to keep track of new events as they happen. Of course, the only downside to this is that all that motion detection and video capture requires a relatively powerful computer. Official documentation claims that even an ancient Pentium II should be able to do motion detection and capture for one camera at 25 FPS.

On the old computer I’ve configured (with a Pentium 4 Northwood 2.8 GHz and 2 GB of RAM), I’ve found that adding an 8 FPS VGA network camera and doing motion detection and capture adds between 10-20% CPU load.

Installation

Luckily ZoneMinder is relatively easy to setup if you’ve ever been near a modern linux distro with aptitude. As I noted earlier, ZoneMinder should ideally be run on a LAMP or similar web server, however, they claim that distro, web server, and SQL database support is actually quite diverse. I performed my installation on a fresh install of Ubuntu 9.10 Karmic by following instructions similar to Linux * Screw’s:

sudo apt-get install zoneminder apache2 php5-mysql libapache2-mod-php5 mysql-server ffmpeg

Once that was finished, the following:

sudo ln -s /etc/zm/apache.conf /etc/apache2/conf.d/zoneminder.conf
sudo
/etc/init.d/apache2 force-reload

http://_YOUR WEBSERVER ADDRESS_/zm/

At this point, I’d encourage you to enable user authentication in Options -> System -> ZM_OPT_USE_AUTH. Ticking this box and saving will enable another tab, Users. I generally configure one  admin for making changes and a less privileged “User” account for simply viewing the cameras and motion detection events as shown:

Users configuration

The remainder of options are largely fine in their defaults; the only major thing that you should be concerned with are the paths if you care about certain disks being used. I’ve recorded almost 800 events so far in VGA resolution and have used up an additional 1% of the meager 80 GB HDD on the system.

Adding Sources

Now, to add some video sources. This is where you really have to either know the path to either JPEGs or a MJPEG stream on the camera.

Click “Add New Monitor” in the bottom right. Now, if you have an Axis or Panasonic network camera (arguably the two de-facto industry standards) there are some presets that are worth checking out that you can use by clicking “Presets” in the top right. Change the source type to Remote (if it isn’t already from using a preset), and name it appropriately. I also usually set the FPS to either 5 or 8; from what I’ve seen, higher really isn’t feasible.

Now click the “Source” tab. This is where if you used a preset, your life is much easier, as the remote host path is already filled in. If not, it’s still simple, you just have to know how to get the relevant data from your source. In my example, I have the following:

Adding Axis network camera

From the above screenshot:

Remote Host Name: user:pass@_PATH TO YOUR CAMERA_

Remote Host Port: 80 (Or something else, if you’ve changed it)

Remote Host Path: /axis-cgi/jpg/image.cgi?resolution=640×480

Note that this is for the Axis 207 series cameras, although in general all the Axis cameras follow the same syntax (nice, isn’t it?). You can’t use whatever resolution you want, however, all the major and obvious choices work. You’ll notice I’ve just used VGA instead of the full 1.3 MP 1280×1024 resolution image from the 207MW. This is because using full resolution does seem to generate too much network traffic for my 802.11g network (despite my best efforts, the garage remains a dead zone thanks to chicken wire stucco construction), and FPS takes a large hit. No doubt that if the connection were wired, higher would be feasible. However, VGA is more than enough for now.

Adding the Linksys sources are just as easy given the paths I outlined previously. I haven’t added any internal sources, personally, however I imagine that configuration is the same if not easier; it requires knowing the path and setting a few additional constraints so you don’t overload your server.

Configuration

Now that you’ve added sources, you can configure their function.

Video source functions

You would think these options would be intuitive, however, they caused a bit of confusion for myself personally due to their shortened names. They are as follows, from the ZM documentation:

  • None – The monitor is currently disabled and no streams can be viewed or events generated.
  • Monitor – The monitor will only stream feeds but no image analysis is done and so no alarms or events will be generated,
  • Modect – or MOtion DEteCTtion. All captured images will be analysed and events generated where motion is detected.
  • Record – In this case continuous events of a fixed length are generated regardless of motion which is analogous to a convention time-lapse video recorder. No motion detection takes place in this mode.
  • Mocord – This is a hybrid of Modect and Record and results in both fixed length events being recorded and also any motion being highlighted within those events.
  • Nodect – or No DEteCTtion. This is a special mode designed to be used with external triggers. In Nodect no motion detection takes place but events are recorded if external triggers require it

In practice, my cameras are generally set to Modect, unless I have an indoor camera with particularly high traffic, in which case Monitor makes more sense since all the motion detection events would be me moving around and about (take it from experience, you see yourself doing some pretty strange things). This is also a nice way of judging how much load each camera adds, as the setting is pretty immediate if you’re watching htop.

With time, you should now have a zoneminder console similar to mine:

ZM Main Console

I’ve noted in particular how offline hosts appear red, while online hosts appear green or orange depending on their function.

Perhaps the only last area of configuration are the zones themselves (finally, the zone in ZoneMinder!) These define the regions of interest, in each video source, that will be used for event detection. Clicking on “1″ under Zones will allow you to modify the default zone for the video source. This is where you’re really given a lot of control far beyond anything in-camera will ever offer from even Axis.  You can add points and create polygons, as well as tweak sensitivity on an interface that looks like this:

Here, I've rejected a region that gives false positives; cars going by on the road.

Returning to the main “Console” view, the rest of the interface itself is relatively self explanatory.

Daily Use/Monitoring

If you’re interested in viewing all of the video sources currently enabled, clicking “Montage” should give a view similar to the following:

Montage view, including author

You’re also given the FPS below each camera, this is also handy. Lower resolution (eg I show a 320×240 source blacked out) cameras are somewhat intelligently tiled as well, using the available space pretty well.

Clicking on any of the events back on the console page should bring up something similar to the following, showing details about all the motion detection events from the given source (or all sources):

Hope nobody studies this in too much detail...

Clicking on any of the event IDs or names pulls up a window where you can review the event, from a few seconds before, to just after. You can also click anywhere below on the scrub bar to jump ahead or back, as expected. It isn’t perfect, but does a surprisingly good job:

A car enters my driveway at night...

Perhaps the coolest is “Timeline” view, a high-level plot of motion detection activity across all cameras overlaid on a timeline. This gives you an at-a-glance overview of whether the same events were being detected across all cameras at the same time, or to quickly pick out what time of day generates the most activity. In this view, mousing over times and activity as demarked by red on the plot refreshes the thumbnail appropriately, as well as with the detection region highlighted in red.

My timeline for all zones

It isn’t always the most useful way to review events, but perhaps one of the more unique. I’ve found it useful for reviewing a few days or weeks at a glance when I’m gone. There’s also certainly a nice pattern that emerges over time, at least for me.

Mobile

ZoneMinder supposedly has a nice mobile view available, however, I’ve had relatively little experience with it and had difficulty enabling it on my iPhone 3GS. Viewing the normal ZM site works fine, however, motion detection playback doesn’t work all the time.

In the meantime, I continue to use IP Vision for monitoring all my MJPEG sources:

IP Vision on the iPhone. Streaming MJPEG!

If you’re interested in me detailing this, just let me know. Setup is again straightforward and merely involves knowing the correct paths and forwarding a few ports in your router. Also, it’s a great way to quickly consume tons of 3G bandwidth!

Conclusions

Setting up a robust, nearly commercial-level reliability home video surveillance system is now easier than ever thanks to the huge variety of video hardware and open source software available. I’ve moved from one single camera with in-camera motion detection sending alert emails with 5 second video clips to a gmail account (which quickly filled to the limit), to a secure and expandable motion detection suite monitoring at times 7 cameras that is accessible virtually anywhere I can get online.

If you’re only interested in home monitoring during a vacation or time away, setting up a system like this might not be the best solution, so long as you’re willing to sift through either an FTP dump full of videos and stills or a gmail account choc-full of videos. However, if you’re serious about having a manageable system with a number of fixed (or PTZ!) cameras that you need constantly monitored, ZoneMinder makes sense and gets the job done. In the future, for serious users, it can even be hosted commercially or simply store the event cache on a network share elsewhere to prevent physical tampering or theft.