From IP Security camers to security video archives

IP cams are really cheap now, as low as $30 if you do not need HD. For indoors, can they can replace window door sensors, PIR motion sensors with much easier installation, operation and maintenance. If the window moves, the motion will trigger the cam to send images to any device connected to the internet, trigger an warning alarm to any smart phone connected to the internet, and or trigger an alarm connected to the cam.

For outdoors, uniformed personal are at their best behaviour when they pass by.  Never have any cold door to door calls since, nor spam materials left at my door. And those persistence vandals went to great lengths to test the system and piss me off when they don't dare to challenge the cams directly.

My advice is to get some cams with local storage, from buffering for a few hours of images to a cyclic long term storage. With cyclic storage, the cam can store the most current X GB of images or the latest 7 days or images. Otherwise the cheaper the cam the more work you need.

Also, do you need HD? As a security deterrent, DVD quality is way better than analog tapes used for decades. HD may capture more faces and license plates at longer distances, but there's no guarantee. The disadvantages is that the complexity increases multiple folds down the line. You need a good wifi signal to sustain the higher bit rate. You need a lot more storage. File transfers and processing are also longer.

Power outage

Interestingly, after a total power outage, everything went up by itself! The cable modem and wifi router is expected. Others include, an old router converted to a wireless ethernet bridge, a pair of powerline ethernet, both to extend the wifi range to outdoors. The cheapest cams themselves are solid, though a bit tricky to setup without manual and support.

Image recording

There is no problem of viewing video from the cams over the internet at 30 fps, but recording them is a total different story. If you record the video or image sequence directly, you need a MS Windows computer and an big external USB drive. MS seems to be open source in China so they will not think of supporting or developing anything else. That is bloody inconvenient with all the power consumption and the sole use of a computer for 24/7.

You can use the cloud but say Amazon charge by operating hours and data going out.

Motion detection

Motion detection is a good way to reduce storage, but ... The record speed is totally different.  The cams can push out a normal video at 30 fps but for motion detection, it can only send out via emails or ftp at about 1 frames per second.

I don't think it's the processing power of the cams but the limitation of the network pipes. Remember that for ftp, the client have to download several files concurrently to be efficient?

I brought the Tinycam Android app hopefully to store faster on a spare phone. I don't know if it's only me, but for motion detection with Tinycam I can get only about 1 fps, and in addition, longer delay than straight from the IP cams. Without motion detection, you can record the normal video or image sequence straight without problems.

In conclusion, if you can afford storage for a few days continuously at 1 fps, you don't need motion detection. For outdoors with moving plants and car pass by, you don't save that much.

Night vision

Also, on-board night vision is pretty good unless you need facial recognition at high priority. But night vision is not enough for motion detection. You can spend $10 for a inferred security light, which surely triggers the on-board motion detector when the lights turn on.

Image transfer

Practically, I have 1 image per second recorded on my device, when motion is detected. Cloud storage is out of the question as you have to pay 24/7. Paying for an ftp service is about the same for a few dollars per month. But you pay forever.

An Android device is good for storing the images, a spare phone, a TV box or a stick computer attached to your TV. If you have a spare phone you can use it as a cam too. It's low power, out of the way, and can be left on 24/7.

I have a stick computer and in theory if I plug a 64 GB flash drive or SDCARD into it, all my problems are solved.

The problem is, processing and retrieving on Android is very limiting. I tried a few FTP server on Android. All are good and reliable. But I think Android changed the security mode since 4+ so unless your phone is rooted, you cannot receive files in a sub folder, and you can't even delete files remotely.

All the FTP servers I tried do not support cyclic deletion. So you have to delete old files manually or via a scheduler. Again I see no good scheduler or just no scheduler if you phone is not rooted. Also because flash drive and SDcard usually use FAT, the number of files in a folder is limited. Anyway, I got errors when the number of files that the server received become huge, while I still a couple of GB spare storage on the phone.

Apps such as Tinycam seems to solve the problems. But instead of multiple cams working in parallel hardware, now the phone has to process all the image detection of all cams. It's not too bad, but I find it lagging.

So, an Android device is only good for buffering the images. You need a computer to retrieve the images for "permanently" storage, processing, organizing and retrieval.

I have a huge number of images in a single folder received by the FTP server. It's highly inconvenient to do anything with it or to organize them and send them to the computer.

So I wrote a script in Android to parse the file names and put them into folders, eg, 2014/09/02/camid. I ended up learning to code my 1st Ruby program and executed in Ruboto, Ruby on Android. Now deleting old files and getting new files to remote places are easy.

I even wrote a Ruby script to get files via ftp so as to automate the whole process. But the problem is, if you remember, ftp request need to be concurrent to be fast. I can write Ruby codes for that but I rather use Filezilla in Ubuntu. It's not too bad. You can download all subfolders. You can set not to download duplicated file names. You can still delete files, which will still be there but with 0 size. These are not features of Filezilla but the limitation of ftp servers on Android.

Image retrieval

My aim is to identify every tiny sneaky trick that my persistent vandals tried on my property. It's not easy if I have only a huge number of jpg files.

On Android you have two decent apps to deal with the huge number of files. Quickpic can display huge numbers of thumb nails and allow you to glide through them with your fingers. It's not difficult to spot changes by your eyes when consecutive images are mostly identical. However there's no page up and down buttons so you will not miss when scrolling. Quickpic also did some dirty tricks like 'cookies', leaving some nasty hidden things on the file system so other apps may crash.

Then I discovered time lapse video. This is exactly when you want when you have a huge number of jpg files. The Lapse it app can handle that number of images fast. But the problem is, you cannot go back in time when you spotted someone just walked across the cam. Sadly, if this works, you don't need a computer and everything could be done in a standalone Android.

Computer processing

I can download directly the folder structure on the Android, ie, 2014/09/02/camid. But still I have the same Ruby script on Android (with slight modifications) that parse each filename and put them in the right folder (strictly speaking, directory). I can sync the files on both devices easily and reliably.

Do you remember that analog security recorders have time stamp? If you can find one IP cam that have the option. The problem is, mass produced chips don't do that. It degrades the image, increase the size, and need extra processing. The time stamp is always on the file anyway.

In addition, my recording is not linear, ie, a burst of images only when motion is detected. This made any manual attempt to get an idea of time difficult.

If you watch a video, you spot something and pause, but how do you find out what that time is?

So I add write a script to add a time stamp to every image, according to the info on the file name. It's a pain to use the time info such as modified time of the file. When you copy, most likely the time stamp is not copied. Android doesn't allow you to move a file across different media, such as internal storage, SDcard and USB drive. The program is 'convert', standard in Linux.

There's no usable time lapse programs on Linux to view the sequences. Many time lapse apps cannot set the duration to below 1 sec. Also there's too much overhead to go to the the correct directly and specify the files. It's extremely difficult to go back and forth to find something in a frame.

Creating a security video

I ended up creating a video from the sequences. That's the last thing I tried because many video editors cam import sequences, but it's way too much overhead. And then you have you record each image into video format, which is slow for an old spare computer.

I tried Openshot, Avidemux, which is pretty much like the old Dub, all of which I made a lot of different types of video. Then it dawned on me that the standard linux command avconv can import sequences as easy as video files.

My luck changed when I remembered mpeg, which is just jpg's stacking together into a video. It's fast to put the images into a video with .mpeg as the video file extension. But actually, jpg is also really the video codec for mpeg so the right convert option is '-codec copy'. The conversion rate is over 100 fps for an old computer, while it is slower than real time (<30 conversion="" format="" fps="" if="" is="" p="" real="" required.="">
One more thing. The importing needs the file be named imageXXX.jpg, where XXX stand for consecutive numbers. That forced me to write more Ruby codes. But at the end, you just need on script to store and sort the files, and another to create a video.