From kyle at xkyle.com Mon Mar 23 13:48:09 2009 From: kyle at xkyle.com (Kyle Anderson) Date: Mon Mar 23 13:48:32 2009 Subject: [ale] Help with aligning images Message-ID: <49C7F589.5020600@xkyle.com> Hello, I'm attempting to use ale as a type of pattern recognizer. I have two images that I need it to align. http://xkyle.com/other/25grey.jpg http://xkyle.com/other/613grey.jpg But I think the pixels may be producing false positives, so I'm also doing it with edge-detected images: http://xkyle.com/other/25.jpg http://xkyle.com/other/613.jpg My command line looks like this: ale --translation --follow --exp-meta-only --exp-extend --trans-save=ale.align 25.jpg 613.jpg ale_out.jpg But I've tried many types of variables that I can understand in --help -ha. Can someone give me some pointers to tell me if this is even possible with ale, and if so, what kinda of alignment options should I be playing with? What I'm most interested in is the percent matched (how good of a match it is). I also don't really care how long it takes. Kyle From zds at iki.fi Tue Mar 24 07:58:17 2009 From: zds at iki.fi (Jari Juslin) Date: Tue Mar 24 07:58:25 2009 Subject: [ale] Help with aligning images In-Reply-To: <49C7F589.5020600@xkyle.com> References: <49C7F589.5020600@xkyle.com> Message-ID: <49C8F509.9050509@iki.fi> Kyle Anderson kirjoitti: > Hello, > I'm attempting to use ale as a type of pattern recognizer. I have two > images that I need it to align. > But I've tried many types of variables that I can understand in --help -ha. > Can someone give me some pointers to tell me if this is even possible > with ale, and if so, what kinda of alignment options should I be playing > with? What I'm most interested in is the percent matched (how good of a > match it is). I also don't really care how long it takes. First, make sure you are using the very latest version of ALE (which is v0.9.0.2, unless I'm mistaken). 0.8 series needed all kinds of options to do work 0.9 does out-of-the-box, and in fact, 0.9.0.2 does better than 0.8 with any set of options.. Second, you are facing quite a challenge there. ALE was never meant to be used for that different images, it's been optimized to combine images taken with the same camera, same lens and same settings, and mostly from the very same spot (except the experimental 3D). ALE has problem with all kind of un-evenness that does not occur in the situations listed above - movements and rotations are ok, but for example slight stretching (what can happen when scanning color negatives) is something ALE just cannot work around, neither can it scale. If I had to work with that heterogenic data than the one you showed, I'd first use programs like Autopano-SIFT-whatever and Hugin to find keypoints, to align the images and to normalize the exposure and then finally let ALE calculate the similarity. Unless you know how to program, or at least how to script, this will be lots of manual work with only moderate chance of success. But the problem sounds interesting, so I wish you good luck with it, -Jari From zds at iki.fi Tue Mar 24 08:10:28 2009 From: zds at iki.fi (Jari Juslin) Date: Tue Mar 24 08:10:35 2009 Subject: [ale] Processing color negative scans with ALE (was: Help with aligning images) In-Reply-To: <49C8F509.9050509@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> Message-ID: <49C8F7E4.8090309@iki.fi> Jari Juslin kirjoitti: > ALE has problem with all kind of un-evenness that does not occur in the > situations listed above - movements and rotations are ok, but for > example slight stretching (what can happen when scanning color > negatives) is something ALE just cannot work around, neither can it scale. BTW, now that I mentioned this: is there any chance of getting support for streched scans? What I was doing was taking archival copies of color negatives. Because I don't wish to ever scan them again, I tried scanning them multiple times and combining then with ALE, to get rid of random scanner noise. This, however, failed, because the scanner stretched the images oh-so-slightly here and there, but enough to cause ghosting in ALE-combined results. The same would naturally occur if the scans were of different frames, which takes us to the more regular ALE territory. And hell yeah, I have indeed tried this and that with ALE.. wonderful program it is. Nowadays I'm most often using it to align images in HDR sets, as HDR combining requires perfect alignment, and ALE can provide just that. -Jari From magnus.ekhall at bredband.net Tue Mar 24 09:35:17 2009 From: magnus.ekhall at bredband.net (Magnus Ekhall) Date: Tue Mar 24 09:35:53 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49C8F7E4.8090309@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> Message-ID: <49C90BC5.4010407@bredband.net> > > And hell yeah, I have indeed tried this and that with ALE.. wonderful > program it is. Nowadays I'm most often using it to align images in HDR > sets, as HDR combining requires perfect alignment, and ALE can provide > just that. > I'm very interested in how to align and crop photos in HDR sets. How do you do that in ALE? /Magnus From zds at iki.fi Tue Mar 24 10:15:16 2009 From: zds at iki.fi (Jari Juslin) Date: Tue Mar 24 10:15:28 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49C90BC5.4010407@bredband.net> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> Message-ID: <49C91524.9050401@iki.fi> Magnus Ekhall kirjoitti: > I'm very interested in how to align and crop photos in HDR sets. How do > you do that in ALE? I am using a Perl script available here: I have also modified and appended it with BASH script to use --threads option and to first make both images equally exposed, but that version is still very much in beta, so it needs you to enter an additional parameter yourself.. The reason for equalizing the exposure is that ALE is bad at aligning images that have different exposure, say 0 and +2 EV. As this is the case always with HDR images I use convert to brighten up the darker image. After that treatment ALE does the aligning very nicely. The Perl script then does the cropping with ImageMagic based on the ALE alignment file. Another way to make HDR with ALE is to shoot lots of images that are all underexposed enough so that no spot is burned and then use ALE to combine these. This is something I am experimenting as we speak. Naturally this does not work on scenes where brightness differences are too large, but it could work better in subjects that are moving all the time (like running river). The thinking behind this is that for a single shot the dynamic range is limited, but if you gather dozens of shots, you can figure out the darker end and restore the information there. -Jari From zds at iki.fi Tue Mar 24 12:44:56 2009 From: zds at iki.fi (Jari Juslin) Date: Tue Mar 24 12:45:08 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49C91524.9050401@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> Message-ID: <49C93838.2030303@iki.fi> Jari Juslin kirjoitti: > Another way to make HDR with ALE is to shoot lots of images that are all > underexposed enough so that no spot is burned and then use ALE to > combine these. This is something I am experimenting as we speak. > Naturally this does not work on scenes where brightness differences are > too large, but it could work better in subjects that are moving all the > time (like running river). Which brings me back to the reason I stopped experimenting with this technique - ALE crashes to assertion if you give it too many images. This far I've been able to work by giving ALE 6 images at max, but with today's data it chokes on five.. Unfortunately I am not in position to debug this, but if anyone is willing to take a look at it, I can give you set of images that crash it on every run.. and here's a stack trace: zds@terra:/zds-data/share/digikuvat/hdr/hdr58/ruutu0$ ~/ale-0.9.0.1/ale --threads 2 --16bpc *.png ../hdr51_ale_ruutu0_0a.png Output file will be '../hdr51_ale_ruutu0_0a.png'. Original Frame: 'IMG_3426.png'. Supplemental Frames: 'IMG_3427.png'***** okay (95.869802% match). 'IMG_3428.png'***** okay (96.594264% match). 'IMG_3429.png'***** okay (96.838361% match). 'IMG_3430.png'***** okay (96.942329% match). 'IMG_3431.png'***** okay (97.053369% match). 'IMG_3432.png'***** okay (97.092394% match). 'IMG_3433.png'***** okay (97.237601% match). 'IMG_3434.png'***** okay (97.245518% match). 'IMG_3435.png'***** okay (97.388432% match). 'IMG_3436.png'***** okay (97.249381% match). 'IMG_3437.png'***** (96.446125% match) | Calculating post-alignment match ale: ui/../d2/trans_multi.h:199: d2::trans_multi& d2::trans_multi::operator=(const d2::trans_multi&): Assertion `spatio_elem_map_r' failed. #0 0xb7f91410 in __kernel_vsyscall () #1 0xb78ee085 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78efa01 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb78e710e in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0x08067e6b in d2::trans_multi::operator= (this=0xb65e02c0, tm=@0x6ac27898) at ui/../d2/trans_multi.h:199 #5 0x08082c7a in trans_multi (this=0xb65e02c0, tm=@0x6ac27898) at ui/../d2/trans_multi.h:209 #6 0x080929de in d2::align::diff_stat_generic::diff_subdomain (args=0x6ac27804) at ui/../d2/align.h:966 #7 0xb7a174fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7999e5e in clone () from /lib/tls/i686/cmov/libc.so.6 From dhilvert at gmail.com Sat Mar 28 06:35:34 2009 From: dhilvert at gmail.com (David Hilvert) Date: Sat Mar 28 06:36:11 2009 Subject: [ale] Processing color negative scans with ALE (was: Help with aligning images) In-Reply-To: <49C8F7E4.8090309@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> Message-ID: <20090328143534.c6e2b3fb.dhilvert@gmail.com> On Tue, 24 Mar 2009 17:10:28 +0200 Jari Juslin wrote: > Jari Juslin kirjoitti: > > ALE has problem with all kind of un-evenness that does not occur in the > > situations listed above - movements and rotations are ok, but for > > example slight stretching (what can happen when scanning color > > negatives) is something ALE just cannot work around, neither can it scale. > > BTW, now that I mentioned this: is there any chance of getting support > for streched scans? Sorry for the delay; I seem to be off the net quite a bit these days. This is probably somewhat obvious, but have you tried the --projective option? From dhilvert at gmail.com Sun Mar 29 01:42:10 2009 From: dhilvert at gmail.com (David Hilvert) Date: Sun Mar 29 05:37:37 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49C93838.2030303@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> Message-ID: <20090329104210.34877e7d.dhilvert@gmail.com> On Tue, 24 Mar 2009 21:44:56 +0200 Jari Juslin wrote: > Jari Juslin kirjoitti: > > Another way to make HDR with ALE is to shoot lots of images that are all > > underexposed enough so that no spot is burned and then use ALE to > > combine these. This is something I am experimenting as we speak. > > Naturally this does not work on scenes where brightness differences are > > too large, but it could work better in subjects that are moving all the > > time (like running river). > > Which brings me back to the reason I stopped experimenting with this > technique - ALE crashes to assertion if you give it too many images. > > This far I've been able to work by giving ALE 6 images at max, but with > today's data it chokes on five.. > > Unfortunately I am not in position to debug this, but if anyone is > willing to take a look at it, I can give you set of images that crash it > on every run.. and here's a stack trace: > > zds@terra:/zds-data/share/digikuvat/hdr/hdr58/ruutu0$ ~/ale-0.9.0.1/ale > --threads 2 --16bpc *.png ../hdr51_ale_ruutu0_0a.png > Output file will be '../hdr51_ale_ruutu0_0a.png'. > Original Frame: > 'IMG_3426.png'. > Supplemental Frames: > 'IMG_3427.png'***** okay (95.869802% match). > 'IMG_3428.png'***** okay (96.594264% match). > 'IMG_3429.png'***** okay (96.838361% match). > 'IMG_3430.png'***** okay (96.942329% match). > 'IMG_3431.png'***** okay (97.053369% match). > 'IMG_3432.png'***** okay (97.092394% match). > 'IMG_3433.png'***** okay (97.237601% match). > 'IMG_3434.png'***** okay (97.245518% match). > 'IMG_3435.png'***** okay (97.388432% match). > 'IMG_3436.png'***** okay (97.249381% match). > 'IMG_3437.png'***** (96.446125% match) | Calculating > post-alignment match ale: > ui/../d2/trans_multi.h:199: d2::trans_multi& > d2::trans_multi::operator=(const d2::trans_multi&): Assertion > `spatio_elem_map_r' failed. > > > #0 0xb7f91410 in __kernel_vsyscall () > #1 0xb78ee085 in raise () from /lib/tls/i686/cmov/libc.so.6 > #2 0xb78efa01 in abort () from /lib/tls/i686/cmov/libc.so.6 > #3 0xb78e710e in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 > #4 0x08067e6b in d2::trans_multi::operator= (this=0xb65e02c0, > tm=@0x6ac27898) at ui/../d2/trans_multi.h:199 > #5 0x08082c7a in trans_multi (this=0xb65e02c0, tm=@0x6ac27898) at > ui/../d2/trans_multi.h:209 > #6 0x080929de in > d2::align::diff_stat_generic::diff_subdomain > (args=0x6ac27804) > at ui/../d2/align.h:966 > #7 0xb7a174fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 > #8 0xb7999e5e in clone () from /lib/tls/i686/cmov/libc.so.6 Does --ips 0 fix this problem? This appears to be an out-of-memory condition resulting from keeping pixelwise multi-alignment maps for every frame, and so should be limited to the 0.9 branch (i.e., the 0.8 branch shouldn't have this problem). I've added a ditz issue (ale-13) for this that should show up in the master branch on repo.or.cz eventually. A workaround patch for 0.9.0.2 follows that reduces the size of the map, which should roughly double the number of inputs processed prior to exhausting memory. (Substituting 'char' for 'short' in the patch should double this number again, but will break on large images.) Other possible fixes and workarounds include (a) modifying the behavior of '--multi none' to not generate maps (which would fix the problem only for '--multi none'); (b) encoding pixels in the map for a reduced candidate set size; (c) re-generating maps on every Irani-Peleg pass, instead of keeping them; and (d) compressing maps (e.g., via PNG or something similar). diff --git a/d2/trans_multi.h b/d2/trans_multi.h index fe46d1a..7e14e4e 100644 --- a/d2/trans_multi.h +++ b/d2/trans_multi.h @@ -52,7 +52,7 @@ private: static ale_pos _multi_decomp; static ale_real _multi_improvement; - typedef unsigned int index_t; + typedef unsigned short index_t; std::vector trans_stack; std::vector coord_stack; From dhilvert at gmail.com Sun Mar 29 06:42:29 2009 From: dhilvert at gmail.com (David Hilvert) Date: Sun Mar 29 06:43:10 2009 Subject: [ale] Help with aligning images In-Reply-To: <49C7F589.5020600@xkyle.com> References: <49C7F589.5020600@xkyle.com> Message-ID: <20090329154229.a3cffc3e.dhilvert@gmail.com> On Mon, 23 Mar 2009 16:48:09 -0400 Kyle Anderson wrote: > Hello, > I'm attempting to use ale as a type of pattern recognizer. I have two > images that I need it to align. > > http://xkyle.com/other/25grey.jpg > http://xkyle.com/other/613grey.jpg > > But I think the pixels may be producing false positives, so I'm also > doing it with edge-detected images: > > http://xkyle.com/other/25.jpg > http://xkyle.com/other/613.jpg > > My command line looks like this: > ale --translation --follow --exp-meta-only --exp-extend > --trans-save=ale.align 25.jpg 613.jpg ale_out.jpg Using the following seems to get the bumpers to at least touch: ale --gs-mo 20% --translation --exp-noregister --lod-max=0 --perturb-upper=8 --trans-save=ale.align 25.jpg 613.jpg ale_out.jpg In general, features to be aligned should have similar tones. There's also a control point option, which has been not very extensively tested, that could be used for integration with a SIFT tool. I've avoided integrating SIFT directly into ALE because, last that I checked, those who devised the technique had decided on a patent regime excluding its use from GPL code. It is possible that an alternative to SIFT exists without such encumberments, but I haven't looked at this issue recently. > But I've tried many types of variables that I can understand in --help -ha. > Can someone give me some pointers to tell me if this is even possible with > ale, and if so, what kinda of alignment options should I be playing with? > What I'm most interested in is the percent matched (how good of a match it > is). I also don't really care how long it takes. This should probably have a FAQ entry somewhere, as it's probably the most frequently encountered request. The best 'however long it takes' technique is currently using --gs-mo and --perturb-upper along with '--gs all'. Generally, it matters very much how long it takes, if only because of the considerable resources required for testing very expensive algorithms when compared against the utility of such research for most users. Susceptibility to distraction aside, this is one of the reasons that I am currently working on putting together an OpenCL port of the current code instead of tweaking algorithms for 3D reconstruction (probably my favorite thing to do in this codebase). From dhilvert at gmail.com Mon Mar 30 06:24:09 2009 From: dhilvert at gmail.com (David Hilvert) Date: Mon Mar 30 06:24:23 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49C91524.9050401@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> Message-ID: <20090330152409.e5880d86.dhilvert@gmail.com> On Tue, 24 Mar 2009 19:15:16 +0200 Jari Juslin wrote: > The reason for equalizing the exposure is that ALE is bad at aligning images > that have different exposure, say 0 and +2 EV. As this is the case always > with HDR images I use convert to brighten up the darker image. After that > treatment ALE does the aligning very nicely. There is an --ev option for specifying exposure (which the wrapper script tries to set automatically from EXIF when exiftool is present), but it's currently necessary to also specify --exp-meta-only for --ev to have much effect. (I've added a ditz issue about this (ale-14) to the git master branch, so that it might be addressed in future versions.) > Another way to make HDR with ALE is to shoot lots of images that are all > underexposed enough so that no spot is burned and then use ALE to combine > these. This is something I am experimenting as we speak. If possible, ensure that --bayer and --black are set correctly for this case, and that the output format is suitable for HDR. Ubuntu, for example, ships with a build of libGraphicsMagick that is 8-bit, which is unsuitable. To be safe, specify --16bpc manually, as some versions of ALE may have a bug that causes this to not be set properly by default. If having blown out regions in supplemental frames produces artifacts, you might try some of the following (but note that this case is not extensively tested):- - For 0.9.x branch runs, use '--multi none'. - Ensure that --bayer is set correctly. - If blown out regions are large, try --ev (with --exp-meta-only). - For small input sets, try '--ips 0', and maybe also '--dchain triangle:2'. - If possible, ensure that --black is set correctly for all frames. From zds at iki.fi Mon Mar 30 06:34:27 2009 From: zds at iki.fi (Jari Juslin) Date: Mon Mar 30 06:34:48 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <20090329104210.34877e7d.dhilvert@gmail.com> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> Message-ID: <49D0CA63.8040609@iki.fi> David Hilvert kirjoitti: >> 'IMG_3437.png'***** (96.446125% match) | Calculating >> post-alignment match ale: >> ui/../d2/trans_multi.h:199: d2::trans_multi& >> d2::trans_multi::operator=(const d2::trans_multi&): Assertion >> `spatio_elem_map_r' failed. > Does --ips 0 fix this problem? It gets me a bit farther, thanks :-). Here is a comparison: zds@terra:/zds-data/digikuvat/hdr/hdr401$ ale --threads 2 --16bpc IMG_3963.png IMG_3964.png IMG_3965.png IMG_3966.png IMG_3967.png _IMG_3968.png IMG_3969.png IMG_3970.png IMG_3971.png ale_bug_test.png Output file will be 'ale_bug_test.png'. Original Frame: 'IMG_3963.png'. Supplemental Frames: 'IMG_3964.png'***** okay (94.710798% match). 'IMG_3965.png'***** okay (95.393201% match). 'IMG_3966.png'***** okay (96.029818% match). 'IMG_3967.png'***** (84.680275% match) | Calculating post-alignment match ale: ui/../d2/trans_multi.h:192: d2::trans_multi& d2::trans_multi::operator=(const d2::trans_multi&): Assertion `spatio_elem_map' failed. zds@terra:/zds-data/digikuvat/hdr/hdr401$ ale -ips 0 --threads 2 --16bpc IMG_3963.png IMG_3964.png IMG_3965.png IMG_3966.png IMG_3967.png _IMG_3968.png IMG_3969.png IMG_3970.png IMG_3971.png ale_bug_test.png Output file will be 'ale_bug_test.png'. Original Frame: 'IMG_3963.png'. Supplemental Frames: 'IMG_3964.png'***** okay (94.710798% match). 'IMG_3965.png'***** okay (95.393201% match). 'IMG_3966.png'***** okay (96.029818% match). 'IMG_3967.png'***** okay (96.357041% match). '_IMG_3968.png'***** okay (96.466493% match). 'IMG_3969.png'***** (85.254806% match) | Calculating post-alignment match ale: ui/../d2/trans_multi.h:192: d2::trans_multi& d2::trans_multi::operator=(const d2::trans_multi&): Assertion `spatio_elem_map' failed. I'll try your patch next. What makes me think this is a bit more complex issue than a simple out-of-memory error is that it happens at different points for different input data, sometimes with just two images, sometimes as far as seven.. and all of them are 14.4MP. However, with the same data, the error strikes at exactly the same time (at least in the cases I've tested this far). I have 3G of physical memory in the system running ALE and a 32-bit kernel. Can 32-bit kernel limit the memory available to the ALE process in this kind of system? BTW, thanks a lot for your work on ALE.. I have used it a lot and it produces great results. It also enables me to shoot HDR handheld, if necessary, as with ALE I can align the images properly for HDR composition. -Jari From gmaxwell at gmail.com Mon Mar 30 06:53:03 2009 From: gmaxwell at gmail.com (Gregory Maxwell) Date: Mon Mar 30 06:53:08 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49D0CA63.8040609@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> Message-ID: On Mon, Mar 30, 2009 at 9:34 AM, Jari Juslin wrote: > I have 3G of physical memory in the system running ALE and a 32-bit kernel. > Can 32-bit kernel limit the memory available to the ALE process in this kind > of system? [snip] With a 32bit userspace a process can't use more than ~2GB of memory (minus whatever space is taken up for mapping shared libraries). If you have 64bit hardware its time to upgrade your operating system. :) From silfreed at silfreed.net Mon Mar 30 07:08:00 2009 From: silfreed at silfreed.net (Douglas E. Warner) Date: Mon Mar 30 07:08:05 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> Message-ID: <49D0D240.2070606@silfreed.net> Gregory Maxwell wrote: > On Mon, Mar 30, 2009 at 9:34 AM, Jari Juslin wrote: >> I have 3G of physical memory in the system running ALE and a 32-bit kernel. >> Can 32-bit kernel limit the memory available to the ALE process in this kind >> of system? > [snip] > > With a 32bit userspace a process can't use more than ~2GB of memory > (minus whatever > space is taken up for mapping shared libraries). > > If you have 64bit hardware its time to upgrade your operating system. :) > If your processor supports PAE [1] individual processes can support up to 4GB of RAM with the OS supporting 4-64GB of RAM. -Doug [1] http://en.wikipedia.org/wiki/Physical_Address_Extension -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature Url : http://ventricle.dyndns.org/pipermail/ale/attachments/20090330/846a58d9/signature.pgp From zds at iki.fi Mon Mar 30 07:39:31 2009 From: zds at iki.fi (Jari Juslin) Date: Mon Mar 30 07:39:41 2009 Subject: [ale] ALE with large image set (was: Processing color negative scans with ALE) In-Reply-To: <49D0D240.2070606@silfreed.net> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> <49D0D240.2070606@silfreed.net> Message-ID: <49D0D9A3.2060101@iki.fi> Douglas E. Warner kirjoitti: > If your processor supports PAE [1] individual processes can support up to 4GB > of RAM with the OS supporting 4-64GB of RAM. I read the Wikipedia PAE article once again and asked from a Linux admin I happen to know. It would seem that the PAE does *not* change user space process limit at all, only the amount of physical memory available to kernel. Ie. process limit should be the same with and without PAE. Naturally the memory can be fragmented, ie. process cannot just ask for all the available memory and get it as one block. That said.. anyone have a 64-bit ALE compiled to Windows handy? This XP workstation happens to be 64-bit and have 6G of memory. -Jari From silfreed at silfreed.net Mon Mar 30 08:04:46 2009 From: silfreed at silfreed.net (Douglas E. Warner) Date: Mon Mar 30 08:05:18 2009 Subject: [ale] ALE with large image set In-Reply-To: <49D0D9A3.2060101@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> <49D0D240.2070606@silfreed.net> <49D0D9A3.2060101@iki.fi> Message-ID: <49D0DF8E.1030102@silfreed.net> Jari Juslin wrote: > Douglas E. Warner kirjoitti: >> If your processor supports PAE [1] individual processes can support up >> to 4GB >> of RAM with the OS supporting 4-64GB of RAM. > > I read the Wikipedia PAE article once again and asked from a Linux admin > I happen to know. It would seem that the PAE does *not* change user > space process limit at all, only the amount of physical memory available > to kernel. Ie. process limit should be the same with and without PAE. Nothing I read [1] [2] [3] [4] states that processes are limited to 2 GB (PAE or otherwise). On a non-PAE system you would be limited to available RAM (<4GB), on a system w/ PAE each process can address up to 4GB of RAM (assuming you have more than that available). -Doug [1] http://linux.derkeiler.com/Mailing-Lists/Fedora/2008-07/msg03434.html [2] http://kerneltrap.org/node/2450 [3] http://osnews.com/story/20211/CPU_Rings_Privilege_and_Protection [4] http://www.linuxquestions.org/questions/linux-general-1/32-bit-os-and-4gb-memory-limit-707762/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature Url : http://ventricle.dyndns.org/pipermail/ale/attachments/20090330/3641b5fe/signature.pgp From gmaxwell at gmail.com Mon Mar 30 08:17:51 2009 From: gmaxwell at gmail.com (Gregory Maxwell) Date: Mon Mar 30 08:18:00 2009 Subject: [ale] ALE with large image set In-Reply-To: <49D0DF8E.1030102@silfreed.net> References: <49C7F589.5020600@xkyle.com> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> <49D0D240.2070606@silfreed.net> <49D0D9A3.2060101@iki.fi> <49D0DF8E.1030102@silfreed.net> Message-ID: 2009/3/30 Douglas E. Warner : > Jari Juslin wrote: >> Douglas E. Warner kirjoitti: >>> If your processor supports PAE [1] individual processes can support up >>> to 4GB >>> of RAM with the OS supporting 4-64GB of RAM. >> >> I read the Wikipedia PAE article once again and asked from a Linux admin >> I happen to know. It would seem that the PAE does *not* change user >> space process limit at all, only the amount of physical memory available >> to kernel. Ie. process limit should be the same with and without PAE. > > Nothing I read [1] [2] [3] [4] states that processes are limited to 2 GB (PAE > or otherwise). ?On a non-PAE system you would be limited to available RAM > (<4GB), on a system w/ PAE each process can address up to 4GB of RAM (assuming > you have more than that available). Reading is nice? practice is nicer. In practice the kernel virtual mapping (Even, HIGHMEM doesn't remove the need to have part of the kernel in the userspace memory map) and shared libraries will still get in your way. The default userspace/kernel split in the no highmem case is 3G(userspace)+1G(kernel). From zds at iki.fi Mon Mar 30 08:18:06 2009 From: zds at iki.fi (Jari Juslin) Date: Mon Mar 30 08:18:16 2009 Subject: [ale] ALE with large image set In-Reply-To: <49D0DF8E.1030102@silfreed.net> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> <49D0D240.2070606@silfreed.net> <49D0D9A3.2060101@iki.fi> <49D0DF8E.1030102@silfreed.net> Message-ID: <49D0E2AE.5010009@iki.fi> Douglas E. Warner kirjoitti: > Nothing I read [1] [2] [3] [4] states that processes are limited to 2 GB (PAE > or otherwise). On a non-PAE system you would be limited to available RAM > (<4GB), on a system w/ PAE each process can address up to 4GB of RAM (assuming > you have more than that available). Yes. What I tried to say is that on system with 4G or less memory, having PAE or not having it does not change anything when it comes to memory available to a single process. Unless I read you wrong, we agree on this one. I'm not a native, so please correct me if I took you wrong. -Jari From dhilvert at gmail.com Tue Mar 31 00:47:12 2009 From: dhilvert at gmail.com (David Hilvert) Date: Tue Mar 31 04:55:56 2009 Subject: [ale] Processing color negative scans with ALE In-Reply-To: <49D0CA63.8040609@iki.fi> References: <49C7F589.5020600@xkyle.com> <49C8F509.9050509@iki.fi> <49C8F7E4.8090309@iki.fi> <49C90BC5.4010407@bredband.net> <49C91524.9050401@iki.fi> <49C93838.2030303@iki.fi> <20090329104210.34877e7d.dhilvert@gmail.com> <49D0CA63.8040609@iki.fi> Message-ID: <20090331094712.0105d9bc.dhilvert@gmail.com> On Mon, 30 Mar 2009 16:34:27 +0300 Jari Juslin wrote: > What makes me think this is a bit more complex issue than a simple > out-of-memory error is that it happens at different points for different > input data, sometimes with just two images, sometimes as far as seven.. and > all of them are 14.4MP. There was apparently a serious memory leak present; new releases without the leak are being prepared for 0.8 and 0.9 branches. For 3-channel input, the leak would result in roughly 172MB of lost memory for each 14.4MP frame, or about 1GB after 6 14.4MP frames. > I have 3G of physical memory in the system running ALE and a 32-bit kernel. > Can 32-bit kernel limit the memory available to the ALE process in this kind > of system? One simple and cheap way to discover what is going on with memory prior to abnormal termination is to monitor top output (e.g., top -b -p ). > BTW, thanks a lot for your work on ALE.. I have used it a lot and it produces > great results. It also enables me to shoot HDR handheld, if necessary, as > with ALE I can align the images properly for HDR composition. I'm glad that you find it useful. One of the goals of ALE is to allow HDR output to be produced directly. If ALE can't do this, then something is wrong, and the program should be patched. Bug reports in this area would be useful. From dhilvert at gmail.com Tue Mar 31 07:33:54 2009 From: dhilvert at gmail.com (David Hilvert) Date: Tue Mar 31 07:34:35 2009 Subject: [ale] 0.8.11.2 and 0.9.0.3 Message-ID: <20090331163354.e6a00af6.dhilvert@gmail.com> Releases 0.8.11.2 and 0.9.0.3 are now available for download. URLs: http://auricle.dyndns.org/ALE/download/ale-0.8.11.2.tar.gz http://auricle.dyndns.org/ALE/download/ale-0.9.0.3.tar.gz Overview: 0.8.11.2 fixes a serious memory leak, adds a manual page, and refines checks for bit depth capabilities. 0.9.0.3 fixes a serious memory leak, fixes an output bit depth bug, and refines bit depth checks. 0.8.11.2 Changelog: o Revise --version to include ImageMagick library information. o Add web documentation to this tree. o Add ditz issue database and package document BUG-REPORTING. o Add a manual page, as advised by Debian Developer Ruben Molina. o Refine warnings for 8-bit output when --16bpc (incl. default). o Fixed serious memory leak in alignment code, and less serious leaks. 0.9.0.3 Changelog: o Revise --version to include ImageMagick library information. o Add ditz issue database and package document BUG-REPORTING. o Refine warnings for 8-bit output when --16bpc (incl. default). o Fix --16bpc default behavior. o Fixed serious memory leak in alignment code, and less serious leaks.