Asterism - Image preProcessing and Conversion for Linux

General discussion about StarTools.

Asterism - Image preProcessing and Conversion for Linux

Postby Rowland » Wed May 13, 2015 11:18 pm

I have been working on a Linux preprocessing utility for about two years, primarily for my own use as an educational tool and the essential ingredients, curiosity and fun. Now in its fourth iteration, it is a more functional package with a new name features and interface.

Noted: there are plenty of excellent preprocessing programs out there... and I use PixInsight for many tasks. I am just sharing what I have been doing in astrophotography - DIY software.

Primarily intended for consumer digital camera RAW data, the Asterism user manual explains far more than I should include here.

Asterism is a bash script that performs a number of tasks more clearly arranged in a tabbed user interface - compared to previous versions.

Asterism_superbias creates a superbias, significantly reducing random noise in the master bias.

Asterim uses existing programs to do the preprocessing work. Yad (GUI) dcraw ImageMagick (Q16) and Panotools are required. Hugin MacroFusion and Enfuse are optional. Note: libtiff should be installed on most Linux distros - if not it is also a requirement.

Asterism's primary function is preprocessing. That is image reduction / calibration deBayer alignment and stacking. Secondary processes are stand-alone image calibration, without alignment and stacking. Master frame creation. Conversion (16 bit monochrome Bayer) - not that useful unless you have special requirements.

There are range of other options, again better explained by viewing the documentation and playing with a small image set to illustrate the various functions.

Not much more to say on the topic. It's a work in progress, but functional, all the same.

There is a link to the Asterism page in my signature block.

The remainder of this thread is outdated. I have left images in place to show development with free programs such as DCRAW and ImageMagick and more recently, Panotools.
Last edited by Rowland on Thu Jul 06, 2017 1:58 am, edited 120 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 24, 2015 12:40 am

Early work. Image processing with DCRAW ImageMagick and Panotools. The highly saturated colours are just that and not the default output. I use the scientific colour module to get a good look at the luminance and saturation capability of the image. It's testing....
Attachments
M8-rawprepro - other programe calibration.jpg
M8-rawprepro - other programe calibration.jpg (193.13 KiB) Viewed 5128 times
Last edited by Rowland on Fri Jun 09, 2017 3:18 am, edited 12 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Mon May 25, 2015 11:27 pm

Image calibration with ImageMagick
Attachments
M8_divide_flat.jpeg
M8_divide_flat.jpeg (91.35 KiB) Viewed 5066 times
M8_no_flat.jpeg
M8_no_flat.jpeg (77.09 KiB) Viewed 5066 times
flat.jpeg
flat.jpeg (29.19 KiB) Viewed 5085 times
Last edited by Rowland on Fri Jun 02, 2017 6:14 am, edited 4 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sat May 30, 2015 12:31 am

This is the basic calibration method with ImageMagick.

Jim Solomon's Astrophotography Cookbook
The formula that relates these physical phenomenon, and the actual frames we'll collect over a night of imaging, are as follows:

(1) Light = (Signal * Flat Signal) + Dark + Offset

where Signal is the image of the target object we wish we could collect under ideal circumstances, and Light is the image we actually captured. Rearranging the terms, we have:


(2) Signal = Light - (Dark + Offset)/Flat Signal

But realize that the Flats we capture with the camera will, in turn, be "polluted" by Darks and Offsets in their own right, and so we must subtract Flat Darks and Flat Offsets from the Flat Lights as follows:

(3) Flat Signal = Flat Light - (Flat Dark + Flat Offset)

So, plugging equation (3) into equation (2), yields this general formula:


(4) Signal = Light - (Dark + Offset)/Flat Light - (Flat Dark + Flat Offset)

Here, "Dark" refers to the thermal noise signal of the imaging camera; i.e., the noise signal that varies in proportion to temperature, ISO, and exposure length. Note, however, that any exposure we take with a digital camera contains the Offset, and "Darks" are no exception. So, if we define Dark' to be an exposure of some length with the body cap in place, then Dark' = Dark + Offset, and, similarly, Flat Dark' = Flat Dark + Offset. Plugging these values into Equation 4 yields the following simplified form:


(5) Signal = Light - Dark'/Flat Light - Flat Dark'

And just to make things even simpler, let's drop the prime indicators (the apostrophes) that we stuck on "Dark" and "Flat Dark", and just remember that by "Dark" and "Flat Dark" we mean frames captured with the body cap in place but with the same ISO and exposure length as the Lights and Flat Lights, respectively. That gives us our final form:


(6) Signal = Light - Dark/Flat Light - Flat Dark (bias) - editor's comment

Equation 6 gives us our marching orders for astrophotography, providing us with a set of Frames that must be captured for each imaging session. The actual order in which I choose to capture these frames is as follows, the reasons for which will be made clear in the acquisition section below:

Flat Darks (bias) editor's comment
Flat Lights
Lights
Darks
Last edited by Rowland on Fri Jun 02, 2017 6:16 am, edited 6 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 31, 2015 3:43 am

ImageMagick auto-level. Normalising data. This process is useful for fixing bad flat frames where flats blow out image features.
Attachments
medianflatAUTOLEVELandNoautolevel.jpg
medianflatAUTOLEVELandNoautolevel.jpg (238.53 KiB) Viewed 5004 times
Last edited by Rowland on Fri Jun 09, 2017 3:20 am, edited 3 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 31, 2015 9:56 am

Calibrated aligned and stacked with an Asterism predecessor, using. DCRAW ImageMagick and Panotools.

Please visit http://www.synergous.com/flatpress

StarTools processing - Develop, Wipe, Develop, HDR Reveal, Color, Sharp - as below - no noise reduction

Code: Select all
StarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDRStarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]StarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Attachments
EtaCarinaSTIM.jpeg
EtaCarinaSTIM.jpeg (432.11 KiB) Viewed 4994 times
Last edited by Rowland on Fri Jun 09, 2017 3:22 am, edited 2 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun Jun 07, 2015 6:45 am

Attachment deleted.

Please visit www.synergous.com/flatpress
Last edited by Rowland on Fri Jun 02, 2017 6:22 am, edited 3 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: Image calibration, processing and RAW conversion - Linux

Postby smokey688i » Tue Dec 13, 2016 4:41 pm

Rowland,

Old thread I know...I've just discovered how much better Regim is than DSS, but I came across your FAICS program and am working on setting a Linux virtual machine on my Win10 OS to give it a trial run. I notice you've recommended Regim fairly recently though - have you dropped this project, or do you still prefer the results out of FAICS over Regim?

-Chris
smokey688i
 
Posts: 18
Joined: Mon Feb 16, 2015 2:52 am

Re: Image calibration, processing and RAW conversion - Linux

Postby Rowland » Fri Dec 16, 2016 11:35 pm

Hi Chris.

Regim is more mature than FAICS and I prefer Regim over DSS. Regim is an excellent, very flexible preprocessing utility.

FAICS is a little unconventional (and now redundant) but does a very good job as a rule.

As you are using Windows stick with Regim.

Cheers

Rowland
Last edited by Rowland on Tue Jun 27, 2017 4:46 am, edited 2 times in total.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Re: RAW Image Processing and Conversion Utility for Linux

Postby Rowland » Sat Jun 03, 2017 7:29 am

FAICS is redundant - see here for a more functional and up-to-date package.
Rowland
 
Posts: 207
Joined: Sat Mar 05, 2011 10:54 pm

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest