Asterism - Image preProcessing and Conversion for Linux

General discussion about StarTools.

Asterism - Image preProcessing and Conversion for Linux

Postby Rowland » Wed May 13, 2015 11:18 pm

I have been working on a Linux preprocessing utility for use with consumer digital camera (CDC / DSLR) RAW image data, primarily for my own use, education, curiosity and fun.

Ivo has very generously permitted this thread in the StarTools forum. Asterism is a free and open source contribution to the art and began as a command line image conversion script. I have taken the liberty of continuing to up-date the evolution and ongoing development of this basic image preprocessing utility.

Having used several high end processing programs and then finding StarTools, a simple and reliable method of CDC RAW image reduction (as well as several other formats) was an obvious step. Within reason, the process had to be data safe and negotiate some of the common difficulties experienced by newcomers - for example, inconsistent flat illumination and dark frame to light frame, temperature mismatch. Asterism adjusts flat illumination and automatically selects the most suitable master frames (loaded by the user). Asterism has also been used to process photometric RAW data.

A welcome outcome is the easy removal of gradients with the StarTools Wipe module. The most difficult, being images acquired under light polluted skies, with significant gradients.

Asterism functions and settings are clearly arranged in a tabbed user interface and described in the user manual

0-main.png
0-main.png (84.05 KiB) Viewed 704 times


Videos
Master frames demo
Light frames demo
Develop - digital development

The posts that follow show some of the earlier development work.
Last edited by Rowland on Tue Jun 26, 2018 10:23 am, edited 134 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 24, 2015 12:40 am

Early work. Image processing with DCRAW ImageMagick and Panotools. The highly saturated colours are just that and not the default output. I use the scientific colour module to get a good look at the luminance and saturation capability of the image. It's testing....
Attachments
M8-rawprepro - other programe calibration.jpg
M8-rawprepro - other programe calibration.jpg (193.13 KiB) Viewed 9373 times
Last edited by Rowland on Fri Jun 09, 2017 3:18 am, edited 12 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Mon May 25, 2015 11:27 pm

Image calibration / flat division with ImageMagick. This was a properly matched set of flats and lights. Sometimes not the case. Please read a few posts down for more information about flat division in Asterism.
Attachments
M8_divide_flat.jpeg
M8_divide_flat.jpeg (91.35 KiB) Viewed 9311 times
M8_no_flat.jpeg
M8_no_flat.jpeg (77.09 KiB) Viewed 9311 times
flat.jpeg
flat.jpeg (29.19 KiB) Viewed 9330 times
Last edited by Rowland on Sun Jun 17, 2018 12:22 am, edited 6 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sat May 30, 2015 12:31 am

The basic image calibration/reduction method (performed by ImageMagick's scripting language) is described in the user manual.

Further reference material - Jim Solomon's Astrophotography Cookbook
[quote] - this was the original source material and describes the traditional approach to DSLR image reduction - the bias in the dark methodology.

It is probably worth mentioning that dark frame to light frame temperature mismatch is often a source of poor image quality. Dark current (noise) is a function of temperature and exposure time and unless you are using a program that performs dark scaling (and your dark frames are several minutes longer in duration than your light frames), then matching dark and light frame exposure time and temperature is the key to proper dark noise reduction. In Asterism, the Defect map process is a work-around for mismatched dark and light frames but more often, dithering and the Asterism light frame stacking method produce a reasonable result, providing bias and flat frames are available.
Last edited by Rowland on Sun Jun 17, 2018 12:33 am, edited 10 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 31, 2015 3:43 am

ImageMagick auto-level. This a redundant approach to the problem of differences in flat and light frame illumination. It worked for some images and not for others - confusing and inconsistent.

The problem of incorrect flat illumination is solved by simple mathematics. In Asterism, the mean exposure value of the target light frame is calculated and the master flat exposure mean value adjusted to produce near perfect, if not perfect to the eye, flat division. The process is so reliable that poor results can be traced to the image set - flats or lights.
Attachments
medianflatAUTOLEVELandNoautolevel.jpg
medianflatAUTOLEVELandNoautolevel.jpg (238.53 KiB) Viewed 9249 times
Last edited by Rowland on Sun Jun 17, 2018 12:17 am, edited 5 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun May 31, 2015 9:56 am

Calibrated aligned and stacked with an Asterism predecessor, using DCRAW ImageMagick and Panotools. Note that the immense star field of the Milky Way, in the vicinity of Eta Carina is revealed, showing the extent of the data captured by this set of images, nicely revealed. No attempt has been made to darken the area around EC for artistic purposes - that is, user preference.

StarTools processing - Develop, Wipe, Develop, HDR Reveal, Color, Sharp - as below - no noise reduction

Code: Select all
StarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDRStarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]StarTools 1.4.302
Sun May 31 17:25:07 2015
-----------------------------------------------------------
File loaded [/home/geoland/Pictures/Eta_Carina/EtaCarina.tiff].
---
--- Bin
Parameter [Scale] set to [(scale/noise reduction 50.00%)/(400.00%)/(+2.00 bits)]
--- Crop
Parameter [X1] set to [61 pixels]
Parameter [Y1] set to [42 pixels]
Parameter [X2] set to [2761 pixels (-56)]
Parameter [Y2] set to [1842 pixels (-34)]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [83.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [5.0 pixels]
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [UNKNOWN] set to [Yes]
Parameter [Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [2 pixels]
Parameter [Drop Off Point] set to [0 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [90.00 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal DSO Core]
Parameter [Dark/Bright Response] set to [Full]
Parameter [Detail Size Range] set to [64 pixels]
Parameter [Strength] set to [1.0]
Undo.
Redo.
Undo.
Redo.
--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [RGB Ratio, CIELab Luminance Retention]
Parameter [Dark Saturation] set to [3.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.33]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.21]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Wavelet Sharpen
Parameter [Intelligent Enhance] set to [Yes]
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Amount] set to [100 %]
Parameter [Small Detail Bias] set to [75 %]
Undo.
Redo.
File saved [/home/geoland/Pictures/Eta_Carina/EtaCarinaST.tiff].
Attachments
EtaCarinaSTIM.jpeg
EtaCarinaSTIM.jpeg (432.11 KiB) Viewed 9239 times
Last edited by Rowland on Sun Jun 17, 2018 12:39 am, edited 6 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: DSLR RAW 16bit CFA conversion script - master frame crea

Postby Rowland » Sun Jun 07, 2015 6:45 am

Update - new stable version

All the old and now redundant posts have been removed. In answer to the next post - Asterism replaced FAICS, which was another development version and an attempt to give the utility a meaningful name.
Last edited by Rowland on Sun Jun 17, 2018 8:56 am, edited 6 times in total.
Rowland
 
Posts: 226
Joined: Sat Mar 05, 2011 10:54 pm

Re: Image calibration, processing and RAW conversion - Linux

Postby smokey688i » Tue Dec 13, 2016 4:41 pm

Rowland,

Old thread I know...I've just discovered how much better Regim is than DSS, but I came across your FAICS program and am working on setting a Linux virtual machine on my Win10 OS to give it a trial run. I notice you've recommended Regim fairly recently though - have you dropped this project, or do you still prefer the results out of FAICS over Regim?

-Chris
smokey688i
 
Posts: 18
Joined: Mon Feb 16, 2015 2:52 am


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron