Fancy processing my M78 stack, anyone?

Questions and answers about processing in StarTools and how to accomplish certain tasks.

Fancy processing my M78 stack, anyone?

Postby Melty » Sun Jan 20, 2019 6:20 pm

Hi,

I have been really struggling to get a good image out of an HDR DSS stack of M78 data i compiled a few weeks ago. All in all, I have over 7 hours of data.

All my images from Startools exhibit an unattractive brown mottle, where there should be smooth dust surrounding this reflection nebula.

A fellow imager, has kindly had a go of processing my stack and has produced an image which was so good, it literally made my jaw drop. His weapon of choice is Pixinsight.

I've been using Startools for a good year now, with some pretty good images under my belt, but this image has defeated me, and I am lost in trying to smooth out the noise.

Here is a sample workflow of my numerous attempts:

StarTools 1.4.340
Sun Jan 20 12:45:56 2019
-----------------------------------------------------------
File loaded [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).fts].
---
Type of Data: Linear, was not Bayered, or was Bayered + white balanced
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [2.20]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [24.07 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
--- Crop
Parameter [X1] set to [4121 pixels]
Parameter [Y1] set to [3121 pixels]
Parameter [X2] set to [135 pixels (-4505)]
Parameter [Y2] set to [353 pixels (-3119)]
--- Bin
Parameter [Scale] set to [(scale/noise reduction 89.20%)/(125.68%)/(+0.33 bits)]
Image Size 3557 x 2470 pixels
--- Wipe
Parameter [Mode] set to [Correct Color & Brightness]
Parameter [Temporary AutoDev] set to [Yes]
Parameter [Precision] set to [512 x 512 pixels]
Parameter [Dark Anomaly Filter] set to [20 pixels]
Parameter [Drop Off Point] set to [100 %]
Parameter [Corner Aggressiveness] set to [100 %]
Parameter [Aggressiveness] set to [90 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [92.06 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [Off]
Mask used (BASE64 PNG encoded)

--- Color
Parameter [Cap Green] set to [To Yellow]
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Dark Saturation] set to [2.00]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.07]
Parameter [Green Bias Reduce] set to [1.00]
Parameter [Red Bias Reduce] set to [1.75]
Parameter [Mask Fuzz] set to [1.0 pixels]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).tiff].
--- Wavelet De-Noise
Parameter [Filter Type] set to [Gaussian Noise Diffusion]
Parameter [Grain Size] set to [6.5 pixels]
--- Wavelet De-Noise
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Scale Correlation] set to [6]
Parameter [Color Detail Loss] set to [200 %]
Parameter [Brightness Detail Loss] set to [200 %]
Parameter [Grain Size] set to [30.9 pixels]
Parameter [Read Noise Compensation] set to [100.00 %]
Parameter [Smoothness] set to [100 %]
--- Layer
Parameter [Layer Mode] set to [Darken]
Parameter [Cap Mode] set to [Clip]
Parameter [Brightness Mask Mode] set to [Off]
Parameter [Filter Type] set to [Gaussian (Fg)]
Parameter [Blend Amount] set to [100 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Filter Kernel Radius] set to [1.0 pixels]
Parameter [Offset X] set to [0.0 pixels]
Parameter [Offset Y] set to [1.5 pixels]
Parameter [Brightness Mask Power] set to [1.00]
File saved [G:\Astronomy\Astrophotography Images\06-01-19 M78 Z61 E-PL5\08-01-19\Lights\M78 (No Drizzle).tiff].

Then over to GIMP to stretch out detail Startools just doesn't bring out (in my hands).

My earlier efforts went along the lines of: AUTODEV->CROP->BIN->WIPE->DEV->DECON->CONTRAST->HDR->SHARP->COLOUR->TRACKING OFF/DENOISE->LAYER (fix trailed stars)->GIMP

Here is my FITS file.

https://drive.google.com/open?id=16Z9f7 ... wcMuVM_s3a

And one of my noisey efforts:

https://drive.google.com/open?id=1b0mxe ... RMXplxoWxr

I would love to know what Startools can muster with more experience hands.

Cheers
Melty
 
Posts: 9
Joined: Fri Dec 22, 2017 1:18 pm

Re: Fancy processing my M78 stack, anyone?

Postby almcl » Sun Jan 20, 2019 11:04 pm

Had a very quick go, not that I am any sort of expert, using the work flow Richard Konrad outlined a few weeks ago for dealing with noisier images (mine are mostly pretty noisy, so it was of great interest).

Not sure if it's any better and there are several things I am not sure about - there seems to be some walking noise and light pollution as well as some banding which I've attempted to crop out, and I haven't tried dealing with the star halos, but see what you think?
Attachments
M78 (No Drizzle).jpg
M78 (No Drizzle).jpg (324.89 KiB) Viewed 282 times
Skywatcher 200P with an astro modded Canon 700d, cls ccd filter, field flattener, guided by QHY 5Lm, PHD2
almcl
 
Posts: 86
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: Fancy processing my M78 stack, anyone?

Postby Melty » Tue Jan 22, 2019 6:19 pm

Thanks for making the effort and trying. That looks similar to my earlier efforts.

I still can't get a result comparable to Pixinsight.

Tips anyone?
Melty
 
Posts: 9
Joined: Fri Dec 22, 2017 1:18 pm

Re: Fancy processing my M78 stack, anyone?

Postby happy-kat » Wed Jan 23, 2019 6:33 pm

It would help if you want comments please share how many light files and what calibrations files etc.
On my monitor this looks OK but it may look dark on other monitors or mobiles.
StarTools.
Resized 75% and saved jpeg slightly lower quality which has stripped it a bit, the colour was better before the resize in the emissions.
M78 (No Drizzle).ftsv2rslq.jpg
M78 (No Drizzle).ftsv2rslq.jpg (460.64 KiB) Viewed 244 times
happy-kat
 
Posts: 45
Joined: Sun Feb 01, 2015 11:31 am

Re: Fancy processing my M78 stack, anyone?

Postby admin » Thu Jan 24, 2019 4:48 am

Hi,

Appreciate everyone's advice here already. :thumbsup:

I had a look at the dataset as well and there are a number of issues/concerns I have regarding your dataset, even before going into processing;

  • High Dynamic Range imaging (e.g. blending exposure times) is rarely needed. The only exception I can think of would be M42. In the case of most/all other objects, stars would blow out much earlier than the actual DSO.
  • Blending different exposure times will mean blending different noise amounts and stats. This makes noise propagation tracking impossible; noise reduction or measurements that work in one area, would be completely inappropriate in another. My strong advice; just don't do it! If you absolutely need a HDR image (I would not think M78 is not an object that requires this though), process the images separately and then composite them in StarTools in the Layer module (see here).
  • The dataset appears to be pre-color calibrated. If using DSS, please observe the setting seen here.
  • The dataset does not appear to be calibrated with flat frames. Flat frame calibration, especially when doing DSOs, is not optional. Currently, your dataset shows "dark nebulae" that don't really exist. They are dust donuts caused by minute particles of dust on your sensor. If you're after the smooth faint brown dust, you cannot go without flats.
  • As others already mentioned, walking noise (streaks) is visible. Please dither between frames. Walking noise is correlated noise that noise reduction routines find hard to distinguish from real detail (as it doesn't look like random Poisson noise). Bias frames will also help reduce walking noise, but dithering is less time consuming and tends to to the trick as well.
  • The unfortunate fact is that, if, after 7 hours of imaging, you still cannot be sure whether something is real detail or a dust bunny in your data, you are not spending your precious time on the right things (namely sorely needed calibration frames!).

We could show you some trickery and "hacks", sure.
However, would you learn anything useful from it that you can use later on? Probably not!

M78 (No Drizzle).jpg
M78 (No Drizzle).jpg (108.7 KiB) Viewed 230 times

EDIT: The above is probably something i would be the most comfortable with, given that some of the darker detail just doesn't exist in reality...

It would all mostly entail working around the above issues which really just require changing/improving your acquisition methods.
It is unfortunately rarely a good idea to try to address acquisition issues using post-processing.
The only exception is where post-processing workarounds are advisable is when these acquisition issues cannot be helped in any other way, for example due to fixed circumstances or budget gear.

I hope the above "tough love" is useful!
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
 
Posts: 1752
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne

Re: Fancy processing my M78 stack, anyone?

Postby Melty » Thu Jan 24, 2019 4:59 pm

Happy-Kat and Ivo - thanks so much for chiming in and taking time to process my stack. :thumbsup:

Sorry for not posting my dataset content previously, but I must say from the outset your surmisation that I am not utilising calibration frames is incorrect - I nearly always religiously get into the habit of shooting calibration frames with all my images (Darks, Flats and Bias frames). If anything, I have been known to go too overboard, if that's possible.

Quite why the flats have not removed the dust donuts (which I was aware of during processing) is beyond me. I have made a couple of custom LED flat-field lightboxes from day one for this very purpose, that has served me well up to this point. The only group i failed to do flats for in this stack was the 60s group. Could that be the source of the noise issues?

The reason for opting for a HDR stack was because my long exposure subs are with CLS filter, and the short ones are without (as so to regain acceptable star colour). I normally stack Kappa-Sigma. I will look into the Layer module in greater detail (i normally only use it to correct elongated stars) in the future.

My mount is a simple Star Adventurer tracker, so any dithering is only available in RA. I have also been led to believe dithering in RA only may actually introduce walking noise, so hence never bothered. I am looking to upgrade to a better full-blown EQ mount, though.

My Data consists of:

141x 30s, 12x 300s, 62x 180s, 20x 60s, 19x 360s lights (in separate DSS groups)
325 Flats
166 Darks
302 Bias

I have since had another attempt at processing the data, this time increasing the dark anomaly filter and denoise grain size up to the max within the process. The results are certainly much better, but still nothing like the image from the other software program. I can't seem to get a filter/grain size big enough to smear out the noise. The dust bunnies/motes were Healed out. Sadly, looks like I lost or overwrote the image, too. :roll:

But here's the Pixinsight image I've just figured out how to upload, from said fellow imager, for comparison.

Image

Cheers.
Melty
 
Posts: 9
Joined: Fri Dec 22, 2017 1:18 pm

Re: Fancy processing my M78 stack, anyone?

Postby admin » Sat Jan 26, 2019 1:54 am

Melty wrote:Happy-Kat and Ivo - thanks so much for chiming in and taking time to process my stack. :thumbsup:

Sorry for not posting my dataset content previously, but I must say from the outset your surmisation that I am not utilising calibration frames is incorrect - I nearly always religiously get into the habit of shooting calibration frames with all my images (Darks, Flats and Bias frames). If anything, I have been known to go too overboard, if that's possible.

That's good to know!
Something must have gone wrong applying them then. They should have reduced unevenness, as well as walking noise, but they have not.
An easy way to find out is to just stack the light frames only without calibration and see what that looks like.

The reason for opting for a HDR stack was because my long exposure subs are with CLS filter, and the short ones are without (as so to regain acceptable star colour). I normally stack Kappa-Sigma. I will look into the Layer module in greater detail (i normally only use it to correct elongated stars) in the future.


Then coloring would still be off in places unfortunately. In addition you're creating severe star halos by doing HDR stacking.
For your intended purpose, indeed try shooting with and without CLS in place. Use the CLS filtered stack for your detail (luminance) and the non-CLS filtered stack for your color information. Then process both stacks separately and combine in the results in the Layer module. This will give you the best of both worlds.

My mount is a simple Star Adventurer tracker, so any dithering is only available in RA. I have also been led to believe dithering in RA only may actually introduce walking noise, so hence never bothered. I am looking to upgrade to a better full-blown EQ mount, though.

Even manually dithering/pushing the mount between frames may help.

I have since had another attempt at processing the data, this time increasing the dark anomaly filter and denoise grain size up to the max within the process. The results are certainly much better, but still nothing like the image from the other software program. I can't seem to get a filter/grain size big enough to smear out the noise.

The simple answer is that we're almost certainly not looking at Poisson (shot) noise. It's uneveness/"signal" that originates from another source, most likely introduced during the stacking or calibration process.
Unlike PI's traditional noise reduction routines that will happily smooth/obliterate everything in the image you pass it, StarTools noise reduction routines use "3D" data, the third dimension being time (signal evolution). If the "signal"/feature was there in your linear data, StarTools will do its best to keep it. In StarTools you will have a harder time destroying features in your image that are not the result of Poisson noise; StarTools assumes clean data (e.g. photon counts), where the only undue influence is Poisson (shot) noise.

IMHO, the PI result is heavily artifacted, showing boundaries that do not exist in reality (for example the star halos). It's one of the objections I have against using TGV for astrophotography purposes; it's a prime example of terrestrial-oriented noise reduction routine that looks for edges and contiguous areas. Such hard boundaries and smooth contiguous areas are virtually never the case in outer space.
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
 
Posts: 1752
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne

Re: Fancy processing my M78 stack, anyone?

Postby Melty » Sat Jan 26, 2019 11:32 am

Hi, Ivo

I think I'll try abandoning HDR, re-stacking into separate files as you suggested, and take it from there, when I get the opportunity.

So to conclude, ive never heard of the term Poisson noise, but are your saying it's not a case of my micro 4/3 camera's sensor introducing the mottling as a result of the long exposures, rather the way I stack and calibrate?

Sorry for all the questions, but I'm always looking for ways to improve my AP with your fab software.

Thanks
Melty
 
Posts: 9
Joined: Fri Dec 22, 2017 1:18 pm

Re: Fancy processing my M78 stack, anyone?

Postby admin » Sun Jan 27, 2019 5:22 am

Melty wrote:Hi, Ivo

I think I'll try abandoning HDR, re-stacking into separate files as you suggested, and take it from there, when I get the opportunity.

So to conclude, ive never heard of the term Poisson noise, but are your saying it's not a case of my micro 4/3 camera's sensor introducing the mottling as a result of the long exposures, rather the way I stack and calibrate?

It really looks like that is the case, yes!

Poisson noise is by far the biggest (virtually "only") source of noise in our images - provided all other sources of unwanted signal are taken care of properly.
It is essentially caused by your sensor's inability to count every incoming photon. Sensors will "miss" some photons randomly at a rate called the sensor's Quantum Efficiency.

So, even if two separate photosites/pixel on your sensor are hit with, say, exactly 100 photons. One photosite/pixel may count 93, while the other might count 87. This discrepancy is visible to us as fine noise grain. It is known as Poisson noise or "shot" noise. If the noise grain is not fine (e.g. you're experiencing mottle/clumps or streaks), something else is at play; such issues are usually caused (and fixed) during acquisition, stacking or incorrect RAW conversion/debayering.

StarTools focuses primarily on Poisson noise, as it is a constant "unknown quantity" throughout your processing that everyone has to deal with. It is the only form of noise that is "unfixable", no matter who you are, what exposure lengths you use and what your equipment is. It is therefore managed and tracked with the utmost care at every stage of processing.

Sorry for all the questions, but I'm always looking for ways to improve my AP with your fab software.


Please keep them coming - asking questions is the only way you and I learn stuff!
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
 
Posts: 1752
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne


Return to Image Processing Troubleshooting

Who is online

Users browsing this forum: No registered users and 10 guests