Struggling newcomer looking for guidance

Questions and answers about processing in StarTools and how to accomplish certain tasks.
User avatar
admin
Site Admin
Posts: 2543
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Struggling newcomer looking for guidance

Post by admin »

Hi Lucas,

I too had a quick look. One thing that stands out is that your dataset appears to suffer from some sort of rather severe pattern noise;
Selection_229.png
Selection_229.png (84.06 KiB) Viewed 481 times
(300x zoom)

This is not a normal/expected noise signature, e.g. it is not shot noise, but is introduced by something else.

Your first order of business should be to figure out what is causing this, as this is severely detrimental to your signal. StarTools is exceptionally sensitive to noise that is artificially correlated like this, as its algorithms will regard this as detail that needs preserving, which, under normal circumstances, is precisely StarTools' strength. Right now, StarTools' algorithms will latch on to the "web-like" artifical structure.

It appears to me the drizzling is unwarranted, as the dataset is oversampled at its native size. It also appears the drizzling is causing zipper artifacting.
Binning the dataset to 50% X/Y will make it easier to process

The Samyang will cause chromatic aberration, particularly visible around the stars. It's not the end of the world, but can/will throw off the Color module's automatic color balancing.

When processed at 50% bin, it is possible to achieve an image that is somewhat representative of the object's intrinsic qualities;
Helix_Nebula-Stack.jpg
Helix_Nebula-Stack.jpg (146.81 KiB) Viewed 481 times
Compare with;
https://3.bp.blogspot.com/-pDbjnBAtYCI/ ... -HD-19.jpg

There is just enough data in there for StarTools to be able to discern (and depict) faint radial detail and separate it from the noise.
(processed in 1.7.435 alpha)
Ivo Jager
StarTools creator and astronomy enthusiast
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

Hi Ivo! I've re-processed the data on the Helix Nebula, taking out those frames that don't have the same exposure time and gain, following your advice. My remaining integration time is about 11 hours and I asked APP to just select the top 90% so what I got is ~10 hours of exposure.

Also, I didn't do any drizzling (and you'll see it crearly) so AFAIK the data is undersampled. APP's settings also were modified to follow StarTools needs: no adaptive pedestal, no white balance, no background neutralization.

I've just loaded the FITS file fresh from APP and did an standard AutoDev run. Here's what I'm seeing at 400% zoom (would be nice to set up the forum to accept at least 1MB images):
Captura de Pantalla 2020-10-29 a la(s) 11.33.48.png
Captura de Pantalla 2020-10-29 a la(s) 11.33.48.png (355.47 KiB) Viewed 471 times
I believe there's the same noise pattern that you mentioned, do you agree?

If so, maybe there's some other APP setting I should be checking?

Also, a related question: My sub-exposures have been done from both sides of the meridian. That's to say that the gradients due to light pollution are on both sides, is that something to avoid when using StarTools? APP's background extraction tool seem to handle it without issue, but now that I saw the stretched preview of this new integration, I remember seeing some weird background results when using the Wipe module.

Just in case you want to give a look to the new dataset, I've uploaded it to Dropbox:

The sub-exposures settings are: Nikon D500 + Samyang 135mm f/2 @ 100 ISO // 300 secs // f/4.5 (to minimize chromatic aberration)

https://www.dropbox.com/s/xu5oaljglhltg ... .fits?dl=0

Thanks for all your help!!
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

Following up on this, I think I got something that starts to be acceptable.

I had to do lots of cropping to remove the gradient, not sure if it was really necessary but when using the Wipe module, it didn't go away otherwise.
Helix_Nebula-20201008_20201009_20201011_20201012_20201013_20201016-ST1.7-436alpha.jpg
Helix_Nebula-20201008_20201009_20201011_20201012_20201013_20201016-ST1.7-436alpha.jpg (194.9 KiB) Viewed 470 times
This is the log without encoded masks. What do you think? (have I used too much modules?)

-----------------------------------------------------------
StarTools 1.7.436alpha
Thu Oct 29 08:37:04 2020
-----------------------------------------------------------
File loaded [/Volumes/Lucas Media/Helix Nebula/Helix_Nebula-20200920_20201005_20201007_20201008_20201009_20201011_20201012_20201013_20201016.fits].
Image size is 5062 x 3660
---
Type of Data: Linear and was Bayered, but not whitebalanced
File loaded [/Volumes/Lucas Media/Helix Nebula/Helix_Nebula-20201008_20201009_20201011_20201012_20201013_20201016.fits].
Image size is 5714 x 3899
---
Type of Data: Linear and was Bayered, but not whitebalanced
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [0 pixels]
Parameter [RoI Y1] set to [0 pixels]
Parameter [RoI X2] set to [5714 pixels (-0)]
Parameter [RoI Y2] set to [3899 pixels (-0)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
--- Crop
Parameter [X1] set to [2089 pixels]
Parameter [Y1] set to [1475 pixels]
Parameter [X2] set to [3636 pixels (-2078)]
Parameter [Y2] set to [2507 pixels (-1392)]
Image size is 1547 x 1032
--- Wipe
Parameter [Synthetic Dark/Bias] set to [Off]
Parameter [Gradient Edge Behavior] set to [Absorb 50%]
Parameter [Synthetic Flats] set to [Off]
Parameter [Sampling Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [5 pixels]
Parameter [Gradient Falloff] set to [0 %]
Parameter [Synth. Bias Edge Area] set to [100 %]
Parameter [Gradient Aggressiveness] set to [90 %]
--- Auto Develop
Parameter [Ignore Fine Detail <] set to [Off]
Parameter [Outside RoI Influence] set to [15 %]
Parameter [RoI X1] set to [0 pixels]
Parameter [RoI Y1] set to [0 pixels]
Parameter [RoI X2] set to [1547 pixels (-0)]
Parameter [RoI Y2] set to [1032 pixels (-0)]
Parameter [Detector Gamma] set to [1.00]
Parameter [Shadow Linearity] set to [50 %]
--- Crop
Parameter [X1] set to [220 pixels]
Parameter [Y1] set to [140 pixels]
Parameter [X2] set to [1320 pixels (-227)]
Parameter [Y2] set to [873 pixels (-159)]
Image size is 1100 x 733
--- Wipe
Parameter [Synthetic Dark/Bias] set to [Off]
Parameter [Gradient Edge Behavior] set to [Absorb 50%]
Parameter [Synthetic Flats] set to [Off]
Parameter [Sampling Precision] set to [256 x 256 pixels]
Parameter [Dark Anomaly Filter] set to [5 pixels]
Parameter [Gradient Falloff] set to [0 %]
Parameter [Synth. Bias Edge Area] set to [100 %]
Parameter [Gradient Aggressiveness] set to [75 %]
--- Develop
Parameter [White Calibration] set to [Use Stars]
Parameter [Gamma] set to [1.00]
Parameter [Skyglow] set to [0 %]
Parameter [Digital Development] set to [95.31 %]
Parameter [Blue Luminance Contrib.] set to [100 %]
Parameter [Green Luminance Contrib.] set to [100 %]
Parameter [Red Luminance Contrib.] set to [100 %]
Parameter [Dark Anomaly Headroom] set to [5 %]
Parameter [Dark Anomaly Filter] set to [6.0 pixels]
--- Contrast
Parameter [Expose Dark Areas] set to [No]
Parameter [Compensate Gamma] set to [No]
Parameter [Precision] set to [512 x 512 pixels]
Parameter [Dark Anomaly Filter] set to [5 pixels]
Parameter [Aggressiveness] set to [50 %]
Parameter [Dark Anomaly Headroom] set to [50 %]
--- HDR
Parameter [Small Detail Precision] set to [Max]
Parameter [Channels] set to [Brightness Only]
Parameter [Algorithm] set to [Reveal All]
Parameter [Dark/Bright Response] set to [5.00]
Parameter [Detail Size Range] set to [1000 pixels]
Parameter [Strength] set to [1.5]
--- Wavelet Sharpen
Parameter [Structure Size] set to [Large]
Mask used (BASE64 PNG encoded)
--- SNR-aware Wavelet Sharpening
Parameter [Scale 1] set to [100 %]
Parameter [Scale 2] set to [100 %]
Parameter [Scale 3] set to [100 %]
Parameter [Scale 4] set to [100 %]
Parameter [Scale 5] set to [100 %]
Parameter [Mask Fuzz] set to [4 pixels]
Parameter [Amount] set to [300 %]
Parameter [High SNR Size Bias] set to [85 %]
Parameter [Low SNR Size Bias] set to [0 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
--- Deconvolution
Parameter [Image Type] set to [Deep Space]
Parameter [Secondary PSF] set to [Off (Primary Only)]
Parameter [Primary PSF] set to [Moffat Beta=4.765 (Trujillo)]
Parameter [Primary Radius] set to [1.5 pixels]
Parameter [Iterations] set to [11]
Parameter [Error Diffusion] set to [Off]
Parameter [Mask Fuzz] set to [8.0 pixels]
Parameter [Deringing] set to [50 %]
Mask used (BASE64 PNG encoded)
--- Color
Parameter [Bias Slider Mode] set to [Sliders Reduce Color Bias]
Parameter [Style] set to [Scientific (Color Constancy)]
Parameter [LRGB Method Emulation] set to [Straight CIELab Luminance Retention]
Parameter [Matrix] set to [Identity (OFF)]
Parameter [Dark Saturation] set to [2.0]
Parameter [Bright Saturation] set to [Full]
Parameter [Saturation Amount] set to [200 %]
Parameter [Blue Bias Reduce] set to [1.10]
Parameter [Green Bias Reduce] set to [1.82]
Parameter [Red Bias Reduce] set to [1.24]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Cap Green] set to [0 %]
Parameter [Highlight Repair] set to [2 pixels]
--- Super Structure
Parameter [Detail Preservation] set to [Linear Brightness Mask Darken]
Parameter [Compositing Algorithm] set to [Multiply, Gamma Correct]
Parameter [Brightness, Color] set to [Process Both]
Parameter [Brightness Retention] set to [Local Median]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Airy Disk Radius] set to [50 %]
Parameter [Gamma] set to [0.50]
Parameter [Detail Preservation Radius] set to [20.0 pixels]
Parameter [Saturation] set to [50 %]
Parameter [Strength] set to [100 %]
--- Filter
Parameter [Filter Mode] set to [Conservative Nudge]
Parameter [Sampling Method] set to [3x3 Average]
Parameter [Filter Width] set to [5]
Parameter [Mask Fuzz] set to [1.0 pixels]
--- Entropy-driven Detail Enhancement
Parameter [Resolution] set to [Medium]
Parameter [Channel Selection] set to [All]
Parameter [Strength] set to [100 %]
Parameter [Midtone Pull Filter] set to [20.0 pixels]
Parameter [Midtone Pull Strength] set to [50 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
Mask used (BASE64 PNG encoded)
--- Shrink
Parameter [Mode] set to [Tighten]
Parameter [Halo Extend] set to [1 pixels]
Parameter [Iterations] set to [10]
Parameter [Regularization] set to [0.85]
Parameter [De-ringing] set to [2 pixels]
Parameter [UNKNOWN] set to [1.5 pixels]
--- Wavelet De-Noise
Parameter [Grain Size] set to [6.0 pixels]
Parameter [Walking Noise Size] set to [1.0 pixels]
Parameter [Walking Noise Angle] set to [0]
Mask used (BASE64 PNG encoded)
--- Wavelet De-Noise
Parameter [Grain Size] set to [4.5 pixels]
Parameter [Walking Noise Size] set to [1.0 pixels]
Parameter [Walking Noise Angle] set to [0]
--- Wavelet De-Noise
Parameter [Scale 1] set to [95 %]
Parameter [Scale 2] set to [95 %]
Parameter [Scale 3] set to [95 %]
Parameter [Scale 4] set to [95 %]
Parameter [Scale 5] set to [50 %]
Parameter [UNKNOWN] set to [10 %]
Parameter [Scale Correlation] set to [50 %]
Parameter [Color Detail Loss] set to [50 %]
Parameter [Uniform Grain Retention] set to [50 %]
Parameter [Grain Dispersion] set to [4.5 pixels]
---
Parameter [UNKNOWN] set to [Small]
Mask used (BASE64 PNG encoded)
--- Fractal Flux
Parameter [Brightness Mask Mode] set to [Off]
Parameter [Algorithm] set to [Modulate Unsharp Mask]
Parameter [Negative Flux] set to [0 %]
Parameter [Positive Flux] set to [500 %]
Parameter [Detail Filter] set to [5.6 pixels]
Parameter [Filter Fuzz] set to [1.0 pixels]
Parameter [Filter Radius] set to [1.5 pixels]
Parameter [Filter Amount] set to [200 %]
Parameter [Mask Fuzz] set to [1.0 pixels]
Parameter [Brightness Mask Power] set to [1.0]
Mask used (BASE64 PNG encoded)
--- Repair
Parameter [Radial Samples] set to [32]
Parameter [Sub Sampling] set to [4x]
Parameter [Algorithm] set to [Warp]
Parameter [Grow Mask] set to [1 pixels]
Mask used (BASE64 PNG encoded)
--- Entropy-driven Detail Enhancement
Parameter [Resolution] set to [Medium]
Parameter [Channel Selection] set to [All]
Parameter [Strength] set to [100 %]
Parameter [Midtone Pull Filter] set to [20.0 pixels]
Parameter [Midtone Pull Strength] set to [50 %]
Parameter [Dark/Light Enhance] set to [50% / 50%]
File saved [/Volumes/Lucas Media/Helix Nebula/Helix_Nebula-20201008_20201009_20201011_20201012_20201013_20201016-ST1.7-436alpha.tiff].
User avatar
admin
Site Admin
Posts: 2543
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Struggling newcomer looking for guidance

Post by admin »

Hi Lucas,

Thank you for uploading the new stack. It makes it a lot easier to further diagnose the noise issue!
It appears to be caused by not dithering (or not dithering sufficiently?) between frames. The background is showing the tell-tale stringy "worm"-like artefacts that ensue;

Dithering in particular is extremely sensitive to this and will magnify the problem immensely.
Selection_230.png
Selection_230.png (45.02 KiB) Viewed 454 times
Fixing this should/will really clean up your datasets going forward.

If your dataset contains some extreme gradients (not uncommon under Bortle 9 skies!), and you are imaging small objects like these, try starting off with the "Uncalibrated" preset. It's quite aggressive, but that tends to be OK when the object(s) of interest are small and mostly centered (e.g. objects like the Helix or galaxies).

The use of the Contrast module mostly useful for locally optimizing dynamic range in large scale image. Given there is not too much large scale detail here, you can safely dispense with it.

The Color module will struggle a little with giving you a good color balance straight off the bat (it uses the stars to color balance on and they are suffering from chromatic aberration), so you may want to tweak this a little, (e.g. bring back some red).
Ivo Jager
StarTools creator and astronomy enthusiast
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

Hi Ivo!

Thanks for taking the time to answer and look at my dataset.

About dithering: I'm doing it with every exposure. I'm taking 5 minutes exposures and dithering with a max pixel setting of 10 (using the Asiair Pro). Do you think I should shorten the exposure time (opening the lens) to have more dithering?

Another thing could be that AstroPixelProcessor is mis-behaving, I could try stacking the same dataset with DSS to see what's going on.

Last night I did a ~4hr exposure of the Sculptor Galaxy with the same settings (300 secs subexposures @ ISO100 + f/4.5 + dithering between every frame) and got the same worm-like background noise. I could try to re-stack that on DSS that's a lot less data to crunch.

Best regards,
Lucas.
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

Update: I've re-stacked the dataset from last night with DSS and this worm-like background noise is still there (or at least I think so), and also I think APP is doing a better job than DSS:
DSS_stack.png
DSS_stack.png (410.8 KiB) Viewed 426 times
APP_stack.png
APP_stack.png (304.34 KiB) Viewed 426 times
Even with this kind of background noise, I was able to get something I like, not sure how it differs this image from the Helix Nebula but it seemed to me that it was a lot easier to process:
Sculptor_Galaxy-StarTools-1.7-436-2nd-Try.jpg
Sculptor_Galaxy-StarTools-1.7-436-2nd-Try.jpg (437.64 KiB) Viewed 426 times
So tonight I'll try doing exposures of 150 secs so that I can dither twice as much, just to see if there's some difference on the stacked dataset.

Cheers!
Lucas.
User avatar
admin
Site Admin
Posts: 2543
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Struggling newcomer looking for guidance

Post by admin »

It sound like you're doing everything right!
To get to the bottom of what's going on, can you try stacking without any calibration frames and see if the correlated noise disappears?
Ivo Jager
StarTools creator and astronomy enthusiast
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

Hello Ivo!

I don't have good news, the pattern noise seems to be there no matter what. Here's an example of stacked 300 secs subs without calibration frames:
Sculptor-300sec-uncalibrated.jpg
Sculptor-300sec-uncalibrated.jpg (275.54 KiB) Viewed 411 times
And here's an example of stacked 150 sec subs with calibration frames:
Sculptor-150sec.jpg
Sculptor-150sec.jpg (312.35 KiB) Viewed 411 times
Both were dithered between every subexposure. As I've read, for wide angle images, the dither distance should be 10 pixels.

Tonight I'll go back to 300 sec exposures and dither with a different distance.
User avatar
admin
Site Admin
Posts: 2543
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Struggling newcomer looking for guidance

Post by admin »

Hmmmm... Do let us know how you go!
If you are already dithering, then I cannot think of any reason why that pattern noise would still be there... That's really odd. :think:
Perhaps indeed try increasing the amount by which you dither (a spiral pattern is preferable).
Ivo Jager
StarTools creator and astronomy enthusiast
ldipenti
Posts: 17
Joined: Fri Oct 16, 2020 1:13 pm

Re: Struggling newcomer looking for guidance

Post by ldipenti »

admin wrote: Thu Nov 05, 2020 4:05 am Hmmmm... Do let us know how you go!
If you are already dithering, then I cannot think of any reason why that pattern noise would still be there... That's really odd. :think:
Perhaps indeed try increasing the amount by which you dither (a spiral pattern is preferable).
Hello Ivo,

Yes, I've been dithering all the time, sadly the ASIAIR Pro uses a dumbed down version of PHD2 and only allows for a maximum of 10 pixels.
I've read on cloudynights and similar forums that for uncooled DSLRs, the pattern noise issue is not so uncommon. Some of the posts suggest to add a lot more subexposures, so now that I'm imaging NGC253, I left all the data processing last night, around 13 hs and this is the result:
Captura de Pantalla 2020-11-05 a la(s) 09.14.25.jpg
Captura de Pantalla 2020-11-05 a la(s) 09.14.25.jpg (438.38 KiB) Viewed 382 times
Do you think the pattern noise is getting better? I'll continue imaging in the coming days to add more data and see if that helps.

Another thing that I read on forums is that I should be adding a lot more dark frames (I take around 20 per session), and that information may be combined with APP's dark frame scaling feature, that seems to allow creating a master dark at day time with lots of frames and then use it in any integration. So, just in case, I've left the ASIAIR shooting 160 x 5 minutes dark frames :)
Post Reply