Another histogram clipping question

Questions and answers about processing in StarTools and how to accomplish certain tasks.
almcl
Posts: 263
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Another histogram clipping question

Post by almcl »

Seeing the thread about histogram clipping in the Feature Requests prompts me to ask a question of my own.

For some while I have been concerned that the histogram (as shown in film dev) always appears cut off at the left at the end of processing:
after superstructure still looks clipped.jpg
after superstructure still looks clipped.jpg (33.72 KiB) Viewed 4340 times


So I've just checked my last set of images to try discover where I am getting this effect and it seems that after Autodev -Bin -Crop -Wipe-2nd Autodev the histogram is well over to the left.

I saved at this stage and loaded it into ImageJ and got this result:
histogram after 2nd autodev imagesJ.jpg
histogram after 2nd autodev imagesJ.jpg (95.97 KiB) Viewed 4340 times
That seems to me to indicate a degree of clipping: 10 mega pixels in the first three bins of the 256 bin histogram.

I may be misunderstanding clipping, and of course it's easily fixed by upping the Skyglow in Film Dev by 1%, but I wonder if it's something in the settings in the Autodevs that I am doing wrong? Using the 'Ignore Fine Detail' slider in conjuction with the 'Outside ROI' one does a magnificent job on background noise, but may be introducing this problem?
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Another histogram clipping question

Post by admin »

That histogram indeed doesn't look amazingly healthy...
Be careful with the outside ROI parameter, from the docs;
It is important to note that AutoDev will never clip your blackpoints outside the RoI, unless the 'Outside RoI Influence' parameter is explicitly set to 0% (though it is still not guaranteed to clip even at that setting). Detail outside the RoI may appear very dark (and approach 0/black), but will never be clipped.
E.g. don't set it to 0.

The Wipe module will clip dead pixels and dark anomalies. Other than that, I don't know what else could be causing clipping. Perhaps sharing the dataset and log would shed some light on this?
Ivo Jager
StarTools creator and astronomy enthusiast
almcl
Posts: 263
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: Another histogram clipping question

Post by almcl »

Thanks, Ivo.

As far as I recall the Ignore ROI was 5% and the Ignore fine detail at about 5.5 .

I've put the stack and the extracted log file in a dropbox folder:
https://www.dropbox.com/sh/d4nzfrxvahfj ... _GJXa?dl=0

Please ignore the satellite trails, dust bunnies and horrific vignetting - I am having calibration issues at the moment despite having what appear to me to be good flats, dark flats, bias and dark frames avaialable in a multitude of combinations, so end up cropping most of the frame out.
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Another histogram clipping question

Post by admin »

almcl wrote: Fri Apr 09, 2021 10:12 am dust bunnies and horrific vignetting - I am having calibration issues at the moment
That looks to be part of the issue. You have some pretty extreme noise correlation going on in your data;
Selection_558.jpg
Selection_558.jpg (209.18 KiB) Viewed 4305 times
That's very bad. :(

Those dark clumps will be filtered by the "Dark Anomaly" filter and the "Ignore Fine Detail <" filters during Wipe and AutoDev, and many of thse outliers may be clipped as they do not constitute reliable background readings. As it stands AutoDev and Wupe are working as expected.

This correlation should be urgently addressed. Are you dithering? The datasets are quite big - you're not drizzling, are you?

The satellite trails should not be in your image - they should have been stacked out if, you used any sort of outlier rejection method (median, kappa/sigma) which is highly recommended.
Ivo Jager
StarTools creator and astronomy enthusiast
almcl
Posts: 263
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: Another histogram clipping question

Post by almcl »

Thanks for taking a look, Ivo, very much appreciated.

I am dithering (every frame) and no, I don't drizzle. The 30 or so light frames are stacked with a Sigma clip, usually in DSS (but see below).

The output from the ASI 2600 mc pro is a fairly massive .fits file ~ 50Mb. It's presently running at Gain 100, offset 30 and -10°C. Most commonly I am using 180 sec exposures. The statistics look like this:
light stats.jpg
light stats.jpg (34.03 KiB) Viewed 4298 times
But I *think* the problem is coming from the flats which either over correct, under correct , introduce artefacts that weren't there before and/or fail to remove the dust bunnies even though they can be seen on both lights and flats, with a bit of a stretch. I've tried flats with ADUs of 10k, 15k, 20k and 25k. None of them seemed to work or made much difference. I've tried stacking in DSS, ASTAP (40 minute wait to see the results) and Astroart which just choked and refused to process (at least I think that is what happened - its user interface is a nightmare). I've also tried with and without darks, with and without bias, and with or without dark flats.

One thing that worries me about the flats is the histogram :
flat histogram.jpg
flat histogram.jpg (35.68 KiB) Viewed 4298 times
which looks markedly different to that of the lights:
light histogram.jpg
light histogram.jpg (33.79 KiB) Viewed 4298 times
In case it helps, I'll put an individual light, flat and bias frame in the dropbox folder referenced above. Light file prefixed L, flat file prefixed F.

Grateful, as ever, for any insights you may care to offer.
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
Burly
Posts: 387
Joined: Mon Aug 31, 2015 7:06 am
Location: Northamptonshie uk

Re: Another histogram clipping question

Post by Burly »

What source are you using for you flats panel ?
almcl
Posts: 263
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: Another histogram clipping question

Post by almcl »

Tried several, the method used for the last 7 years was a sheet of white acrylic illuminated by a suitably distanced mains LED bulb, but during this saga I have tried using the stretched T-shirt dawn sky method and more recently an EL tracing panel held in place over the end by a wooden frame.

All similarly unsuccessful. The acrylic panel worked fine with my Canon DSLR.
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
woldsstargazer
Posts: 3
Joined: Wed Jul 06, 2022 8:24 pm

Re: Another histogram clipping question

Post by woldsstargazer »

I save at each major stage of processing in ST and check the histogram. Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness. Superstructure & denoise are other areas where blackpoint can be clipped so again save before these stages & redo if the histogram is clipped. You also want a nice inverted U at the autodev or filmdev stage and again in the colour module.
almcl
Posts: 263
Joined: Wed Jan 21, 2015 7:15 pm
Location: Shropshire. UK

Re: Another histogram clipping question

Post by almcl »

woldsstargazer wrote: Mon Oct 10, 2022 2:50 pm Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness.
Yes, indeed the problems do start with wipe. Nothing I've tried in the parameters in the wipe module can prevent it clipping the background:
clipping.jpg
clipping.jpg (31.89 KiB) Viewed 2488 times

So it is now dealt with towards the end of processing, in the film dev module.

Probably if I knew what 'correlated noise' was and how to prevent or deal with it, (as per Ivo's earlier advice) the wipe module might perform better.
Skywatcher 190MN, ASI 2600 or astro modded Canon 700d, guided by OAG, ASI120, PHD2
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Another histogram clipping question

Post by admin »

almcl wrote: Mon Oct 10, 2022 9:14 pm
woldsstargazer wrote: Mon Oct 10, 2022 2:50 pm Your problems could start at the Wipe stage. Definitely check here and go back if you are clipping and change settings eg gradient aggressiveness.
Yes, indeed the problems do start with wipe. Nothing I've tried in the parameters in the wipe module can prevent it clipping the background:
clipping.jpg


So it is now dealt with towards the end of processing, in the film dev module.

Probably if I knew what 'correlated noise' was and how to prevent or deal with it, (as per Ivo's earlier advice) the wipe module might perform better.
Almost nothing in StarTools clips your data (unless specifically stated in the UI). Wipe - at most - will clip some dark anomalies (because - well - they are anomalies), but that's the extent of it.

The histogram you are displaying is an 8-bit histogram; it will be rather useless when trying to evaluate high bit-depth data that has most of its detail compressed in the extreme lower band of the dynamic range. Incidentally, the latter is precisely Wipe's job.

Why are you concerned about clipping if I may ask?

Correlated noise is noise grain that is bigger than one pixel; if two pixels influence each other's noise levels, it is said to be "correlated noise". Correlated noise may look like bigger-than-one-pixel "blobs" of noise, or may look like tiny vertical or horizontal single-pixel width (or height) "rectangles", extending out two or three pixels.
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply