NomalizeScaleGradient

General discussion about StarTools.
Post Reply
Russ.Carpenter
Posts: 159
Joined: Thu Jul 17, 2014 8:20 pm
Location: Green Valley, Arizona

NomalizeScaleGradient

Post by Russ.Carpenter »

I use PixInsight for preprocessing. In the PI world, there is a lot of excitement over a new script called "NormalizeScaleGradient." I've only tried it once, followed by post processing in StarTools. StarTools seemed to behave normally, but do any of you have advice on whether this script is compatible with StarTool's requirement for "pure" linear data?

Thanks in advance.

Russ
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: NomalizeScaleGradient

Post by admin »

Hi Russ,

From what I understand, the NormalizeScaleGradient script chooses one "best" gradient from all frames and corrects all other frames so that their gradients look exactly the same.

The problem with this (in terms of noise signature), is that this local correction of the signal in the individual frames, also impacts its noise component locally. The end result, is that that in the final stack, noise signatures may vary significantly depending on the location in the image.

The main reason for preferably keeping the stack as "virgin" is possible, is that StarTools is able to track precisely how signal and noise are changed, to the benefit of virtually all algorithms/modules. Hence the recommendation to keep any such local manipulations to a minimum.

All that said, of course, calibration with flats also modifies signal (and thus noise) locally. Some of my R&D efforts have already gone into figuring out a way to model these modifications, but more is needed. However, it appears to me NormalizeScaleGradient adds further unwelcome distortions to local signal-to-noise ratios.

It may, of course, depending on your dataset ,be worth the trade-off if NormalizeScaleGradient somehow allows PI's stacking to achieve markedly better results, but I can't readily see how that would be the case right now.

I hope this helps / makes sense! If you have any results/findings you would like to share in due course, I'd be very interested in terms of whether it is worth the trade-off to better understand what PI needs, and what can be best left out.
Ivo Jager
StarTools creator and astronomy enthusiast
Russ.Carpenter
Posts: 159
Joined: Thu Jul 17, 2014 8:20 pm
Location: Green Valley, Arizona

Re: NomalizeScaleGradient

Post by Russ.Carpenter »

It may, of course, depending on your dataset ,be worth the trade-off if NormalizeScaleGradient somehow allows PI's stacking to achieve markedly better results, but I can't readily see how that would be the case right now.
Hi Ivo. As usual, your thinking is really interesting. Sometimes I look at StarTools as 50 percent software, 50 percent graduate course in Astro Imaging.

I also wonder about the actual value of this new script. The commentary in the PI forum is dense and erudite, but I'm not sure that it adequately deals with the ultimate benefits of using the script.

Before the new script got trendy, I was content with the results I got from PI preprocessing. WeightedBatchPreprossing is a nicely designed package solution. I've used it to produce calibrated, aligned subframes and then have integrated each channel into masters using ImageIntegration. The masters are free of hot pixels, satellite trails, cosmic ray artifacts, and column defects. As far as I know, stacking has appropriately improved SNR. StarTools appears to function properly with the masters.

I'll let you know if I can achieve better insight on this issue. In the meantime, I think I'll play it safe and not get on the NormalizeScaleGradient band wagon.

Russ
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: NomalizeScaleGradient

Post by admin »

Thanks Russ - I'd definitely be interested in any reports of significant improvements.

I can definitely see a how variable/changing gradients could cause problems for some outlier rejection methods that require good normalization.

On the flipside, however, I can also see that at some stage, the losses incurred in the quest to perfectly normalize the individual frames, start to defeat the purpose of using more sophisticated outlier rejection to begin with. E.g., if you significantly have to "fudge" your sub-frames to be able to use a "superior" outlier rejection algorithm, is it actually empirically worth it? Particularly if the stacking procedure that follows, is not aware of how gradient removal impacted/modified SNR per-pixel for each frame... :confusion-shrug:
Ivo Jager
StarTools creator and astronomy enthusiast
hixx
Posts: 250
Joined: Mon Sep 02, 2019 3:36 pm

Re: NomalizeScaleGradient

Post by hixx »

Hi Ivo,
I think the importance of the 'virgin' data discussion will increase going forward, even beyond the flat calibration. Stacking solutions becoming more and more advanced. In APP, a tool of such category is 'Local Normalization Correction'. While ST may have disadvantages with the slightly non-linear undulation on noise floor data, I found more benefits in applying these. Most likely this is highly depending on SNR and/or general quality of the dataset, but as long as noise is not a severe problem, I tend to give the stacker its way, so ST can start off using a high quality stack...

Thats not ideal, though. I think, what really would be required is a standardized interface between stacking and post prod stages, including per pixel meta data . With such interface ST could include a lot of the stacking changes to the noise floor into Tracking. This is not only pertaining to Flats only but to all calibration and normalization activity (E.g. Darks & FlatDarks do add noise, normalization correction may add local undulation etc.)

ok, you would need lots of folks to agree upon one format, just dreaming...But why not use what is already available? I am not sure about PI, but APP provides Normalization maps (effectively a "negative" stack image of the values subtracted during Normalization), plus statistical values in the FITS header. Would there be a way to make general information like that provided by PI,APP etc work for tracking? With such a normalization map, ST could "reverse engineer" whatever the stacker did per pixel, examine the (linear) resulting per-pixel noise floor stats and work on the normalized stack. It would benefit the users allowing them to create cleaner stacks or advanced stacks like mosaics, deep fields by stacking multiple users etc. and feed these into StarTools.

As you already booked out 1.9 & 1.10 on AI stuff, how about logging this for 1.11 :D ?
clear skies,
jochen
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: NomalizeScaleGradient

Post by admin »

Thanks Jochen,
hixx wrote: Mon Jul 19, 2021 6:21 pm I think the importance of the 'virgin' data discussion will increase going forward, even beyond the flat calibration. Stacking solutions becoming more and more advanced. In APP, a tool of such category is 'Local Normalization Correction'. While ST may have disadvantages with the slightly non-linear undulation on noise floor data, I found more benefits in applying these. Most likely this is highly depending on SNR and/or general quality of the dataset, but as long as noise is not a severe problem, I tend to give the stacker its way, so ST can start off using a high quality stack...
Indeed, some sort of normalization may be beneficial or even essential depending on the outlier rejection algorithms.
Thats not ideal, though. I think, what really would be required is a standardized interface between stacking and post prod stages, including per pixel meta data . With such interface ST could include a lot of the stacking changes to the noise floor into Tracking. This is not only pertaining to Flats only but to all calibration and normalization activity (E.g. Darks & FlatDarks do add noise, normalization correction may add local undulation etc.)

ok, you would need lots of folks to agree upon one format, just dreaming...But why not use what is already available? I am not sure about PI, but APP provides Normalization maps (effectively a "negative" stack image of the values subtracted during Normalization), plus statistical values in the FITS header. Would there be a way to make general information like that provided by PI,APP etc work for tracking? With such a normalization map, ST could "reverse engineer" whatever the stacker did per pixel, examine the (linear) resulting per-pixel noise floor stats and work on the normalized stack. It would benefit the users allowing them to create cleaner stacks or advanced stacks like mosaics, deep fields by stacking multiple users etc. and feed these into StarTools.
That would absolutely be ideal. I still have the APP normalization maps you sent over, and being able to output such things is a good start.

When implementing something like this, the trouble is always twofold; one is the implementation itself (which is difficult enough), the other is making it easy/streamlined enough to use for users in a way that is reliable, replicable and clearly shows the benefits every single time.
As you already booked out 1.9 & 1.10 on AI stuff, how about logging this for 1.11 :D ?
:lol:

Don't worry - it is something I am still very keen to pick up again. :thumbsup:
Ivo Jager
StarTools creator and astronomy enthusiast
hixx
Posts: 250
Joined: Mon Sep 02, 2019 3:36 pm

Re: NomalizeScaleGradient

Post by hixx »

When implementing something like this, the trouble is always twofold; one is the implementation itself (which is difficult enough), the other is making it easy/streamlined enough to use for users in a way that is reliable, replicable and clearly shows the benefits every single time.
How about just adding "Compose" channels for "Flat" and "Normalization Map"? For user that would be the same look and feel as every other load. Depending whether anything got loaded or not, ST would just de-flat / de-normalize and feed Tracking accordingly. With nothing loaded it would just be BAU. For the rest of the workflow, Flats and Maps would not be interfering with the signal of course
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: NomalizeScaleGradient

Post by admin »

hixx wrote: Tue Jul 20, 2021 1:59 pm
When implementing something like this, the trouble is always twofold; one is the implementation itself (which is difficult enough), the other is making it easy/streamlined enough to use for users in a way that is reliable, replicable and clearly shows the benefits every single time.
How about just adding "Compose" channels for "Flat" and "Normalization Map"? For user that would be the same look and feel as every other load. Depending whether anything got loaded or not, ST would just de-flat / de-normalize and feed Tracking accordingly. With nothing loaded it would just be BAU. For the rest of the workflow, Flats and Maps would not be interfering with the signal of course
The Compose module is absolutely where import would go. :)
The problem is not so much how it would work in StarTools; it is how it would work in/with all the different stacking solutions out there in a way that is consistent, clear, and yields the promised improvements.

I am still hoping I can get my initial idea to work; import a calibrated stack, and import an uncalibrated stack and let StarTools do all the reverse engineering of the signal and the modifications made. This would be, by far the easiest and most universal way of implementing this...
Ivo Jager
StarTools creator and astronomy enthusiast
hixx
Posts: 250
Joined: Mon Sep 02, 2019 3:36 pm

Re: NomalizeScaleGradient

Post by hixx »

I get this, its probably the easiest way to implement this
the downside would be the user would need to create duplicate stacks for each integration, e.g. R,B,G + NBA = 8 integrations or Lum, NBA = 4 integrations
also he would need to be very precise on what means "uncalibrated stack" for his stacking solution (i.e. disable all relevant features)
I am not sure if this approach opens up the door for all kinds of ST misbehave due to "almost uncalibrated" stacks and/or swapped uncal stacks
Isn't there a way to make DSS, PI (or others) to output its "activity" map as 32bit integer?
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: NomalizeScaleGradient

Post by admin »

hixx wrote: Wed Jul 21, 2021 2:37 pm I get this, its probably the easiest way to implement this
the downside would be the user would need to create duplicate stacks for each integration, e.g. R,B,G + NBA = 8 integrations or Lum, NBA = 4 integrations
also he would need to be very precise on what means "uncalibrated stack" for his stacking solution (i.e. disable all relevant features)
I am not sure if this approach opens up the door for all kinds of ST misbehave due to "almost uncalibrated" stacks and/or swapped uncal stacks
It would probably be fairly fool proof (for example, in DSS you just uncheck you calibration frames and stack again).
Isn't there a way to make DSS, PI (or others) to output its "activity" map as 32bit integer?
PI definitely not, and never will, as everything is modular/object-oriented and has no idea what is being done before or after a modification is made (the single biggest problem with PI and the single biggest reason why it is yielding poorer results). There is nothing overseeing/recording those modifications. :( This modularity is the whole reason why one can normalize with different algorithms like NormalizeScaleGradient in the first place.
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply