Spatially Variant PSF Deconvolution (SVDecon) Module Use (v1.9)

[Guy]

Here are some notes relating to using this module. It is not the only way to use the module and experimentation is encouraged.
They relate to StarTools version 1.9.541alpha and later.
Please let me know if anyone sees any errors or has any additional advice they think helpful.
I will update this post as needed.
To see a full alphabetical list of module topics click here

Spatially Variant PSF Deconvolution (SVDecon) module (v1.9)
Purpose:

• The SVDecon module tries to reverse the effects that atmospheric turbulence and, optionally, the optical train, has on the data. It allows recovery of detail in seeing-limited data sets that were affected by atmospheric turbulence and diffraction. It can model the variation in the effects across the image.

Description:
The SVDecon module allows the modification of the deconvolution algorithm across the image. By deconvolving selected sample stars across the image a mapping of how the atmospheric and optical distortion - and hence deconvolution algorithm - varies across the image.
There are three modes of operation:
3 Modes:

1. Synthetic mode - When no star samples are selected - uses one synthetic atmospheric distortion model for the whole image - As pre v1.7 StarTools did. A number of synthetic models are available to choose from. This is useful in Lunar, Planetary and Solar imaging where there is a large starless area in the central part of the image.
2. Single-sample mode - When one star sample is selected - uses that sample to as the basis to calculate the custom (atmospheric + optical) distortion model PSF for the whole image - with an option to to further compensate for the optical distortion by choosing a synthetic optical distortion model - Similar to StarTools v1.7
3. Spatially variant mode - When multiple star samples are selected - uses the samples to define how the custom (atmospheric + optical) distortion model PSF varies across the image. There is an option to further compensate for the optical distortion by choosing a synthetic optical distortion model.

For a good description see SVDecon: Detail Recovery through Spatially Variant Distortion Correction
The SVDecon module is noise-aware and is able to generate its own de-ringing mask. De-ringing will still try to coalesce singularities.

Useful Sources
SVDecon: Detail Recovery through Spatially Variant Distortion Correction
Links and Tutorials

When to use:

• The SVDecon module should be used after the final global stretch (FilmDev or OptiDev) and local stretch (Contrast, Sharp and HDR) modules:
• SVDecon will be able to achieve better results the closer you get to a final image - since it has better information from tracking.
• The HDR module (and to some extent the new Sharp module) can exacerbate any residual ringing.
• If there is no oversampling, or if there is a lot of noise, the benefit of this module will be limited.
• Use only once.

Example Workflow (v1.9):
{Compose}-OptiDev-{Lens}-Bin-Crop-Wipe-OptiDev(or FilmDev)-{Contrast/HDR/Sharp/SVDecon}-Color-{Shrink/Filter/Entropy/SuperStr/NBAccent}-Track/NR(Unified-Denoise)-{Flux/Repair/Heal/Layer/Synth}

Key: {...} optional modules.

Method
For DSOs - using 'Deep Space' Mode - this method engages Spatially variant mode - with multiple star samples selected:
This is a way of using the module which should give good results in most cases:

1. Load the module
2. At the 'Mode of Operation' prompt select 'Deep space (stars available)' to create an apodization mask - a form of star mask - this selects the stars to be processed, and acts as a guide for the de-ringing algorithm.
3. Select a preview area rectangle in the image to see the effect of changes - this speeds up analysis. Choose a bright, detailed and noise-free area.
4. Click on 'Sampling' button to show the possible sample stars.
5. Keeping Zoom at 100% - should allow the stars to be large enough to evaluate and for you to position the sliders at the edge of the image easily to cover the whole image in 6-9 segments.
6. Select multiple samples - This will engage Spatially Variant mode. Try to get at least one, preferably more, from each segment.
7. Samples should be green without red centres. See 'Choosing sample stars' below.
8. Add more samples if there is a lot of variation in the way the stars are deformed.
9. Put a sample in any area where the stars have been deformed in a slightly different way.
10. Increase 'Sampled Iterations' until you see no further improvement
11. Ensure 'Synthetic Iterations' is 'Off' unless you want to use the Synthetic PSF 'Circle of Confusion (Optics Only)' model
12. Zoom in and out so you can see the effect in the detail and as a whole.
13. Toggle top "PreTweak/PostTweak" button to see effect of last adjustment - if needed.
14. Normally other default values work well - but you can experiment if you want.
15. When done, select 'All' to apply this to the whole image - this may take some time.
16. Press 'Keep'.

For Lunar/Planetary targets - using Lunar Mode - this method engages Synthetic mode - with no star samples selected:

1. At the 'Mode of Operation' prompt select 'Lunar, solar, or planetary (no stars available)'. No mask will be created.
2. Don't try and select a sample star. You will use Synthetic mode.
3. Select a 'Synthetic PSF model' to model the atmosphere.
4. Increase 'Dyn. Range Extension' to increase the dynamic range.
5. Increase the 'Synthetic Iterations' until no improvement is seen.
6. Adjust the 'Synthetic PSF Radius' until ringing occurs - then back off a little.

What result to look for:

• Elements should appear more focused as the blurring effect of atmospheric turbulence is compensated for.
• Edges should look more distinct and without the exaggerated coalescing caused by increasing the radius a lot.
• If there are ringing artefacts around the stars it indicates the 'Synthetic PSF Radius parameter' or 'Sampled Iterations' may be too high.

Ways of getting better results:

• The SVDecon module works best when there is little noise - Bin your oversampled data to improve the SNR if needed - but consider leaving some degree of oversampling to allow deconvolution to bring out finer detail. See the Bin module notes for a discussion of the issues relating to Bin vs. deconvolution.
• Data that is not oversampled is not a candidate for deconvolution if the aim is to reverse seeing-related issues.

After Use:

• Use the Color module.

Special Techniques:

Severely and non-uniformly deformed stars:

• Use as described in the method
• Increase 'Spatial Error' and 'Sampled Iterations' together to improve the stars shape.

Description of Controls:

All:
If you selected a preview area - this will apply the deconvolution to the whole image.

LoadPSFs:
This allows you to select previously saved PSF files.
PSF samples are stored in the StarTools logs in BASE64 encoded format and can be extracted from it.
To do this:

• Look for the line(s) of BASE64 code below the line 'PSF samples used (...'
  and above the line '--- Spatially Variant PSF Deconvolution'
• Copy only the BASE64 code from the log and paste it into Notepad or similar.
• Make sure there are no blank lines before or after it the BASE64 code.
• Save it into a .txt file.

Sampling/Result:
Toggles between sampling view and result view.
The Sampling view allows the selection of sample stars used for creating the PSF models.
It provides useful information to help you select good sample stars:

• Sampling View:
  • Suitable candidate stars are automatically selected and outlined in white.
  • The outline contains the pixels which are considered to relate to the star - often producing a 'starfish' pattern.
  • The 'starfish' outline may contain some turquoise/light blue sections which indicates the star is degraded by some aberration such as dust or nebulosity.
  • Red pixels within the outline show low quality areas
  • Yellow pixels within the outline show borderline usable areas.
  • Green pixels show high quality areas.
  • Blue pixels show the sampled area around the sample star.
• Results View:
  • Shows the results of deconvolution.
  • Hold the left mouse button to toggle highlighting sample stars on and off.
  • To disable a sample star just click on it.

Selecting Sample Stars
Sample star selection is critical as the star is the basis on which the custom (atmospheric + optical) distortion model PSF is calculated.

• Choosing Good Sample Stars:
  • Stars rather than other objects
  • Stars that are circular or oval - and similar in outline to other stars nearby
  • Stars not sitting on any nebulosity.
  • Stars that are well separated from other stars
  • Good samples are green stars without a red or yellow centre.
  • Acceptable samples are green stars with a yellow centre.
• Sample Star number and distribution:
  • The number of samples needed depends on the how severe the variation of the PSF across the image is.
  • When sampling it is important to provide samples over the whole image.
  • The selection of high quality samples is secondary to the coverage over the whole image.
  • So make sure you sample over the whole image even if some of the samples are of lower quality.

Synthetic Iterations:
Controls the use of the Synthetic PSF Model.

• Synthetic Iterations are automatically set to 'Off' (0) when a sample is selected.
• Use a synthetic model if no samples are selected - e.g. for Planetary/Lunar images.
• You can optionally use a synthetic model to model the Optics when one or more samples are selected. To do this increase this value from 'Off' (0).
• You can switch off synthetic PSF modelling by setting this parameter to 'Off'(0)
• For Planetary/Lunar images it may help to Increase this value significantly - into the hundreds.
• Default is 10, Range is Off(0) to 500

Sampled Iterations:
When one or more samples have been selected this parameter specifies the number of times the deconvolution algorithm is repeated.

• Not used in Synthetic mode. Only relevant when one or more samples are chosen.
• You can switch off sample modelling by setting this parameter to 'Off'(0)
• Increasing this parameter will make processing take longer.
• Increase this value incrementally if further improvement can be seen - there will be a point beyond which you will not get a better result.
• Increasing this parameter too far can lead to ringing and increased noise grain. The ringing needs to be minimised here. Any remaining noise grain can be handled by the denoise module later.
• Default is 10, Range is Off (0) to 199

Synthetic PSF Radius:
The size of the turbulence blurring that the deconvolution algorithm (specified by the 'Synthetic PSF Model' setting) will try and remove:

• Only used when the Synthetic model is enabled.
• If one or more samples are selected then the Synthetic model is by default switched off. It can be enabled by increasing 'Synthetic Iterations' from 0 [Off].
• This value can be increased until ringing starts to occur on small non-overexposed stars - then back off a little.
• Related to the seeing - Seeing-induced blur is normally 3-4.5 arc-seconds. The camera/lens combination gives a resolution between 1 and 5 arcsec/pixel depending on the equipment combination.
• Under the same seeing conditions a wide field image will have a smaller blur. A narrow field image will have a larger blur.
• Adjust the Radius until just before the smaller stars start showing signs of ringing.
• Default is 1.5, Range is 1.0 to 20.0 pixels

Spatial Error:
This control helps when stars are badly deformed by allowing the algorithm to reconstruct stars from pixels that are further away than normal.
It defines a 'blurry area' around a star where the pixels can change.

• Not used in Synthetic mode.
• Increase this value in cases where the PSF is heavily distorted causing stars to be very deformed.
• This can be due to: astigmatism, field curvature, tracking error, or other issues causing stars to severely deform.
• There is dedicated 'e' button - Euler's number preset. This provides a good starting point for defining a blurry area around a central point.
• Default is 1.00, Range is 1.00 to 6.00

Synthetic PSF Model:
Defines which model of atmospheric turbulence or optical blurring is used in the reversal process:

• If no samples are selected (Synthetic mode) the model chosen is the one applied to the whole image. In this case choose one of the atmospheric models unless the image was taken in space.
• If one or more samples are selected then the Synthetic model is by default switched off. It can be enabled by increasing 'Synthetic Iterations' from 0 [Off].
• If one or more samples are selected then there is only one model available - the one to model the optics - the 'Circle of Confusion'. With sampling the combined atmospheric and optical distortion model is calculated from the sample. The option to additionally use the 'Circle of Confusion' synthetic PSF model allows additional tweaking to the optical model if needed.
  The PSF Model used for the atmosphere will be derived from the sample star(s).
• Default is Moffat Beta=4.765 (Trujillo) (Atmosphere) with no samples selected, Circle of Confusion (Optics) when samples are selected.
• Range is:
  • Gaussian Beta=Infinite (Atmosphere) - Model used in Decon previous to StarTools 1.6
  • Circle of Confusion (Optics Only) - models basic focusing assuming no atmosphere (e.g. in space).
  • Moffat Beta=5.5 (Atmosphere)
  • Moffat Beta=5.0 (Atmosphere)
  • Moffat Beta=4.765 (Trujillo) (Atmosphere) - Uses a Moffat distribution with Beta factor 4.765. Recommended by Trujillo et al (2001)
  • Moffat Beta=4.5 (Atmosphere)
  • Moffat Beta=4.0 (Atmosphere)
  • Moffat Beta=3.5 (Atmosphere)
  • Moffat Beta=3.0 (Saglia, FALT) (Atmosphere) - Uses a Moffat distribution with Beta=3.0. As implemented in the FALT software at ESO.
  • Moffat Beta=2.5 (IRAF) (Atmosphere) - Uses a Moffat distribution with Beta=2.5. As implemented in the IRAF software by USNOAO.

Dyn. Range Extension:
This parameter allows the reallocating of dynamic range so that reconstructed highlights can show their detail correctly.

• Increase this parameter from 1.0 if significant new detail has been recovered.
• Increase it until the clarity of the new detail stops improving.
• Default is 1.00 - no reallocation, Range is 1.00 to 3.00

Deringing Focus:
Controls how much the deringed areas should still coalesce to a point of light.

• If ringing around stars is a problem the adjust this control, often in conjunction with Deringing Fuzz, until it is gone.
• Default is 50%, Range is 0% to 100%

Linearity:
Specifies at what part of the dynamic range the response becomes non-linear.

• Non-linearity can be due to:
  • The non-linear response of the sensor.
  • Stacking of sub-frames captured under different atmospheric or focus conditions.
• Default is 100% - linear across the full dynamic range, Range is 0% to 100%

Deringing Fuzz:
Controls the smoothness of the transition between the de-ringed areas and the surrounding parts of the image.

• If ringing around stars is a problem then adjust this control, often in conjunction with Deringing Focus, until it is gone.
• Lowering the value can control ringing in some cases. Even down to around 10%.
• Default is 50%, Range is 0% to 100%

Lunar, Solar or Planetary mode default settings:
If the image is of Planetary and Lunar and other non-DSO subjects

• Selecting this mode clears the apodization mask
• 'Synthetic Iterations' are increased to 50x
• 'Synthetic PSF Radius' is increased to 10.5 - something more suitable to high magnification.
• 'Sampled Iterations' are automatically set to 'Off' (0), and the sampling mode disabled, when this is selected.
• Star samples, if any, would not help for these sorts of images where there is a large starless area in the centre of the image.

Background Notes

Sample Stars and deconvolution
Sample Stars: Good candidates for sample stars have the following characteristics:

• Are set in an even background.
• Are neither over-exposed or dim.
• Ideally their profile covers most of the linear dynamic range of the image.
• Ideally they are located towards the centre of the image.

Spatial variation of the Point Spread Function
The spatial variance of the Point Spread Function can be due to any issue which has deformed the stars in a non-uniform way - such as:

• field rotation
• field curvature
• tracking error
• coma
• camera mounting stability
• some other issue which has deformed the stars in a non-uniform way

Apodizaton mask
The big difference between the apodization mask and other masks is in the way it is used.
The apodization mask has the following functions:

• To identify the boundaries of the stars where the local PSF function is applied.
• To help in the selection of sample stars - if used.
• As a guide for the de-ringing algorithm.

In v1.9 the apodization mask is generated automatically and replaces any existing mask set outside the module.

• The apodization mask is generated in the same way as other masks.
• Note: You can no longer use the Mask module to adjust the mask once created as you could in v1.8.

PSF Resampling:
SVDecon resamples the stars as they are being de-convolved. Resampling of the stars is done as it tends to reduce the development of ringing artifacts and can also improve results.

• In v1.9 the Point Spread Functions are always re-sampled.
• In v1.9 the mode used is called "Intra-Iteration + Centroid Tracking Linear" this is where all samples are resampled during the deconvolution but only after their locations have been recalculated.
• Resampling in this way while a preview is being used means it will will only re-sample the samples contained within the preview area.
• You will not see the full effects of this resampling unless you look at the whole image without a preview area selected.

Deconvolution and Oversampling
Deconvolution is used to undo the blurring effect of an unstable atmosphere and an imperfect optical train, however this can amplify noise and introduce artefacts including ringing. It works best on data which is oversampled and has a high signal to noise ratio.
However, data that is on the cusp of being oversampled, where faint stars are spread over 3 pixels, may still benefit from a small amount of deconvolution. Every optical system, no matter how expensive, spreads a point light over multiple pixels to a degree (see Airy disk). SVDecon can reverse this spreading as well - just take it easy - it is very easy to overdo this.
For further details regarding oversampling, binning and deconvolution see the Bin module background notes.

Deconvolution and singularities
Singularities in the data are those areas where there is a discontinuity in the valid data - where the valid data is missing - such as in the saturated white cores of stars. These areas normally cause bad ringing artefacts.
StarTools uses a novel de-ringing algorithm which ensures stars are protected from the Gibbs phenomenon (also known as 'panda eye effect'), while actually being able to still coalesce singularities, such as over-exposed white cores of stars, into point lights.

Tracking
Tracking in StarTools is the name given to the way in which StarTools gathers information about the signal and its evolution through different modules.
It provides each module with as much information as possible to allow it to get the best results.
Each module can:

• Understand how each pixel has been modified by previous modules.
• Influence data earlier in the processing chain (e.g. linear data) and re-apply the modifications made since then.
• Use the information gained from previous modules to understand how the signal has been changed and where the noise is.

Some benefits of Tracking:

• Deconvolution only works on linear data - but the SVDecon Module is used after the data has been stretched.
• In the SVDecon module we look at the result of a stretched and processed image and apply deconvolution on the linear data and watch its effect on the processed image.
• Noise reduction is applied at the end of processing where Tracking has gained the most information about noise.
• Noise reduction is automatically targeted most at the areas where it is needed.

Lunar/Planetary/Solar
The SVDecon module behaves slightly differently with Lunar, Planetary and Solar images when it comes to reconstructing highlights.
With these images if a reconstructed highlight requires more dynamic range it is allocated it. The reconstructed highlights are not allowed to over-expose. The dynamic range of the complete image is adjusted to accommodate the new highlights.
With DSOs, if a reconstructed highlight is not given any more dynamic range and is allowed to over-expose.
Also, these images do not require an aggressive deringing strategy.

See also StarTools help section in SVDecon on 'Planetary, Solar and Lunar datasets'

Synthetic Models of the atmosphere
The way a point source has its light scattered around its actual location is called a Point Spread Function (PSF)
Deconvolution does its best to model this PSF and then reverses it to get back to the original.
Over the years synthetic models have been developed for the PSF of atmospheric blurring.
Five of these models are available to select from in the SVDecon Module.

• Gaussian (Fast)
  • Uses Gaussian distribution to model atmospheric blurring
  • Model used previous to Startools 1.6
  • Fast
• Circle of Confusion
  • Models the way a lens focuses the light of a point source assuming no atmosphere
  • Suitable for images taken outside the Earth's atmosphere.
• Moffatt Beta=4.765 (Trujillo)
  • Uses Moffat distribution to model atmospheric blurring
  • Uses a Beta factor 4.765.
  • Recommended by Trujillo et al (2001) [PDF] as best fit for prevailing atmospheric turbulence theory
• Moffatt Beta=3.0 (Saglia, FALT)
  • Uses Moffat distribution to model atmospheric blurring
  • Uses a Beta factor of 3.0.
  • This is a rough average of the values tested by Saglia et al (1993)[PDF]
  • It corresponds with findings of Bendinelli et al (1988)[PDF]
  • As a result of studying the Mayall II cluster - Implemented as the default in the FALT software at ESO.
• Moffatt Beta=2.5 (IRAF)
  • Uses Moffat distribution to model atmospheric blurring
  • Uses a Beta factor of 2.5.
  • As implemented in the IRAF software by US National Optical Astronomy Observatory.

Selection of Beta for Moffat PSF

• A Moffat PSF with a value of 10 is similar to a Gaussian PSF.
• Pixinsight defaults to a Beta of 4
• Trujillo, Saglia, Bendinelli and others all did evaluations and came to answers varying from 2.5 to 4.765
• Take your pick - try each one out and see what you think.

[Mike in Rancho]

Hi Guy,

Great compendium as usual!

I think maybe just a couple clean-ups: I'd have to go through the old versions or maybe the changelog to see where stuff differentiates, but currently the blue sampling box is also fully automated and can no longer be resized. And after a few fixes along the way, it's actually now pretty good at picking a reasonable size, though some datasets can throw it off. But either way it's very difficult to get a lot of samples across the entire image the way we did back in 1.8, so I'm not sure the guidance about being able to pick several stars per segment still holds.

Also the white starfishy outlines can also now include light blue, as a hint that you may not be able to fully trust that star due to dust, nebulosity, or other aberrations.

There was also an upgrade whereby SVD can look beyond certain flaws, one of them being a "hole" outline within a starfishy, thus permitting them to still be viable samples. I actually wouldn't mind further exposition and guidance on that ability and to what extent we can allow those flaws, but would probably require Ivo's input.

Finally as a beta I think some things (maybe the deringing regime?) may still be on the table for changes.

Good stuff again!

[jlh]

Hi Guy,

I am curious why the PSF Resampling parameter is removed in version 1.9. Using it in version 1.8 seemed to help in certain cases.

Thanks for the new excellent write up.

Jeff

[decay]

I am curious why the PSF Resampling parameter is removed in version 1.9. Using it in version 1.8 seemed to help in certain cases.

Hi Jeff,

if I remember correctly, Ivo removed this parameter and changed programming to always use "Intra-Iteration + Centroid Tracking" after there has been a great improvement in processing speed. So no need to use one of the low quality options any longer.

viewtopic.php?p=14446#p14446

Best regards, Dietmar.

[Guy]

Hi Mike,

Thanks for the notes and comments.
I have removed the section on Sampled PSF Area - a left-over from 1.8.
I have added a note about the light blue outline.

Hi Jeff ,

Dietmar beat me to an answer - I couldn't find that reference!

Hi Dietmar,

Thanks for the information and reference to the PSF Resampling now being fixed to "Intra-Iteration + Centroid Tracking" mode.

Insights on how the modules work and how best to use them come to light all the time.
So, as usual, if there are any other tips or methods anyone thinks would be useful to others please let me know.

Thanks and regards,

Guy

[Mike in Rancho]

Hi Guy,

A couple more possible suggestions --

Under the Description of Controls category:

I was thinking some additional guidance for the Load PSF explanation, maybe especially for beginners in order to be comprehensive with the steps. SVD is expecting a text file here, so the BASE64 code can be copied from the log, pasted into something like Notepad, and then saved. Conversion unnecessary. This is unlike Mask where ST is expecting an image file, and if it was BASE64 taken from the log it has to be converted first.

For Dynamic Range Extension I wonder if "new detail created" should perhaps be "new detail recovered." I am possibly being pedantic here.

Under Background Notes:

The apodization mask can no longer be adjusted in Mask after creation. I am quietly (lol) lobbying for that power to be returned to us, for purposes of tailoring deringing support.

Great job as always, Guy.

[Guy]

Thanks for the suggestions Mike.
I have updated the sections accordingly. I definitely didn't mean to say detail was 'created'
Regards,
Guy

[Mike in Rancho]

No worries, Guy.

Also I think I may have come across that phrase before in SVD explanations, so it may have propogated from somewhere else.