matrix grid shown on first autodev

General discussion about StarTools.
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

matrix grid shown on first autodev

Post by Lawrence »

After finding out how to de-Bayer properly I have been de_bayering the raw images prior to processing. However, even with the raw image not showing any pattern, the first analysis - autodev - is bringing up the matrix pattern. Is there something that I have overlooked?

Lawrence Harris
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: matrix grid shown on first autodev

Post by admin »

Hi Lawrence,

Quite possibly you may have overlooked something.

Does the following sound like something that might be happening, e.g. does the pattern change when using the zoom functionality? (from the documentation)

Zooming, panning and scaling

Image
^ StarTools' astrophotography-optimised scaling algorithm can highlight latent pattern issues. It also was designed to show constant noise levels regardless of zoom level.

Even the way StarTools displays and scales images, has been created specifically for astrophotography.


StarTools implements a custom scaling algorithm in its user interface, which makes sure that perceived noise levels stay constant, no matter the zoom level. This way, nasty noise surprises when viewing the image at 100% are avoided.


Image
^ At 100% zoom level a barely distinguishable horizontal pattern can indeed be seen.

Even more clever, StarTools scaling algorithm can highlight latent and faint patterns (often indicating stacking problems or acquisition errors) by intentionally causing an aliasing pattern at different zoom levels in the presence of such patterns.
Ivo Jager
StarTools creator and astronomy enthusiast
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

Re: matrix grid shown on first autodev

Post by Lawrence »

Hello Ivo

The full details:
I am using one image of 5 minutes so that the basic flow can be validated for me. The first process is to deBayer the raw (colour) image. There are 4 possibilites: RGGB, BGGR, GBRG, GRBG. I tried using the output of each of these selections (using a deBayer program) to see which looked best. The recommended one and the others all show the original Bayer matrix graticule as soon as 'autodev' is run. Until I can get this problem identified I suspect that later processing would be pointless.

Lawrence
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

Re: matrix grid shown on first autodev

Post by Lawrence »

I have had another look at the (apparently) new manual regarding this. Most curiously, different zoom levels on the images (one image per selected Bayer matrix option) shows different results at different zoom levels - so I am more confused than before I asked the question! One zoom level shows clear blue image with no matrix, another zoom level shows different colour - and so on. I would not know where I was if the zoom level results change.

Lawrence :?
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: matrix grid shown on first autodev

Post by admin »

Lawrence wrote:I have had another look at the (apparently) new manual regarding this. Most curiously, different zoom levels on the images (one image per selected Bayer matrix option) shows different results at different zoom levels - so I am more confused than before I asked the question! One zoom level shows clear blue image with no matrix, another zoom level shows different colour - and so on. I would not know where I was if the zoom level results change.

Lawrence :?
Hi Lawrence, per the manual and section quoted above that is exactly what it is supposed to do; it is alerting you to an otherwise latent issue by generating aliasing artefacts at different zoom levels.

There clearly is an issue with the debayering... What camera are you using?
Ivo Jager
StarTools creator and astronomy enthusiast
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

Re: matrix grid shown on first autodev

Post by Lawrence »

Hello Ivo

Many thanks for your quick reply; it enables me to know what to look for. I found the relevant bit in the new manual. However, the first image gives different results to the second image with identical debayering. The first (M42) image debayered using GRBG gives consistent zoom level results. The second (M66) however gives poor zoom results using GRBG although images were taken just minutes apart. Ihave two cameras; this one is my SX M25C. The other (with a set of images waiting successful processing) is an SX Ultrastar (uncooled).

I understand what the manual implies now; I hadn't realised its implications before. I would expect that the debayering process should be constant for each camera?

I will repeat and record the results today.

thanks

Lawrence
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

Re: matrix grid shown on first autodev

Post by Lawrence »

Without having an ability to interpret the results I am unable to progress. I have been using an alternative program to examine the images previously referred to. They do show that using Bayer settings of GBRG and also GRBG the images can be zoomed in to 200% and then show only noise - but no matrix. Using ST I am finding that both these image settings produce images with a vivid matrix displayed. This means that I cannot make any further progress without knowing (1) which setting is actually correct, and (2) how I might remove the matrix from whichever image is the correct one. Subsequent processing needs to be able to remove this problem.

Can anyone *please* advise? I could post the images somewhere if required.

Lawrence Harris
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: matrix grid shown on first autodev

Post by admin »

Hi Lawrence,

It'd be great indeed if you could upload some data for us to look at.
For the record, zoomed into beyond 100% in StarTools the intentional aliasing does not take effect (you're looking at enlarged pixels at that stage); the intentional aliasing only happens at zoom levels less than 100%.
Ivo Jager
StarTools creator and astronomy enthusiast
Lawrence
Posts: 73
Joined: Thu Jun 04, 2015 1:30 pm

Re: matrix grid shown on first autodev

Post by Lawrence »

https://www.dropbox.com/sh/aw6l0xfd3r1r ... fc78a?dl=0

If I have set this up correctly, there should be two files in 'Startools samples', GBRG and GRBG. The suggestion from Terry Platt (SX) is that GRBG should be the correct one (unfiltered reddish). MY concern is that the only way to get rid of the matrix is to bin the image. I would have expected that the software could cope with a full sized image. Please advise.

BTW, the raw images have not been bias or dark reduced. They are mere 5-minute images that I am trying to see how well I can process them in their raw state. Until I can convince myself that I have the correct Bayer, I don't want to be producing loads of wrongly deBayered images for processing.

FWIW I was puzzled when the first message came up telling me that the (single unstacked) image appeared to have 'stacking artefacts'.

The manual that I believed was a new one turned out to (apparently) be an older one with several sections labelled 'unwritten' or equivalent. Unfortunately none of the versions appear to be labelled with the version in the title, resulting in having to download every manual found and then go by the number inside. Could the manuals be clearly labelled please? I currently have several different versions short-cut on my desktop.

TIA

Lawrence Harris
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: matrix grid shown on first autodev

Post by admin »

Many thanks Lawrence!
Lawrence wrote:https://www.dropbox.com/sh/aw6l0xfd3r1r ... fc78a?dl=0

If I have set this up correctly, there should be two files in 'Startools samples', GBRG and GRBG. The suggestion from Terry Platt (SX) is that GRBG should be the correct one (unfiltered reddish). MY concern is that the only way to get rid of the matrix is to bin the image. I would have expected that the software could cope with a full sized image. Please advise.
Indeed, StarTools is rightfully alerting you to an issue with the data; severe debayering artifacts (one of the worser cases I've seen) are visible. I'll stress again, this is a StarTools feature, not a bug! Pattern noise like this is, for example, hard to deal with by noise reduction algorithms which expect the noise to be random (e.g. photon/shot noise) - this is important stuff the user needs to be made aware of and StarTools delivers.

As a quick reminder of what debayering does (in this case); for every 2x2 patch, there are only 2 green samples, 1 blue sample and 1 red sample. Debayering (in this case) interpolates the missing 2 green samples, 3(!) blue samples and 3(!) red samples. E.g. 50% of your green data, 75% of your blue data and 75% of your red data is completely made up by the debayering algorithm.

As such, your observation that binning to 50% seems to work well is absolutely correct; this "undoes" the interpolation and makes sure that there are 2x real green samples and 1x real blue and red samples for every pixel.

If you really must stick with a single frame (not recommended), binning is indeed the answer. More preferable, however, is to stacking multiple frames that have been properly dithered (so that they are slightly offset) - you will likely see the problem disappear.

Another recommendation would be to use a debayering algorithm that is better than the one used right now (avoid AHD-based algorithms, possibly use VNG or blinear).
BTW, the raw images have not been bias or dark reduced. They are mere 5-minute images that I am trying to see how well I can process them in their raw state. Until I can convince myself that I have the correct Bayer, I don't want to be producing loads of wrongly deBayered images for processing.
The GRBG file indeed looks to be correct when looking at the colors produced; it has the usual light pollution signature expected from a dataset that has been color balanced.
FWIW I was puzzled when the first message came up telling me that the (single unstacked) image appeared to have 'stacking artefacts'.
The stacking artifact detector looks for unusal geometric patterns at the borders - the zipper artifacts can indeed trigger a warning (note the word "possible" in "possible stacking artefacts detected" :) )
The manual that I believed was a new one turned out to (apparently) be an older one with several sections labelled 'unwritten' or equivalent. Unfortunately none of the versions appear to be labelled with the version in the title, resulting in having to download every manual found and then go by the number inside. Could the manuals be clearly labelled please? I currently have several different versions short-cut on my desktop.
The manual linked in the Download section (as well as by clicking on the PDF icon in the "transform" tool bar will always give you the latest version of the manual/content).

Would you be open to letting me use this data as an example in the manual of how StarTools highlights latent patterns? It would be quite useful to other people as it is such a good example. Credits for acquisition to you of course!
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply