DSS set black point to zero now automatic

[decay]

@Startrek Hi Martin, I'm not sure about your last post. Maybe something went wrong?

[decay]

Addendum and correction to my post above: ASTAP makes use of both libs, DCRAW and LIBRAW. A (modified) version of a LIBRAW tool named "unprocessed_raw" is used to transform RAW files to FITS format for importing into ASTAP. This is written down in the ASTAP documentation as well.

This "unprocessed_raw" tool "generates unprocessed RAW images without black subtraction" as documented in its source code:

2023-11-01 10_35_40-D__tmp_astap-program-code_libraw_mod_unprocessed_raw.cpp - Notepad++.jpg (61.3 KiB) Viewed 15364 times

So ASTAP behaves like DSS having SBPTZ box always checked.

Best regards, Dietmar.

[decay]

as well as googling Pixinsight and "no black point correction" which is their version of the exact same issue.

It's all about the extraction from RAW, by way of the options that can be fed into DCRAW or whatever. But DSS' terminology lends itself to confusion, IMO.

PI explains that you want the "no black point correction" option (and I believe it is a default setting) in order to get your pure RAW data. Or at least as pure as one can get out of a DSLR. If one does choose to use black point correction, then DCRAW will subtract off some form of the bias level, using a value from the DSLR's data (if so recorded and written to the RAW file). One does not normally want that. Instead, you want to do a full calibration yourself using more precise sets of bias, darks, etc.

In DSS, SBPTZ is the same as PI no black point correction. Zero = "no." Meaning zero (nothing) gets subtracted off of your RAW files by DCRAW.

So, if using DSLR, I imagine you would almost never want to uncheck SBPTZ.

Yeah, that's it, Mike. I agree with all this.

I suppose you could uncheck it and try to do a lights - flats only calibration, as DCRAW will then subtract off the camera's estimated bias level?

I'm still not sure about this, too. I remember I read about this, but I will have to check and rethink this.

I can also see now why you might not want this to be fully automated. Defaulted to checked, yes. But, especially if I was trying to do some kind of unique calibration for a particular circumstance, I would be wary of DSS unchecking SBPTZ behind the scenes based on an algorithm (i.e. trying to out-think me) and having things go haywire.

Again agreed - and this is almost what I wrote ealier in this thread.

Well I still haven't found a totally clear, nuts and bolts explanation of things. [...] But this is a DSLR RAW issue only, so nothing that needs to be accounted for with astrocam FITS.

Both true. So what do we do now? Most people here are using dedicated astrocams these days, not DSLRs. So it's kind of irrelevant from that point of view. Besides, it's still not "completely clear" as you wrote, and I'm afraid that's not enough to get into a discussion with the developers of DSS One option might be to wait until someone here encounters a problem. Having an example would make it much easier to open a thread in the DSS forum.

Best regards, Dietmar.

[decay]

Hi Martin,

I noticed Han has not included a checkbox for “setting black point to 0” so like Mike mentioned previously it maybe a “behind the scenes” process based on type of DSLR used and it’s RAW data

I think we can now be pretty sure that this is not the case. (Please see my post above).

From my experimentation with Mono data sets in ASTAP so far , you must use the full suite ie Darks ( or Bias ) Flats and Flat Darks ( or Bias ) otherwise your Flats don’t to the job , I ended up with an inverse Vignetting when I left the Darks tab empty. Han confirmed with me on the ASTAP forum that I should use my Bias in both Darks tab and Flat Darks tab, which I do now.

I stumbled across exactly the same thing when I switched to ASTAP a year ago or so. DSS is a bit smarter in this regard as it automatically uses Bias to get the calibration equation right in case of missing Darks. I think.

Getting back to ASTAP not having an option for the black point to 0 setting , the full Calibration suite might be the answer to it Yes/ No / Maybe ??

I'm still not sure. This might be a different story or at least only a part of it? But I still haven't got the point when one would intentionally switch this box off and use camera black subtraction. Maybe that's the missing key for us. I will have to read and think about it again.

Best regards, Dietmar.

[Mike in Rancho]

DSS is a bit smarter in this regard as it automatically uses Bias to get the calibration equation right in case of missing Darks. I think.

I wouldn't go quite this far. It's maybe more of an accident, or just the way DSS goes about things that you can set up a bias-only (to be used on both sides) calibration with ease.

It has always been the case that if you put something in the bias tab in DSS, that master bias will be subtracted off of everything. That's just how DSS rolls. You can see it on the DSS website, on the page where the three calibration option (there are more) charts are shown.

Thus, to get L-B / F-B, you only need to load lights, flats, and bias. You could also dup the bias and load all of them instead into the darks and dark flats tabs. This will get you L-D / F-DF, but since both D and DF are B, the end result is really L-B / F-B. Same thing.

It's just a matter of knowing how DSS works. It can make things easier, or it can also go terribly wrong, if you misunderstand it and set up your calibration without knowing all that - especially if reusing old masters.

----------------------

Anyway, moving on....

I pulled up a few of my old D5300 NEF's in XnView and used the EXIFtool to check out the metadata. It is indeed written in by Nikon that the black level is 600 (14 bit).

So I did some experiments in DSS, utilizing both a light frame and an old bias frame. If you load it into the lights tab (just one file), register it (regardless of actual stars), and then run "stacking" with settings to output intermediate calibrated/debayered files, and then discard the autosave, you will get your original file back - except - it will have been converted to 16-bit fits. Because I didn't load any calibration files, no calibration is done, even though it now says xxxx.cal.fts. Again I did this with both a light and bias exposure, and with SBPTZ and no SBPTZ.

There is very much a difference, as if SBPTZ is unchecked, DSS (actually DCRAW I believe) will simply hack 600 off the bottom of the file.

That's a big enough difference that you might even be able to visually notice it in the linear display.

You can also see the differences if you then open those files in PI or whatever else and run statistics. I did that, and also opened them in ASIFITSVIewer and turned on the histogram.

The histogram and the PI statistics showed the SBPTZ output with means up in the 2400+ range (4x as the conversion to 16-bit), whereas the no SBPTZ files were down near 0. So things work as we have surmised.

And indeed it is potentially troublesome. I know from the past that my D5300 bias/dark level is actually 598 for the mean. The histogram has some pixels lower, down to maybe 580, and some are higher, to maybe 620. Typical distribution. But, if SBPTZ is unchecked that bias frame just gets brutally clipped when converted from RAW. And that renders the "precision" of using actual bias worthless. Theoretically the same could happen with short darks and dark flats.

Try it out, it's very enlightening. Or more accurately and literally, darkening.

My conclusion therefore is that SBPTZ in DSS when using RAW DSLR should always be checked, unless you are trying to do something unique and know what you are doing it for.

You may still get a result out of DSS when not using SBPTZ (a lot of stuff will wash out, and we are also talking detail that is buried way down near the bias level), but I do not think it will be optimal.

I have not yet tried lights and flats only stacking with SBPTZ unchecked. But, based on the stats and histogram experiments I ran, I think it would by and large "work." Instead of actual darks, or dark flats, or bias, you would instead be "calibrating" using the camera's dark level subtraction at every pixel. In effect it would be a synthetic dark/bias or global pedestal removal. In my D5300's case, that is 600 ADU.

[decay]

I also did some tests with DSS last evening but it has been more confusing to me instead of enlightening. I ran into exactly this pitfall:

or it can also go terribly wrong, [...] - especially if reusing old masters.

I reused an old Bias master and DSS crashed It drove me nuts. I will have to take new Bias frames first.

I will hopefully have some time at the weekend - it's going to be rainy and windy. No way to do AP or to get some work done outside. And then I will respond in detail.

What I already can contribute: I tried calibration with Flats only and in this case with SBPTZ being checked it resulted in inverse vignetting!

2023-11-03 07_22_46-Window.jpg (165.96 KiB) Viewed 15135 times

[Mike in Rancho]

What I already can contribute: I tried calibration with Flats only and in this case with SBPTZ being checked it resulted in inverse vignetting!

Cool, Dietmar!

That would be expected behavior, no? You have calibrated light-flat without any correction of dark level on either side, thus reverse vignetting. As maybe described by Han in the ASTAP link provided by Martin.

I quickly did the same and continued the test by unchecking SBPTZ for a comparison, thus using the "synthetic" camera dark level as subtraction from the RAW files, in this case the light and the flat.

Here are ASIFITSView autostretches of both. This is just one 10-second light calibrated with one flat, both originally RAW NEF's from the f/s D5300 with no corrections made (such as by Mark Shelley's ring remover).

SBPTZ:

Light cal w flat only DSS SBPTZ.jpg (103.16 KiB) Viewed 15056 times

No SBPTZ:

Light cal w flat only DSS NOSBPTZ.jpg (299.09 KiB) Viewed 15056 times

So at least in the case of this particular DSLR, it would seem that unchecking SBPTZ causes synthetic dark/bias (black level) subtraction, and would let you get away with making a stack even without separate darks, dark flats, or bias. The result would not be a perfect and precise calibration, but it's in the ballpark and probably corrects vignetting and dust spots to some degree.

[decay]

You have calibrated light-flat without any correction of dark level on either side, thus reverse vignetting.

Agreed. I was still being frustrated about the crashes and haven't analysed that yet this morning. Just wanted to tell.

So at least in the case of this particular DSLR, it would seem that unchecking SBPTZ causes synthetic dark/bias (black level) subtraction, and would let you get away with making a stack even without separate darks, dark flats, or bias. The result would not be a perfect and precise calibration, but it's in the ballpark and probably corrects vignetting and dust spots to some degree.

Yeah, that's a poor man's calibration. But why not.

I already/also did that test (light-flat calibration) with the current version 5.1.3 of DSS (which lacks the SBPTZ box) and I can report that this automatic 'virtual' de-selection of SBPTZ seems to work fine. No inverse vignetting. That's good, I guess

So my next attempt would be to introduce bias frames and do that tests again. In this case it should work fine with SBPTZ checked, right? But first I will have to take new bias frames. I haven't managed that yet.

BTW: Sky here is surprisingly and suddenly clear for a few hours. And I still missed to get my annual image of Saturn. But I'm too tired. And too lazy

[decay]

nd continued the test by unchecking SBPTZ for a comparison, thus using the "synthetic" camera dark level as subtraction from the RAW files, in this case the light and the flat.

I forgot to mention: I did that yesterday as well and it worked fine for me, too.

So, just for clarification: This is right now the only case we can think of where it is useful (or even necessary) to uncheck SBPTZ, right?

[decay]

Here are three stacks
- DSS 4.2.6 with SBPTZ checked and without
- DSS 5.1.3

All calibrated with flats and bias. (I hope the other DSS settings are identical.)
I haven't looked at them yet. Maybe tomorrow.

https://c.web.de/@334960167135216273/Op ... Gkr-lgrzgA