Discussion idea - giving StarTools stdev data from stack?

General discussion about StarTools.
khyperia
Posts: 23
Joined: Wed May 21, 2014 12:36 pm

Re: Discussion idea - giving StarTools stdev data from stack

Post by khyperia »

admin wrote:IMHO, for the UI bit, try keeping UI and functionality as seperate as possible, so that you can 'skin' the core of your aplication any way you want at a later stage. The actual process should, again IMHO, respond gracefully to aborting *at any time*, output an error code if aborted and, while running/doing its thing, output some sort of meaningful message to the host (UI) application that represents a 'progress report', which the host application can then relay in any way it sees fit to the user (e.g. a nice progress bar). The strategy is 'start ugly', make it work, and refine the interface as you go along.

First and foremost, you're tangibly improving people's joy in the hobby by making it easier to obtain better results for more people than ever before. Don't underestimate that impact!
Again, KISS - keep it simple, stupid. Do a few things and do them well. Expand later.
In application and UI design, apply the 80/20 rule, which goes like this; optimise your application for the 20% of functionality that people will be using 80% of the time when they fire up your application. If any stuff in your application doesn't fit the 80/20 rule, you hide it in a sub menu, some preset button or give the user some other means to accomplish the 'more exotic' task.
Apart from delivering better results, don't underestimate the need for convenience. If some core thing/process takes too much time/clicks to accomplish, you'll lose users (e.g. why should they use your application over, for example, DSS?).
Given the state of some of the stackers out there and the strong evidence that people struggle with these applications, you have a lot of leeway to improve the state of the art.
Hrmmm. I've been digging around quite a bit, and I haven't found a C++ gui library that I like yet. SDL has been the most promising yet, but I'd rather not build an entire UI toolkit from scratch. Qt forces you to use their own compiler and toolchain and is a massive install, and distribution is really tricky. GTK is a clusterbomb of dependencies and crazyness. I'm considering just writing the GUI frontend application in Java, or something with a cross-platform UI toolkit built in (because eww java), and just boot off my C++ executable process when the "invoke" button is pressed.

There's one idea floating around in super wishful land, though. I have no idea what StarTools looks like on the inside, or if this is even remotely possible, but I figured I could throw it out there anyway: A StarTools plugin system.
Basically, I would make a .so/.dll plugin that StarTools would load with dlopen/LoadLibrary, and then it'd be something like another button on the home screen dynamically added via the plugin that says "Stack". The huge upside to this for the end user is that there are no intermediate files, *and*, the whole standard deviation second file wackyness is completely hidden from the user (well, maybe informing them it exists and stuff) - the data would go right to what looks like a "normally opened file" with tracking enabled. The huge upside for me is that I might be able to re-use some of ST's GUI widgets (sliders, progress bars, file open window, etc.) instead of doing it all myself :)
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Discussion idea - giving StarTools stdev data from stack

Post by admin »

khyperia wrote: Hrmmm. I've been digging around quite a bit, and I haven't found a C++ gui library that I like yet. SDL has been the most promising yet, but I'd rather not build an entire UI toolkit from scratch. Qt forces you to use their own compiler and toolchain and is a massive install, and distribution is really tricky. GTK is a clusterbomb of dependencies and crazyness. I'm considering just writing the GUI frontend application in Java, or something with a cross-platform UI toolkit built in (because eww java), and just boot off my C++ executable process when the "invoke" button is pressed.
The latter makes sense (including 'eww Java' ;) ) Seperate logic from your UI. At the end of the day, that's what happened with dcraw as well - so many front ends for that (including web-based ones)! Someone else will come along to build a 'better'/different front-end (including StarTools and/or The GIMP!). Get the logic and real magic working first!
Ivo Jager
StarTools creator and astronomy enthusiast
waynef
Posts: 8
Joined: Thu Oct 23, 2014 4:36 am

Re: Discussion idea - giving StarTools stdev data from stack

Post by waynef »

Just another vote from my side for further development on this I would def have a interest and use for this
waynef
Posts: 8
Joined: Thu Oct 23, 2014 4:36 am

Re: Discussion idea - giving StarTools stdev data from stack

Post by waynef »

I am having difficulties with the windows file.If I open it it just quick pops up a command prompt window and then closes it again.The basic command prompt version works fine.This is properly just a problem with the user not understanding the input but I would appreciate if someone can clarify it for me as I am really keen to experiment with this
khyperia
Posts: 23
Joined: Wed May 21, 2014 12:36 pm

Re: Discussion idea - giving StarTools stdev data from stack

Post by khyperia »

waynef wrote:I am having difficulties with the windows file.If I open it it just quick pops up a command prompt window and then closes it again.The basic command prompt version works fine.This is properly just a problem with the user not understanding the input but I would appreciate if someone can clarify it for me as I am really keen to experiment with this
I'm a bit confused. "Basic command prompt version works fine"? I'm pretty sure I didn't release a "basic" version, just a python version, c++ version, and c++/deepskystacker version. I assume you're wanting to use the c++ version, because that's where the meat of the algorithm is and such. http://www.startools.org/forum/viewtopi ... 7&start=30 is the post in question, top of the page... the directions on how to use it are in the post! Let me copy over the relevant bits:
Same with the python version, it's a command-line-only app, usage is something like this:
AstropressPlus.exe --out path/to/output.fits --outstdev path/to/outputStandardDeviation.fits path/to/images/*.fits
Only .fits files are supported at the moment.
I've bolded the important stuff - note that there is no GUI version, so doubleclicking the exe will do absolutely nothing.

The C++/deepskystacker version does in fact have a file selector interface, but as the thing isn't really doing much useful (yet, until there's some way to input stdev into ST), it's kind of pointless.
waynef
Posts: 8
Joined: Thu Oct 23, 2014 4:36 am

Re: Discussion idea - giving StarTools stdev data from stack

Post by waynef »

Thanks for the clarification.My apology for the confusion caused by using the wrong terminology. I guess I did not interpret the post correctly and was expecting a GUI version
Rowland
Posts: 235
Joined: Sat Mar 05, 2011 10:54 pm

Re: Discussion idea - giving StarTools stdev data from stack

Post by Rowland »

I know this is an old post. I have been otherwise detained. This is wonderful. You guys are brilliant - I see a Nobel Prize coming up sometime. Technical stuff is over my head, but I get the gist of things...

Should I mention the non-linearity of DSLR RAW data. I'm sure you have thought of this along the way? Craig Stark's article on the non-linearity of DSLR RAW data. The solution, practically speaking is quite simple if you don't worry yourself too much about what camera manufacturers actually do to the data in camera. To that end, I made a few intuitive assumptions - right or wrong? The end result, no data truncation, typical of linear approaches to DSLR RAW data preprocessing.

In basic terms don't fiddle with the out of camera data, except for the requirements of master flat creation and the necessary pixel rejection on calibrated light frames, because they are exposed to rays.

If all of this is rubbish, let me know. Right now I am achieving the best results ever with cooled DSLR RAW data.

In the interests of brevity, the following work flow for DSLR RAW data produces scaling and weight factors very close to 1, which I understand is close to ideal. That is, DSLR RAW data is better not scaled. To that end calibration is simple, as follows;

Take bias darks and flats...

Stack dark and bias frames - average with no normalization and no pixel rejection.

Subtract master bias from flat and stack - average, normalization multiplicative, pixel rejection and equalize fluxes.

Subtract master dark from light subs - bias is not subtracted from darks or lights - bias in the dark method.

Calibrate lights with master flat

Stack calibrated light subs.

The terminology should be recognizable of another cross platform program which I use for preprocessing - PI.

The rationale is that in camera modification of DSLR RAW data moves it away from traditional 16bit linear CCD data and as such DSLR RAW cannot be processed as for linear data without truncation. I have found the above work-flow less destructive and more reliable.
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Discussion idea - giving StarTools stdev data from stack

Post by admin »

Yes, Rowland, that makes total sense!

Now the idea is that StarTools will have a much earlier handle on the quality of the signal per-pixel. For example, a pixel with a large bias should be deemed of lower quality. Likewise for a pixel whose signal has been boosted a lot by the application of a flat frame. And, as already demonstrated, the variability of a pixel's value between sub frames (as measured by a standard deviation or MAD) is another per-pixel measure of noise vs signal. Right now, StarTools assumes that all pixels are equally noisy when it starts Tracking. This is definitely not the case. But since all stackers throw away any such data, it's the best StarTools can do at the moment.
Ivo Jager
StarTools creator and astronomy enthusiast
Post Reply