Mac 1.9.542alpha: Crash when Sharp Module is loaded

Bugs, glitches and unexpected behaviour.
cytan299
Posts: 71
Joined: Tue Nov 15, 2016 11:49 am

Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by cytan299 »

Hi Ivo,
I want to report a crash on StarTools 1.9.542alpha when I load the Sharp Module. I've attached the crash log from Mac and the StarTools.log file in the crash_log.zip file.
I can send more info if you need it, like the file that I've been processing.

Thanks!

cytan
Attachments
crash_log.zip
(34.01 KiB) Downloaded 207 times
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by admin »

Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...
Ivo Jager
StarTools creator and astronomy enthusiast
cytan299
Posts: 71
Joined: Tue Nov 15, 2016 11:49 am

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by cytan299 »

admin wrote: Mon Feb 27, 2023 2:48 am Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...
Yes, ST 1.9.542alpha crashes immediately when I load Sharp Module. It did this twice and ST1.9 crashes at the same point when loading the Sharp Module, that's the reason why I posted this report. For comparison, ST 1.8.527MR2 does not crash when I do approximately the same sequence of processing actions.

cytan
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by admin »

cytan299 wrote: Mon Feb 27, 2023 3:07 pm
admin wrote: Mon Feb 27, 2023 2:48 am Hi Cytan,

Thank you for reporting this, as well as for attaching the very helpful logs!

It appears the crash happens in the graphics driver, which may be system-specific.
Are you able to reproduce this 100% of the time? If so, does the problem happen when first binning the dataset?

I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements...
Yes, ST 1.9.542alpha crashes immediately when I load Sharp Module. It did this twice and ST1.9 crashes at the same point when loading the Sharp Module, that's the reason why I posted this report. For comparison, ST 1.8.527MR2 does not crash when I do approximately the same sequence of processing actions.

cytan
Thank you. Does the problem happen when first binning the dataset?
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by admin »

I just uploaded 1.9.544. It was missing a text string for an empty mask condition. Does this solve your issue?
Ivo Jager
StarTools creator and astronomy enthusiast
cytan299
Posts: 71
Joined: Tue Nov 15, 2016 11:49 am

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by cytan299 »

admin wrote: Mon Feb 27, 2023 11:38 pm I just uploaded 1.9.544. It was missing a text string for an empty mask condition. Does this solve your issue?
1.9.544 also crashes when I start the Sharp Module. See attached zip file.

I don't bin, so I'm not sure what you mean in the previous message.

cytan
Attachments
crash_log2.zip
(19.1 KiB) Downloaded 203 times
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by admin »

cytan299 wrote: Tue Feb 28, 2023 1:48 am I don't bin, so I'm not sure what you mean in the previous message.
In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.
Ivo Jager
StarTools creator and astronomy enthusiast
cytan299
Posts: 71
Joined: Tue Nov 15, 2016 11:49 am

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by cytan299 »

admin wrote: Tue Feb 28, 2023 2:27 am
cytan299 wrote: Tue Feb 28, 2023 1:48 am I don't bin, so I'm not sure what you mean in the previous message.
In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.
I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan
User avatar
admin
Site Admin
Posts: 3367
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne
Contact:

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by admin »

cytan299 wrote: Wed Mar 01, 2023 1:52 am
admin wrote: Tue Feb 28, 2023 2:27 am
cytan299 wrote: Tue Feb 28, 2023 1:48 am I don't bin, so I'm not sure what you mean in the previous message.
In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.
I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan
Many thanks - this is an important datapoint! :bow-yellow:

Indeed, the engine code changes all the time. There are things like small performance tweaks/improvements by keeping data that is going to be re-used resident in GPU memory, etc.
25MP becomes quite big in memory; at worst, it can becomes 25MP * 64-bit values * 4 channels * 5 (L, R, G, B, NBAccent) datasets = ~4GB. You would typically keep multiple copies of that 4GB around for the algorithms (source, destination, modulator, etc.). Of course, I try to be frugal and not use multiple channels or datasets where I don't have to, so 4GB doesn't really happen. Math on the GPU happens in 32-bit rather than 64-bit so that saves some data too (it does however mean a "copy" of 64-bit translated to 32-bit is needed, so that's extra memory again, etc.).

There is a *lot* of data being shifted around and there is always a tradeoff between processing speed and memory requirements...
Ivo Jager
StarTools creator and astronomy enthusiast
cytan299
Posts: 71
Joined: Tue Nov 15, 2016 11:49 am

Re: Mac 1.9.542alpha: Crash when Sharp Module is loaded

Post by cytan299 »

admin wrote: Wed Mar 01, 2023 2:30 am
cytan299 wrote: Wed Mar 01, 2023 1:52 am
admin wrote: Tue Feb 28, 2023 2:27 am
In your log, I can see a rather large (25MP) dataset is being processed on the iGPU. It may be that the iGPU is getting overwhelmed by either the work load or the memory requirements. Binning would reduce the workload and memory requirements.

If the graphics driver does not crash when processing the reduced dataset, then that would indicate the diagnosis is an under-powered graphics solution. In that case, it would be best to force OpenCL to use the CPU instead.

If the graphics driver still crashes even with a reduced workload, we would have to search for the issue elsewhere.
I did what you suggested and did a bin 50%. Now, no crash in the Sharp Module!

However. what I don't understand is that the same 25 MB file with approximately the same processing does not crash at all in 1.8.527MR2. So, the Sharp Module processing has been changed in 1.9.544alpha?

cytan
Many thanks - this is an important datapoint! :bow-yellow:

Indeed, the engine code changes all the time. There are things like small performance tweaks/improvements by keeping data that is going to be re-used resident in GPU memory, etc.
25MP becomes quite big in memory; at worst, it can becomes 25MP * 64-bit values * 4 channels * 5 (L, R, G, B, NBAccent) datasets = ~4GB. You would typically keep multiple copies of that 4GB around for the algorithms (source, destination, modulator, etc.). Of course, I try to be frugal and not use multiple channels or datasets where I don't have to, so 4GB doesn't really happen. Math on the GPU happens in 32-bit rather than 64-bit so that saves some data too (it does however mean a "copy" of 64-bit translated to 32-bit is needed, so that's extra memory again, etc.).

There is a *lot* of data being shifted around and there is always a tradeoff between processing speed and memory requirements...
I'm glad that my datapoint helped. I hope this will be fixed in the later versions of ST1.9.

Just for completeness, the crash happens right at the start of Sharp Module when it is calculating the scales.

cytan
Post Reply