Limit memory use

Requests for new features and wish-list items.

Limit memory use

Postby jimmy101 » Thu Feb 20, 2020 9:32 am

Drizzel images are almost unusable in startools as the software uses all ram/swap, but it will/does crash the linux 64bit system. We really need a setting to limit ram/swap use to 90%, or startools creates a virtual swap file if it runs out of memory. Normal images still use alot of memory, but is fine. Just these larger images are killing me with crashes! Cheers.
jimmy101
 
Posts: 28
Joined: Tue Apr 16, 2019 4:25 am

Re: Limit memory use

Postby admin » Thu Feb 20, 2020 11:23 am

It is highly recommended to use (or add) a dynamic swap space to your Linux install.

For example, SwapSpace is mature utility that creates a ‘dynamic swap file’. It grows (or shrinks) depending on the requirements of the OS.
Note that it also is able to reduce the space it takes up if swap is not needed. E.g. this behaviour is similar to Windows' pagefile.sys/swapfile.sys files.

Dynamic swap is now the default in most recent Linux distros, but if your distro is not one of them, I'm sure it's possible to bolt it on yourself (e.g sudo apt-get install swapspace or similar).

StarTools is indeed an unapologetic memory hog and if you process large files, you will definitely want to increase your system's RAM to avoid swapping altogether.
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
 
Posts: 2212
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne

Re: Limit memory use

Postby firstLight » Thu Feb 27, 2020 8:58 pm

As I am Linux only since 1993, I simply want to confirm and support Ivo's suggestions for sufficient memory and swap space. I did not take this into account when I started with StarTools recently and experienced many crashes for no obvious reasons. An strace log clearly showed memory related issues. I was quite clueless until Ivo suggested to provide more swap space on my computer.

Conclusion: Swap space! Just do it! ... and enjoy! :)
< = > less is more
User avatar
firstLight
 
Posts: 2
Joined: Thu Feb 27, 2020 1:17 pm
Location: Germany

Re: Limit memory use

Postby jimmy101 » Sun Mar 01, 2020 10:39 pm

Thanks all. Yes I have alot of swap so I'm not getting crashes. But I raise this issue because it's not apparent to a new user and even if you think you have enough RAM/swap but then drizzle you run out and then you crash wasting hours of processing time. The software really shouldn't allow the system to crash.
jimmy101
 
Posts: 28
Joined: Tue Apr 16, 2019 4:25 am

Re: Limit memory use

Postby admin » Sun Mar 01, 2020 11:52 pm

jimmy101 wrote:Thanks all. Yes I have alot of swap so I'm not getting crashes. But I raise this issue because it's not apparent to a new user and even if you think you have enough RAM/swap but then drizzle you run out and then you crash wasting hours of processing time. The software really shouldn't allow the system to crash.


I believe you are misunderstanding the cause of the "crashes". It is actually your operating system unceremoniously killing StarTools through the OOM killer; StarTools has no say in this. As an application (not OS!) developer, at some stage, I have to trust the user to have configured their operating system and hardware correctly for the task they are trying to run. When it comes to virtual memory, Windows and macOS tends to not be a problem, nor are post-2017(-ish) Linux distros. They are all set up to use dynamic swap space.

If you wish to run Linux, the old adage 'with great power comes great responsibility' applies. Many Linux distros are not meant or (primarily made) to be consumer-friendly. You are given tight control over every aspect.
I'm sure you will agree the onus is on the user to make sure the hardware and operating is configured to be fit for purpose, not the application developer.
Ivo Jager
StarTools creator and astronomy enthusiast
User avatar
admin
Site Admin
 
Posts: 2212
Joined: Thu Dec 02, 2010 10:51 pm
Location: Melbourne


Return to Feature Requests

Who is online

Users browsing this forum: No registered users and 3 guests

cron