Can CHKDSK be RAM-constrained?

Jingle · January 27, 2015, 12:06:42 AM

CHKDSK in Windows 7 chews up RAM. More than a minor annoyance if you want to verify NTFS correctness on a system that is in use.

My understanding (I haven't tried it yet) is that Process Lasso can limit how much RAM a process can commandeer. If Process Lasso is set to limit CHKDSK to say 500 MB does anyone know if this will cause problems/instability either with CHKDSK or any other part of the OS?

Jeremy Collake · January 27, 2015, 10:38:27 AM

You can not limit virtual memory usage of a process. If you were to deny a process access to virtual memory, then it would crash (because it needs it for something).

You can, however, under some circumstances, limit the amount of RAM a process is allowed to have in it's working set. That is, the amount of memory that's in RAM instead of the page file. However, this is not always possible, as sometimes *all* the RAM is being actively referenced (needed).

So, presently, you would set a Watchdog rule (see Options menu) to trim the virtual memory of chkdsk.exe when it reached 500MB (or whatever).

This functionality is under active development, so will be improving. You won't have to fiddle with a watchdog rule.

Jingle · January 27, 2015, 03:16:01 PM

Quote from: Jeremy Collake on January 27, 2015, 10:38:27 AM
You can, however, under some circumstances, limit the amount of RAM a process is allowed to have in it's working set.

Does that mean that most of the time a process' working set of RAM *cannot* be limited? And is there a way to determine if a process can have it's working set of RAM limited?

Quote from: Jeremy Collake on January 27, 2015, 10:38:27 AM
You can not limit virtual memory usage of a process. If you were to deny a process access to virtual memory, then it would crash (because it needs it for something).

[snipped]

So, presently, you would set a Watchdog rule (see Options menu) to trim the virtual memory of chkdsk.exe when it reached 500MB (or whatever).

The above two statements look contradictory to me. First statement says, "You cannot limit virtual memory . . .". Second statement says, "set a . . . rule . . . to trim the virtual memory . . .". What am I not understanding?

Quote from: Jeremy Collake on January 27, 2015, 10:38:27 AM
This functionality is under active development, so will be improving. You won't have to fiddle with a watchdog rule.

If we don't set a Watchdog rule I imagine we'd still want to set a limit in some other manner. How would this be less fiddling?

Thanks for your help.

Jeremy Collake · January 27, 2015, 03:20:33 PM

Virtual memory is not RAM, it is an abstraction layer. Virtual memory is what programs allocate and deallocate. The OS decides if that memory is in RAM or the page file.

I mean request for new virtual memory allocations can NOT be denied (hard limit on allocation), but trimming virtual memory to clear the working set is possible to reduce the RAM footprint, *if* those pages are not immediately needed. As soon as the trimmed memory pages get accessed, they get paged back in you see.

Jeremy Collake · January 27, 2015, 03:56:42 PM

Quote
If we don't set a Watchdog rule I imagine we'd still want to set a limit in some other manner. How would this be less fiddling?

I mean it will be a more targeted and easy to use mechanism. The watchdog is robust, offers lots of features. This will be dedicated.

Jingle · January 27, 2015, 07:07:50 PM

Is there any risk in trimming virtual memory? IOW, if a process needs what is being stored in virtual memory and virtual memory is trimmed is it possible the process will halt?

Jeremy Collake · January 28, 2015, 04:58:45 AM

Quote from: Jingle on January 27, 2015, 07:07:50 PM
Is there any risk in trimming virtual memory? IOW, if a process needs what is being stored in virtual memory and virtual memory is trimmed is it possible the process will halt?

The process may briefly pause as it pages back in the memory it actively needs. That is the 'risk', and one of two reasons why most dumb RAM optimizers do more harm than good.

BUT, those pages may not have made it all the way out to the disk, they might still be in the RAM, in the disk cache. If so, they are quickly paged back in. That's called a 'soft page fault'.

A 'hard page fault' is when the backing page file has to actually be referenced.

Most page faults are soft.

Users with no RAM constraints should NOT use SmartTrim or any other RAM optimizer.

Jingle · January 28, 2015, 10:58:20 AM

Thanks for all of your help so far.

Why can't we simply tell the OS to impose a hard limit on how much RAM it can allocate to certain processes?

When I use a system with 2 GB of physical RAM the OS doesn't allocate as much RAM to the same processes as it does when there is 8 GB of physical RAM. This makes it look like the OS has the ability to explicitly limit RAM per process.

Jeremy Collake · January 28, 2015, 11:03:13 AM

Because what happens when the process needs to allocate more virtual memory, but can't get it? You think it has a mechanism to 'wait around' to do whatever it needs it do? NO, it crashes or generates an exception. That's why denying allocations can't be done.

Ideally, you have enough RAM to allow your system to adequately handle the needs of your applications. Virtual memory allows you to use more 'memory' than you actually have physical RAM. That's important, because, as said above, processes *must* get their allocations.

BenYeeHua · January 29, 2015, 09:03:04 AM

So, are we talking about virtual memory or resident set size or working set or?

QuoteWhy can't we simply tell the OS to impose a hard limit on how much RAM it can allocate to certain processes?

When I use a system with 2 GB of physical RAM the OS doesn't allocate as much RAM to the same processes as it does when there is 8 GB of physical RAM. This makes it look like the OS has the ability to explicitly limit RAM per process.

Because the memory has been TRIM by the OS into Page File, that's why you saw much less RAM has been taken by the same processes.
Try disable your Page File and you will know the true.

---
Except, the dev are letting the software decide how much it should be take, based on the physical memory, and release the memory when it is taking too much.
But, except it is a browser(like Firefox, they will release the Picture that's not showing on the screen, when it is not enough memory), normally they will just choose to TRIM the processes to let them look like "smaller".

Jeremy Collake · January 29, 2015, 04:49:34 PM

To summarize:

Yes, and that is what 8.0 will offer - the ability to enforce a 'page out' operation at point X. That can help to limit RAM, in some cases, to a certain degree.

Yes, that is the same capability that already exist as part of the Watchdog feature.

It's just a different mechanism of exposure, and implementation.

Normally it is the OS that makes this decision. So when you use this feature, you are taking control, assuming you do know better, as BenYeeHua mentioned.

DeadHead · January 30, 2015, 12:03:57 PM

Question, and slightly off topic.

Do the typical windows application understand the difference between ram and swapfile, or is virtual memory all that's available (and that's all that's seen to generic apps), leaving all the decision about memory handling to windows itself?

Many times have I wondered, and this is especially true when it comes to gaming, why some programs seems to have this 'urge' to use the harddisk no matter what, even if I have enough ram to keep the program/game in its entirety i ram, and even though on some occasions I've just played that particular level.

Jeremy Collake · January 30, 2015, 12:07:35 PM

Quote from: DeadHead on January 30, 2015, 12:03:57 PM
Do the typical windows application understand the difference between ram and swapfile, or is virtual memory all that's available (and that's all that's seen to generic apps), leaving all the decision about memory handling to windows itself?

Many times have I wondered, and this is especially true when it comes to gaming, why some programs seems to have this 'urge' to use the harddisk no matter what, even if I have enough ram to keep the program/game in its entirety i ram, and even though on some occasions I've just played that particular level.

No. They have no awareness. To them they just see virtual memory. (there are technical rare exceptions, but keeping things simple)

Some programs, do, however, page themselves out from time to time to try to reduce their perceived footprint. That may be what you see in some problematic cases. Or perhaps there is an external actor that forced a trim on the process. This is assuming the disk access is indeed paging activity.

DeadHead · January 30, 2015, 12:30:35 PM

Quote from: Jeremy Collake on January 30, 2015, 12:07:35 PMThis is assuming the disk access is indeed paging activity.

When it comes to games, that are mostly to this day 32-bit, I don't think it's paging activity, but rather "safe" memory handling if you will.

I just wish Windows file caching could be more intelligent, or perhaps more customizable. I remember one particular mmo I used to play - Vanguard. That game was the ultimate 'disk trasher game', that would constantly load stuff from the disk more or less all the time, even though I could have placed the game in it's entirety in ram if I would have wanted to (I have 24 gigs of ram, game folder of that particular game was about ~16 gig).

This is an area I feel could improve a lot when it comes to Windows gaming - if the ram is there, by all means feel free to use it!

Jeremy Collake · January 30, 2015, 01:05:42 PM

QuoteWhen it comes to games, that are mostly to this day 32-bit, I don't think it's paging activity, but rather "safe" memory handling if you will.

Yes, it may very well be a difference in the way Windows is treating the process if it's a 32-bit WOW (Windows32OnWindows64) process.

Microsoft is not into optimizing their current stuff. They are into engineering new things for you to buy. They'd rather you toss that old machine on eBay

QuoteI just wish Windows file caching could be more intelligent, or perhaps more customizable.

AFAIK, there's no way to improve upon Windows disk caching intelligence through customization (efficiency). But I will look into if. If there is a way, I'll find it.

BenYeeHua · January 30, 2015, 03:19:33 PM

At least you will get 3.x GB for 32-bit process with 64-bit OS, and many new game start support 64-bit now, as they are hitting the limit of 32-bit, and always need to careful about the virtual limit when 32-bit version are running.

For the cache, ya, it is hard to tell OS cache the file that you want, but normally OS will cache the file after your game has loaded it.
But the problem is, it need to be access a few times, then it will be cached into the RAM.

QuoteAFAIK, there's no way to improve upon Windows disk caching intelligence through customization (efficiency). But I will look into if. If there is a way, I'll find it.

Maybe create a functions to just read all the files inside the game folder?
This is not the best solution, but it should works!

PS:Based on my exp, you may need to read it a few times(at least 3), before OS finally know it need to be cached.
---
I forgot, if your game provide Validate function, you can use that to force cache the file before you start the game, but be aware that, if you are using HDD, it is hurt for using other software while the OS are "caching" it.