Inside memory priorities...

Started by bertie97, June 07, 2013, 03:27:00 AM

Previous topic - Next topic

bertie97

This feels like a dumb question but ...
The default memory priority setting - which is highest 0 or 7?

The ratings of 1 as 1st priority vs 0 having the lowest weight of importance have me conflicted. ::)

edkiefer

Quote from: bertie97 on June 07, 2013, 03:27:00 AM
This feels like a dumb question but ...
The default memory priority setting - which is highest 0 or 7?

The ratings of 1 as 1st priority vs 0 having the lowest weight of importance have me conflicted. ::)
that is good question , a guess would be 0 is low 7 is highest  as default is 5 .

here some info on memory priority but sadly it doesn't really tell what each value is .

http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx
Bitsum QA Engineer

bertie97

Thanks edkeifer that's a useful article.
I am/was leaning to 0 = lowest, but still ... would be nice to have confirmation.

Where's the resident expert BenYeeHua when I need him?!  ;D

BenYeeHua

Quote from: bertie97 on June 07, 2013, 08:55:07 AM
Thanks edkeifer that's a useful article.
I am/was leaning to 0 = lowest, but still ... would be nice to have confirmation.

Where's the resident expert BenYeeHua when I need him?!  ;D
Sleeping at that time, I think you forget I am a 夜猫, which is like to work on midnight. ;)
And when you asking about it, our supporter has been update this topic quietly. ;D
http://bitsum.com/forum/index.php?topic=2153.0
----TL;DR:You can just read the short Answer, which is using the Red color for find out easily.
I am wrong at the past. :P
There are 2 type of thing that I messing up for the Memory priority.

First, the Real Memory Priority.
QuoteEach thread has its own memory priority

  • 5: normal
  • 1: low
This determines which standby list is used for the page (when/if it arrives on the standby list)
Thread priority comes from process memory priority

  • Can be changed for process or individual thread
  • SetPriorityClass or SetThreadPriority “background mode”
http://ecn.channel9.msdn.com/o9/te/Europe/2010/pptx/wcl401.pptx

QuoteMEMORY_PRIORITY_VERY_LOW 1
Very low memory priority.

MEMORY_PRIORITY_LOW 2
Low memory priority.

MEMORY_PRIORITY_MEDIUM 3
Medium memory priority.

MEMORY_PRIORITY_BELOW_NORMAL 4
Below normal memory priority.

MEMORY_PRIORITY_NORMAL 5
Normal memory priority. This is the default priority for all threads and processes on the system.
QuoteThe memory priority of a thread or process serves as a hint to the memory manager when it trims pages from the working set. Other factors being equal, pages with lower memory priority are trimmed before pages with higher memory priority.
http://msdn.microsoft.com/en-us/library/windows/desktop/hh448387%28v=vs.85%29.aspx

Yes, the Memory Priority for a Process or Thread is a hint for memory manager.
When you Trim the Process Virtual MemoryWorking Set Memory, it will give the hint to memory manager, to let them know which StandBy List Page Priority it should give to that part of Pages.

At that time, you will see the working set of the Process is lowering down, while the StandBy List memory usage increase, as it is pulling the pages out of the working set, and put it into StandBy List.

And yes, the Virtual Memory remain unchanged, and I think it should be change the name for the PL trim function from "Trim Virtual Memory" to "Trim Working Set Memory"
That is the real thing that the system Trim for the process memory. :)
Quote"Standby List
The Standby list contains unmodified pages that have been removed from process working sets, which effectively makes the Standby list a cache. If a process needs a page that is on the Standby list, the memory manager immediately returns the page to its working set.
All pages on the Standby list are available for memory allocation requests. If a process requests memory, the memory manager can take a page from the Standby list, initialize it, and allocate it to the calling process. This is called repurposing a page.
Pages on the Standby list are often from recently used files. By keeping these pages on the Standby list, the memory manager reduces the need to read information from the disk. Disk reads can decrease system responsiveness."
Because you can't "Trim" the Virtual Memory, and put it into a file while reduce the Virtual Memory usage of the process. ;D
But you can Trim the working sets memory, and put that part of Pages into the standby list.

"Trim Virtual Memory" will make someone think that:"As the Game Process/Firefox has been crashed by using too much Virtual Memory, so if I using this PL function to trim the Virtual Memory, it will reduce the chance to crash. :D"
And this is wrong. :(
This function will reduce the leaked memory to using the RAM when you are out of RAM memory, as it is putting the leaked memory to StandBy memory, and follow the StandBy List Pages Priority to increase the chance to put the leaked memory into the PageFile first.
And also which process StandBy List Pages/Memory should be put into the PageFile first, if you has set the Memory Priority. :)
----
Second, Pages Priority(changed as it is not only apply to standby lists, but pages.)
QuoteIn Vista & later, there are 8 prioritized standby lists
Pages are removed from lowest priority list first

  • Low memory priority process will keep re-using low priority pages
  • Higher priority information remains cached

QuoteStandby List Population

Priority 7 come from a static set (pre-trained at Microsoft)

  • Pre-populated at each boot
  • Includes pages related to user input that requires fast responsiveness (right-click, desktop properties, control panel, start menu, etc.)
Priority 6 are pages that SuperFetch considers important, or useful (will rarely get repurposed)
Priority 5 are standard user pages (memory priority 5)
Priority 1 are low priority user pages (memory priority 1)
Priority 0-4 may be Superfetch decayed, cache manager read-ahead and pagefault clustering
http://ecn.channel9.msdn.com/o9/te/Europe/2010/pptx/wcl401.pptx

So, it is like this.(Sorry for copy-paste. ;D)

Priority 0-4
Superfetch decayed, cache manager read-ahead and pagefault clustering

Priority 5
Standard/Normal user pages

Priority 6
Pages that SuperFetch considers important, or useful (will rarely get repurposed)

Priority 7
Pages that has been set by Microsoft, for fast responsiveness.

If you check the RAMMap, you will found out there are empty in the Priority 0, as it is means that.
If the pages is put into Priority 0, it will be empty except it is active/driver locked etc.
PS:The repurposed in the RAMMap means how many amount of Memory/Ram has been remove from that Priority.
----Short Answer here :)
So, the real answer for the Memory Priority is.
1 is the lowest priority.
3 is the default priority for process that running from Task Scheduler.
5 is the default priority.
5 is also the highest priority.

But, for Pages Priority is.
0 is the lowest priority, that can be empty always, except active/driver locked.
0-4 is normally used for file that are reading(and read-ahead first)and file that are unused now(like it has been used before) and might be used for some software you will run(SuperFetch do that works), like game file which I don't running yet. ;)
5 is the standard/normal priority.
6 is the higher priority, which is SuperFetch found out it is useful(also means the files that read/write many times, like Firefox places.sqlite) and put them as 6
7 is the highest priority, set by Microsoft, you can always found out driver and system file dll inside.(Not Intel inside. ;D)
Like DirectX, Graphic Card driver, DWM, Antivirus, windows codec, IE, Office, .Net....

PS:5 hours for this thing, I think I need to improve my writing skill, this is not the first time for that. :o
----
So, I also think that.
We should be update the function of PL, by changing the Memory Priority to 1-5, and change the name "Trim Virtual Memory" to "Trim Working Set Memory". ;)

PS:Read this if you are interested about standby list.
http://www.pcwintech.com/about-other software
PS2:I also only do some research, just like that guy. :)
PS3:For some correction, it seen like the priority don't limited by Standby list, it is apply to Pages, so I changed it.

edkiefer

So, long story, don't mess with it, it gets complicated  ;D
Bitsum QA Engineer

BenYeeHua

Quote from: edkiefer on June 07, 2013, 07:23:46 PM
So, long story, don't mess with it, it gets complicated  ;D
But this is what I always do. ;D

And more simple one.
The memory priority for a process/thread is 1-5, and the default is 5(highest).
You can set the software that you think it is not important with lower memory priority, so it has bigger chance to paged out and giving more RAM to the other software, when your RAM is out of free memory.

Another one is Pages priority, which is out of our control, just ignore it as you want. ;D
It has priority 0-7, priority 0 is MUST page out/cleanup except some process/driver locked it.
5 is the standard/normal priority for pages, 7 is Microsoft keep for Microsoft's software and other driver that can't paged out.
0-4 and 6 is controlled by SuperFetch, 0-4 is also used for system to read the file/pagefile, and might be also for the write cache of copying file.

Finish. 8)
But I think maybe not.

edkiefer

The thing is once you get out of normal priority with memory, do you also set i/o or cpu priorities the similar values (lower or higher ) .
Bitsum QA Engineer

bertie97

Looks like I inadvertently gave you a lot of work BYH!!  :o
I guess this will make you an English language expert also, so not all bad. 
Not sure about the TZ, I had figured you were enjoying yourself away from computer madness.   ;D

BUT Thanks for the detailed feedback.  I think that the superfetch handling may be a key issue for me, I will have to start examining that.
I have been juggling I/O & CPU priorities for key tasks & I think there is a need to test RAM handling in that context, but 0-7 as a scale is inefficient.
This is because, as you show, the scale isn't rigid - like growing importance [number weight]; but more a grid - like a map, with priority pathing.  [this does fit the paradigm, so it's logical if not quite obvious.]

I was messing with DVD burning speeds & found dropping the memory priority from 5 to 3 has improved the speeds.  I'm not exactly clear on why unless this is a superfetch/cache variable that is helping stream data.
Your details seem to support that hypothesis, so I guess I will have to make a few more coasters & see what happens. ;)

BenYeeHua

Quote from: edkiefer on June 08, 2013, 07:01:00 AM
The thing is once you get out of normal priority with memory, do you also set i/o or cpu priorities the similar values (lower or higher ) .
If you think the process should be a background process and don't need to wasting too much CPU usage.(For example, bad coded software)
Then you can lower the CPU priority, and the i/o priority don't help much except the background process is accessing data from Hard Disk/RAM, and you has a slower Hard Disk/RAM for it.
(RAM is means as the data that has been cached by SuperFetch, and also the data/pages that you has TRIM the Working Set memory and put it into StandBy List.)

For Example, Defrag Software and Game Update that need the access of Hard Disk, then you can see the impact by the i/o priority.
Like, I set Defrag Software to Very Low, you can see it hang until the other software don't has to access the Hard Disk.
So don't set it too low. ;)

Quote from: bertie97 on June 08, 2013, 07:34:21 AM
Looks like I inadvertently gave you a lot of work BYH!!  :o
I guess this will make you an English language expert also, so not all bad. 
Not sure about the TZ, I had figured you were enjoying yourself away from computer madness.   ;D

BUT Thanks for the detailed feedback.  I think that the superfetch handling may be a key issue for me, I will have to start examining that.
I have been juggling I/O & CPU priorities for key tasks & I think there is a need to test RAM handling in that context, but 0-7 as a scale is inefficient.
This is because, as you show, the scale isn't rigid - like growing importance [number weight]; but more a grid - like a map, with priority pathing.  [this does fit the paradigm, so it's logical if not quite obvious.]

I was messing with DVD burning speeds & found dropping the memory priority from 5 to 3 has improved the speeds.  I'm not exactly clear on why unless this is a superfetch/cache variable that is helping stream data.
Your details seem to support that hypothesis, so I guess I will have to make a few more coasters & see what happens. ;)
I think TZ is means as Time Zone, GMT+8 here. :)
----
Nope, what I means is the Pages Priority 0-4 is normally set by SuperFetch, not that you set the memory priority to 3 and the data will be handle by SuperFetch.
And if you read this again(and this is what I do after read that you has improve for setting as 3)
QuoteThe memory priority of a thread or process serves as a hint to the memory manager when it trims pages from the working set. Other factors being equal, pages with lower memory priority are trimmed before pages with higher memory priority.

It can be 2 reason you can see the performance increase.
1.The burning software "watch" the working set, and the increase of TRIM reduce the working set, then the next data that need to be burn is cached first by software.
2.The burned data is TRIM to standby list, and the system/SuperFetch think it can read-ahead/cache more file.

I think you can watch the burning software process and the whole system standby list memory usage to know that I am right or wrong. ;)

bertie97

Yeah this is where the grid vs weight interpretation is coming in. 
I wasn't thinking of forcing superfetch handling (if I even could in this context?) but rather, as you say whether superfetch will be called or not by the burning SW by default. 
In this case setting my priority to 3 has made burns smoother & quicker with less buffer ramping.  I'm wondering whether the DVDRW buffer activity has altered now because it isn't waiting for the RAM to catch up.... 
If that is the case, the question is why has that interaction become more obvious recently?  Has PL shown a hole in the RAM-DVDRW pipeline?
Not that I need a definitive answer, just that it is an interesting turn of events that using PL has brought to my attention.

I don't have much deep knowledge of superfetch, not a bad intro: -
http://www.osnews.com/story/21471/SuperFetch_How_it_Works_Myths
In theory Win or the burning SW should be using superfetch for burns anyway, but as I say the feel of all this has changed in v6 of PL.


Yep TZ - timezone.
I will add I am surprised you have not added any red herrings yet  BYH!  ;D

BenYeeHua

Quote from: bertie97 on June 09, 2013, 12:48:30 PM
Yeah this is where the grid vs weight interpretation is coming in. 
I wasn't thinking of forcing superfetch handling (if I even could in this context?) but rather, as you say whether superfetch will be called or not by the burning SW by default. 
In this case setting my priority to 3 has made burns smoother & quicker with less buffer ramping.  I'm wondering whether the DVDRW buffer activity has altered now because it isn't waiting for the RAM to catch up.... 
If that is the case, the question is why has that interaction become more obvious recently?  Has PL shown a hole in the RAM-DVDRW pipeline?
Not that I need a definitive answer, just that it is an interesting turn of events that using PL has brought to my attention.

I don't have much deep knowledge of superfetch, not a bad intro: -
http://www.osnews.com/story/21471/SuperFetch_How_it_Works_Myths
In theory Win or the burning SW should be using superfetch for burns anyway, but as I say the feel of all this has changed in v6 of PL.
I will answer you that, ask Microsoft. ;D
----
Might be the only way is check the RAMMap, and check how many usage of standby list and system standby list.
And what's files is inside of it, just try it before you are running out of Disk. ::)
And you can check for the SuperFetch standby list too. ;)
QuoteYep TZ - timezone.
I will add I am surprised you have not added any red herrings yet  BYH!  ;D
lol, because I having a good dream last night.

And.
I will eat the Red Herrings first, as I love to eat Fish sooooo~ much. ;)

Hotrod

Red herring is an English-language idiom, a logical fallacy that misleads or detracts from the issue.[1] It is also a literary device that leads readers or characters towards a false conclusion, often used in mystery or detective fiction.

In literature, a red herring is a false clue that leads readers or characters towards a false conclusion.

see also: wild goose chase - An impossible pursuit of something unattainable.  ;D

BenYeeHua

Quote from: Hotrod on June 12, 2013, 03:24:12 PM
Red herring is an English-language idiom, a logical fallacy that misleads or detracts from the issue.[1] It is also a literary device that leads readers or characters towards a false conclusion, often used in mystery or detective fiction.

In literature, a red herring is a false clue that leads readers or characters towards a false conclusion.

see also: wild goose chase - An impossible pursuit of something unattainable.  ;D
I know, I also has Google about the meaning. ;)
Just for kidding. ;D

bertie97

Just had a thought about how this is arranged & I got to thinking about matrices.
C++ & arrays - doh!   :)

Jeremy Collake

In version 6.6 of Process Lasso, one might notice the process context menus now label these memory priorities so there isn't confusion. I do still have to migrate these new labels to all other areas of the product.

Software Engineer. Bitsum LLC.

BenYeeHua

So I just playing with memory priority a bit.
The lower the memory priority, the lower the Working Set memory, as the Windows Memory Management will Trim that process starting form 4(I guess).