Log files fragmentation

[smartbit]

Hi,
I've noticed that Process Lasso's Log Files getting extremely fragmented over time (~1k fragments per every 3MB log file).
While the files are tiny, and not that important, it suggests that that they are updated live. Which is desirable for any tracing/debugging activity. But in a day-to-day use it just may cause a constant HDD-writes, wasting battery life and stressing HDD pretty much in vain.

I'm aware about "disable log" option.
However, I wonder whether is there any option to set "delayed write" or something, so that the log data will be written to RAM buffer, and only once in a long while get dumped to the HDD?

[BenYeeHua]

So far, nope.

But I think it can be adding as a feature, and showing the warning like "increase the chance of log missing" etc. 

[Jeremy Collake]

No worries, I have good answers for you.

The fragmentation is probably caused by a couple factors:

1. The slow rate at which they are written to disk, and slow growth relative to time (so lots change on the disk by the time they get written again).
2. Defragmentation method. A good defragmenter that leaves some room for file growth would probably help. That's one reason defrags are sometimes ill-advised, as clumping files together means they have to room to grow, so the next write, if it exceeds the cluster size, causes new fragmentation.

The log file writes are designed to never interfere with anything.

Log events are queued by the Governor a bit, not written immediately.

The log files have a 'temporary' attribute set as well. This causes the OS to cache their writes the maximum time it can before committing them to disk.

Further refinement is possible, and I'll certainly be thinking on this. Since the logs are used for some inter-process communication, and also for activity auditing, it is important that they not be too relegated to the low priority bin.

If you see, or can demonstrate, that there *is* actually *real* log disk I/O during periods where there is contention for the disk, then please let me know -- but I suspect a study would show you that these log writes are never getting in the way for  to the above reasons.

[BenYeeHua]

And if the fragmentation is the disk fragmentation, I don't think we can do anything here, as we can't force windows to get a empty space for the log file, then write continuous on that empty space.

[edkiefer]

One thing is to limit log writes , I always disable log processes launches and terminations , I don't need that option on ,except for debugging some issue .
Those two will stop 80% of logs ,unless you have a lot of rules setup .

[smartbit]

Hi.
Thanks for the response :-)
Your assumption is correct, there is no actual noticeable disk stress.
The log writes are about 2-4 Kb every several seconds, depending on the activity. So, by that, it won't be correct to say that it "stresses the system" really.

But anyways, 2-3 days after full defragmentation, 12 "prolasso.log*" files got 700-1000 fragments each, while the next most fragmented files (of the whole C:\ drive) are: prefetch, *.db, and other *.log files, which has less than 100 fragments each (and this is after a 2-3 days of a very "light" PC use: some web, some docs, etc.)

I once worked analyzing custom hardware device, and we used some SW tool to log various parameters and monitor device's memory for a long periods of time, which would result in a huge log files (up to GB). But the writes to the files themselves were performed every several minutes, each time writing few MB.
I don't know the exact technical details, but the guy who made this tool told me that his SW dumps the log constantly to the OS with some specific parameters, and then OS knows to cache it and perform write backs accordingly.
So, my understanding is that technically it is possible to accumulate 1-2 MB of the data for a while for a delayed write later (up to minutes).

Bottom line: If it can be done by modifying few parameters, without much effort/redesign, and by that decrease the write frequency and increase caching size, (making it as as option of course), it will be nice.

By the way, I noticed that log files' properties show attributes: ATI.
Does it mean:
     A = ARCHIVE
     T = TEMPORARY
     I = Not content indexed
?


P.S. - BenYeeHua, the fragmentation I'm talking about IS the disk fragmentation, and I'm aware of the fact that updating (increasing the size, precisely) of the files will cause fragmentation.
Also, it's not completely true that it's impossible to force windows to save continuous space. It can be done by allocating space for a specific file size, and creating this file in advance. Modifying its contents later as much as you wish, as long as you fit inside the original boundaries. (It is (partially) similar to the way Page File being handled. And many other programs use this technique.)
But doing it here, may need some redesign of the current mechanisms here, and will cause too much trouble to go through for a not-so-important reason. Thats why I'm asking about a possibility to optionally decrease write frequency by more minor SW modifications.

[BenYeeHua]

One thing is to limit log writes , I always disable log processes launches and terminations , I don't need that option on ,except for debugging some issue .
Those two will stop 80% of logs ,unless you have a lot of rules setup .

Yup, and this help me to find out, when the anti-virus and also IME are running some ads software again.

P.S. - BenYeeHua, the fragmentation I'm talking about IS the disk fragmentation, and I'm aware of the fact that updating (increasing the size, precisely) of the files will cause fragmentation.
Also, it's not completely true that it's impossible to force windows to save continuous space. It can be done by allocating space for a specific file size, and creating this file in advance. Modifying its contents later as much as you wish, as long as you fit inside the original boundaries. (It is (partially) similar to the way Page File being handled. And many other programs use this technique.)
But doing it here, may need some redesign of the current mechanisms here, and will cause too much trouble to go through for a not-so-important reason. Thats why I'm asking about a possibility to optionally decrease write frequency by more minor SW modifications.

Hmm, yup, many download software allow allocating space, based on the file size. 

And yes, as it is just a txt file, I think it will be harder to getting allocate a space for it, even most of the space that each file is taking are 3,444 - 3,448KB.
---
I just checked awhile, it look like for me, I has enabled all the log anything, so if you want 1-2 MB delayed write, I think it need system to buffering 4-8 hours, and I wonder did it will works. 

[smartbit]

... so if you want 1-2 MB delayed write, I think it need system to buffering 4-8 hours ...

I gave 1-2 MB just as an example to point that the cache size is not the issue here. There is no reason for actually caching for so long with Process Lasso.

Buffering for a several minutes should be sufficient. And compared to the way it works now (few writes per minute) it should show a good improvement.

[BenYeeHua]

I gave 1-2 MB just as an example to point that the cache size is not the issue here. There is no reason for actually caching for so long with Process Lasso.

Buffering for a several minutes should be sufficient. And compared to the way it works now (few writes per minute) it should show a good improvement.

Roger, but the file is using FILE_ATTRIBUTE_TEMPORARY, like what's you found.
So I guess you always don't has enough available memory to buffering the file, and causing it write each minute, or it is a bug/design for windows?

[smartbit]

Roger, but the file is using FILE_ATTRIBUTE_TEMPORARY, like what's you found.
So I guess you always don't has enough available memory to buffering the file, and causing it write each minute, or it is a bug/design for windows?

Actually it supposed to do the desired delayed write, as stated in this <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx#caching_behavior">MS arcticle (Caching Behavior section)</a>:

Specifying the FILE_ATTRIBUTE_TEMPORARY attribute causes file systems to avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Although it does not directly control data caching in the same way as the previously mentioned flags, the FILE_ATTRIBUTE_TEMPORARY attribute does tell the system to hold as much as possible in the system cache without writing and therefore may be of concern for certain applications.

So it seems like all should be as delayed as it gets, though, I still didn't figured out if there are more parameters to control the delayed write more precisely.

[BenYeeHua]

Hmm, I guess I might do some test on this one, it can be the setup on your side, or it don't works for file that saved on disk.