Windows Server 2008 - Can I disable Memory Mapped IO for a file?

[dpressm]

I am running a program which processes large input files into "load bufffers" (Essbase by the way) I would like for these load buffers to remain in ram and I have enough room for them to do so but the operating system is allocating memory for memory mapped IO in about a 3:1 priority in favor of the load file (versus the load buffer file).  Note: 3:1 is approximately the ratio of the file sizes.

Since I will only be reading the input file once I have no need to have it retained in memory mapped IO.  Is there a way to do prevent this?   (Note: I believe that the same process reads and writes so I can not set priority for the process in its entirety.

[Jeremy Collake]

The obvious question is I suppose you can't simply unmap the view of the file?? Then open it back up and remap if you later need to write to it? That seems most optimal.

[dpressm]

Changed text in bold

I am running a program which processes large input files into "load bufffers" (Essbase by the way) I would like for these load buffers to remain in ram and I have enough room for them to do so but the operating system is allocating memory for memory mapped IO in about a 3:1 priority in favor of the load file (versus the load buffer file).  Note: 3:1 is approximately the ratio of the file sizes.

Since I will only be reading the input file once I have no need to have it retained in memory mapped IO.  Is there a way to do prevent this For the Input file only?   

(Note: I would  not want to turn off memory mapped IO for the process entirely - only for the input file.  The same process that reads the input file reads also performs both reads and writes to the output or load buffer file .


Finally, I would like to add - is there some way to do this at the operating system level as I have no access to the source code for the process.  In other words, I must admit that I have no idea what you mean by "unmap the view of the file" as I suspect that is done within the process.

[Jeremy Collake]

Oh, I see, you don't have access to the source code. In that case, it is a lot more difficult. The OS maps the file into memory because the application has told it do so. To force it to not map the file into memory could cause catastrophic effects. The application itself also expands the memory mapped file to 3x its original size (if I understand you correctly). That isn't the OS, that's the application telling the OS that the memory mapped file could grow, and indeed giving itself a lot of room to grow. Without allowing for space to grow, a file would have to be closed and remapped to expand, that is likely why the application takes things that far.

While you could adjust the 'slack space' externally, or through modification of the binary code of the application, and it might be ok, the effort required is too great to be worth it to ... well, anyone probably ;p. It is not a common problem, so you would be the only one in the universe needing a solution to it. In a best case scenario, contacting the developers of that application would be in order.

[dpressm]

OK I actually do have contact with the developers and would like to suggest it.  Exactly what would I suggest?

And just for clarity: The input file itself does not grow.  The process reads in the input file and writes out an output file of 1/3 the size (this reduction is normal and expected).  So when I look at RamMap the input file is occuping 3 times as much memory mapped IO space as the output file.  The Input file will never be read again but the output file will be read again in subsequent steps (and even with in the ETL step is sometimes partially reread and modified and of course written).

Thank You very much for your interest and assistance.

Hopefully you can phrase the suggestion to the developers something like open the file with priority x or in mode y so thaat it will not be memory mapped or so that it will be at a higher memory map priority than the output file.

Thanks again.

[dpressm]

I know that the program uses different (and multiple) threads for the "inputPreparation" and "outputpreparation" so hopefully that would help.

[Jeremy Collake]

OK I actually do have contact with the developers and would like to suggest it.  Exactly what would I suggest?

Ask them if there is any way, during their operations, to consume virtual memory *or* the part of virtual memory in RAM (the working set) and/or use it less liberally? They may tell you it will be slower or have caveats though, but at least give you some new option.

And just for clarity: The input file itself does not grow.  The process reads in the input file and writes out an output file of 1/3 the size (this reduction is normal and expected).  So when I look at RamMap the input file is occuping 3 times as much memory mapped IO space as the output file.  The Input file will never be read again but the output file will be read again in subsequent steps (and even with in the ETL step is sometimes partially reread and modified and of course written).

It may not grow in the end, but that does not mean its mapping in memory does not need to be larger for some other reason, such as some temporary expansion in size before it is eventually reduced. Or perhaps it is simply an artifact. Who knows why they did that. They would have to be asked.

Now, this is assuming I have understood you correctly. To me, a memory mapped file is one associated with the APIs MapViewOfFile/Ex.. if you speak of something different, then things change.

Let us know what happens! Perhaps even a rationale for their design, so we understand, if they give you one. Not all developers are as open as me.