You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are using MemoryMappedFiles for an immense amount of the data that does not fit totally into the main memory. One file could have at least 50GB and is running through a chain of processing steps where each steps processes another 50GB dataset. But the processing steps don't to this in a sequential manner, but rather it is a pull-based principle where the last processing step acts as an output where each request to process blocks of data is bubbling up the processing chain and each step requests data from the step before. So it is not necessary that the whole data is in memory, but can be dropped depending on what data is currently requested.
Originally I had my own my memorymanager with a multiple backing files, where everything was managed and I was intercepting the GC and checking continuously the RAM situation and did paged/unmapped the data to disk which worked pretty well but had it's own problems. The paging worked by sorting the access time and all the data that hasn't been used was paged (a bit more complicated, just to explain the principle).
MemoryMappedFiles are a good solution to this, but the problem is that the data is processed faster then the memory manager can handle the data. So when we are using large amount of data we see that the RAM is increasing until it reaches the RAM boundary. Because no one can currently advise the memory manager what to do.
As far as I have seen VirtualUnlock, FlushViewOfFile and FlushFileBuffers are the methods to assist the memory manager. Where in my situation VirtualUnlock is maybe the most promising solution. I want to page certain memory regions and make the RAM free for other memory requests.
Currently it seems that none of these methods (or the behavior of them) is exposed in the memory mapped file API. Does it make sense to expose them, or should I go with PInvoke calls on the handles? Using PInvokes directly makes it a little hard for other platforms so the question is if there is a better solution?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
We are using MemoryMappedFiles for an immense amount of the data that does not fit totally into the main memory. One file could have at least 50GB and is running through a chain of processing steps where each steps processes another 50GB dataset. But the processing steps don't to this in a sequential manner, but rather it is a pull-based principle where the last processing step acts as an output where each request to process blocks of data is bubbling up the processing chain and each step requests data from the step before. So it is not necessary that the whole data is in memory, but can be dropped depending on what data is currently requested.
Originally I had my own my memorymanager with a multiple backing files, where everything was managed and I was intercepting the GC and checking continuously the RAM situation and did paged/unmapped the data to disk which worked pretty well but had it's own problems. The paging worked by sorting the access time and all the data that hasn't been used was paged (a bit more complicated, just to explain the principle).
MemoryMappedFiles are a good solution to this, but the problem is that the data is processed faster then the memory manager can handle the data. So when we are using large amount of data we see that the RAM is increasing until it reaches the RAM boundary. Because no one can currently advise the memory manager what to do.
As far as I have seen
VirtualUnlock,FlushViewOfFileandFlushFileBuffersare the methods to assist the memory manager. Where in my situation VirtualUnlock is maybe the most promising solution. I want to page certain memory regions and make the RAM free for other memory requests.Currently it seems that none of these methods (or the behavior of them) is exposed in the memory mapped file API. Does it make sense to expose them, or should I go with PInvoke calls on the handles? Using PInvokes directly makes it a little hard for other platforms so the question is if there is a better solution?
Beta Was this translation helpful? Give feedback.
All reactions