-
Notifications
You must be signed in to change notification settings - Fork 811
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possibly redundant flush when seeking #996
Comments
Hi @stefano-zanotti, thanks for creating an issue. There is unfortunately a tradeoff between performance and code-size when it comes to leveraging caches. It's certainly simpler to just throw away the cache on seek. This is probably a case where making the caching logic a bit smarter is warranted. But at the moment there is a significant amount of work related to files bubbling down the pipeline, so it may be better to wait and revisit this after larger data-structure-related performance issues are potentially shaken out. |
I understand this tradeoff. Could you please just verify one thing? Lines 3698 to 3702 in d01280e
with a simple "true", to always perform the "skip re-caching if unneeded" optimization, but I had problems down the line. However, to me it seems it should be enough to fix my issue. If I'm right, the fix would be extremely simple even now. Or, are there other reasons I'm not seeing, for which this can't work? |
So, the main difference between LFS_F_READING and LFS_F_WRITING is that the data exists on disk when LFS_F_READING, but not when LFS_F_WRITING. The data only exists in the cache. So it's safe to discard the data with LFS_F_READING, but if we discard the data with LFS_F_WRITING, we've lost the data. Consider this sequence:
If our cache size is 2048, there is 1024 bytes in the range 1024-2048 that we lose track of. Unfortunately littlefs doesn't know how much data is going to be written at seek time, so it needs to be conservative and write everything out. I may be wrong, but I think what you're looking for is a sort of write-cancellation API. There's is something in the works on this (lfs_file_desync), but it's only a WIP at the moment... |
I'm not sure lfs_file_desync would help me. I still don't fully understand the issue. If we do:
LittleFS still knows that the file is 2048 bytes long (since overwriting, with the final WRITE, does not imply truncating), and since it flushes out the whole cache anyway, the CLOSE operation should output the whole "bb...ba...aa" data anyway, which would be correct. The fact that the position points at 1024 should be irrelevant. The same would work in this case:
The last write must detect that the cache window must move, so it should flush the current cache before fetching (and overwriting) the next window:
This behavior is needed even without using seek(), that is, when we just keep appending data to the file, so I assume that LFS already knows how to do it (I mean, it knows when to flush the cache to disk, if the new write needs to enter a new window).
The second write must have flushed the data properly, as decribed above. This exact behavior is what is needed in my case; only, the previous seek should not flush the cache (if the window doesn't move), and let the next write/close/sync operation handle the flushing. |
When seeking (lfs_file_seek_), LFS needs to flush out whatever is in its cache, and then refresh the cache with the block where the new file position is.
However, if the file is in read mode, and the new position is in the same block as the one currently in cache, it can just change the position variable, without any disk operation.
See here:
littlefs/lfs.c
Lines 3697 to 3720 in d01280e
The same could be done when writing too: if the new position is in the same block, we can avoid disk operations, and only trigger them on a subsequent sync/write.
However, LFS currently skips this optimization: it always flushes the file, if the cache is dirty.
I tried removing the check on LFS_F_WRITING, but my software misbehaved in different parts of the code, so I assume this sync is needed for some other reason.
Is it really necessary?
Can this optimizazion be implemented in some way?
Is this optimization impossible for some reason due to the internals of LFS?
In my use case, this missed optimization has a big impact on performance, since LFS is called using a pattern like the following:
SEEK 0, WRITE 2048
SEEK 0, WRITE 2048
SEEK 2048, WRITE 2048
SEEK 2048, WRITE 2048
etc.
That is, each write just needs to append 1KB at the end of the file, but it works with 2KB blocks, so it always rewrites a whole block (padded with 0s), unless it needs to switch to the next one.
This causes LFS to flush to disk even if it was not asked to do so by a sync/close call:
SEEK 0, flush, WRITE 2048
SEEK 0, flush*, WRITE 2048
SEEK 2048, flush, WRITE 2048
SEEK 2048, flush*, WRITE 2048
Instead, the flushes marked with * could be avoided: the data would be still in cache, and it would be overwritten by the following write.
Here I'm assuming that LFS's cache size is 2048, but other flushes could be avoided too, if the cache was bigger.
The text was updated successfully, but these errors were encountered: