Now that the full path to a file/lump in the virtual file system is no longer stored as a unit (WadFile and ZipFile store paths using PathDirectory), sorting the contents of the zip LumpDirectory (for the purpose of pruning older lumps) became a significant performance bottleneck during startup.
I also noticed the lump name property stored in LumpInfo was now rather redundant, being as it's value was also present in the first fragment of the containing lump's PathDirectoryNode. The only time this differed was with the special case of lump assemblies, which remap the name used in the virtual filesystem. However, as these names are used to enforce load ordering in the LumpDirectory they are published to, in order to remove this redundancy I would also have to address the LumpDirectory sorting (which, as mentioned above, happened to be the cause of the perfomance issue).
Lump names are no longer duplicated in LumpInfo
All lumps are now ordained with a '.lmp' extension
Hash chains in LumpDirectory now use PathDirectoryNode::Hash
Solved O(n) bottleneck in zip file search using the hash chains
Solved performance issues due to repeatedly compositing paths
LumpFiles are no longer published directly to the LumpDirectory
Do not sort contents of LumpDirectory, use external sort for prune
Relocated lump assembly path mapping to ZipFile temporarily
Todo: Re-implement support for lump assemblies as directories in the local file system. These now need be handled a little differently.
Todo: It should now be possible to remove the external sort used for pruning LumpDirectory contents entirely, re-using the integral hash chains.
Added by danij almost 13 years ago
Added by danij almost 13 years ago
FileSys|Refactor: Various interwoven refactorings
Now that the full path to a file/lump in the virtual file system is
no longer stored as a unit (WadFile and ZipFile store paths using
PathDirectory), sorting the contents of the zip LumpDirectory (for
the purpose of pruning older lumps) became a significant performance
bottleneck during startup.
I also noticed the lump name property stored in LumpInfo was now
rather redundant, being as it's value was also present in the first
fragment of the containing lump's PathDirectoryNode. The only time
this differed was with the special case of lump assemblies, which
remap the name used in the virtual filesystem. However, as these
names are used to enforce load ordering in the LumpDirectory they
are published to, in order to remove this redundancy I would also
have to address the LumpDirectory sorting (which, as mentioned above,
happened to be the cause of the perfomance issue).
Todo: Re-implement support for lump assemblies as directories in the
local file system. These now need be handled a little differently.
Todo: It should now be possible to remove the external sort used for
pruning LumpDirectory contents entirely, re-using the integral hash
chains.