Directory traversal/file discovery hard file cap #131
Description
Hi
We've been using this sink for a while and have identified a nasty one.
We are using the sink for applogs in a blob storage folder which has many many other files, and noticed an issue where the logger would hit 998 files but would not roll over and instead would:
Increase the file size for log 998 endlessly and would also trigger a continual createif not exists which is a price tier Hot operation which was leading to excessive azure costs.
This was an interesting one because we could see the log rollover should be working but because it was trying to list every file/folder in the blob it was unable to find its target files to erase/roll them over.
We made a few patches:
• Increase max file number to 999999
• check if blob exists before trying createifnotexists (which is an expensive hot operation)
• Passed a basefilepath that limits directory searching to the level the user has defined which by default should be to where the application is housed, allowing rollover to start working again as it limited the scope of files it would find (it would still crap out if there are many many files inside this folder but we havent seen it tip until it hits greater then 200,000 ish
Our case might be rare because we had high volume of logs generated, and also many directorys with files but we believe this could occur for others so by limiting the scope of where it should look for its own logs and putting the createifnotexists behind the exist reduces hot operations and azure costs
The increased number of files isnt strictly necessary but we did that for ourselves as we would need in the 10k range.
If you need more info on the issue or a chat on the above please let me know
Kind regards
Mike