Busy/Hot Files Monitoring

Busy/Hot Files Monitoring -------------------------------------------------------------------------------- Specification: # Needs to log bytes Written and Read per file at predefined interval to disk drives (planning to use 1min to 240min interval mostly). And to get total Written/Read per the same predefined interval per disk (just total calculation for all read writes to files). # Only need to monitor all local disks which have NTFS filesystem. (excluding CD/DVD, USB) Should show real file path if symlinks are used ( [login to view URL] ) Real file path is needed to identify the drive/file that was written/read to/from, with 3rd party analytics/monitoring software. # Program should run on background/service with small footprint on system resources. Read/Write performance should not be slowed down significantly. Memory consumption like max. 50MB for 60min interval should be reasonable. # Needs to run on platforms: XP/2003 (32/64bit), Vista/2008 (32/64bit), Win7/2008R2 (32/64bit) This should not matter, but also with XEN virtalising the Windows machine, and Terminal Services running. It is not a problem, if there are different files/drivers/applications per platform, or if they need testsigning mode to be turned on. # It would be good if I can compile the source code myself later, because the "log" format may need to be changed at some point. For starters this CSV "log" output per user defiend interval should do: #Filename examples: #Time Format ( YYYY-MM-DD_hh-mm-ss ) [login to view URL] [login to view URL] [login to view URL] ... #CSV File content example: LastReadWrite,BytesWritten,BytesRead,FullFileName "2013-10-18_22-45-05",1073741824000,8589934592000,"TOTAL C:" "2013-10-18_22-45-05",42949672960,128849018880,"TOTAL D:" "2013-10-18_22-45-05",524288,1572864,"TOTAL E:" "2013-10-18_22-43-55",42949672960,128849018880,"C:\Program Files\Microsoft SQL Server\[login to view URL]\MSSQL\DATA\[login to view URL]" "2013-10-18_22-42-18",5242880,15728640,"D:\Documents and Settings\2_10021\Desktop\[login to view URL]" "2013-10-18_22-41-66",524288,1572864,"E:\Very Very Long\Path\With Special Characters\Sample\Special2_(_)_[_]_{_}_._.Special3_$_€_£_._.Special4_++_--_^_%_¤_#_&.[login to view URL]Ää_Oö_Uü_Oõ_Žž.txt" ... Possible solution -------------------------------------------------------------------------------- * Use windows WDK 7 "minispy" sample to start. * Fix this problem: [login to view URL] By looking how file names are requested in FileMon 4 source code. * Make the driver count only successful IRP-based NonCached Reads/Writes. Cached and FastIO is not needed as they use Cache Manager/Memory Manager. And "Page Faults" -> these generate filesystem -> storage drivers activity. Page fault IRP's should have flags as paged and noncached if I remmeber correctly. * Sum the Read/Write in memory as long as user specified the interval at commandline/configfile. And then create the logfile and clean memory to start over for next logfile. Should be 2 memory buffers to not miss or lose any read/write activity i think. I just have not enought C knowledge, nor time to complete my own version properly. Any other solution that matches specification is also accepted. Available applications related to this topic. And what I have tried to find a quick solution based on these related programs. -------------------------------------------------------------------------------- # Windows 7 "Resource Monitor", Disk Tab, "Disk Activity" ... # Sysinternals process monitor. [login to view URL] ... # Sysinternals FileMon 410 [login to view URL] ... # FileSpy [login to view URL] This program works similar to Filemon. ...