Logging problems

[Coldblackice]

Some issues with the logging:

• In both .CSV logs (prolasso.log.csv + "Export log to CSV" .CSV), there are no column headers on either. This makes it difficult to track data/columns
• In the "Export log to CSV" function, the outputted .CSV has two bytes in the header/beginning (the very first two bytes) which is causing Excel to trip up on and not process correctly as a CSV table/spreadsheet. The bytes are "FF-FE". See attached
• In both CSV's, the ordering of columns doesn't mirror the program (main window or Log Viewer). This wouldn't be as bad if column headers were at least available
• In the prolass.log.csv, there are weird data in columns to the far right, such as random integers, and high enough integers which couldn't be PID's. I thought maybe it could be CPU/running-time (in seconds), but not all rows have it, so it's probably not that
• In the "More info" column in the Log Viewer, some entries have a comma in the "More info" column separating "Parent" from "PATH". In a .CSV, this is classified as a new column, which means that the columns' data aren't all uniform, since some entries push the columns to the right out further because of the extra column inserted
• The Log Viewer only goes back a few hours of data. It'd be great if it could go back as far as possible

[Jeremy Collake]

Thanks, I will address many of these issues going forward. I agree the CSV output needs improved, as does the entire logging subsystem. I plan to move to Windows Event Logs in the near future.

The FFFE bytes are the BOM marker, BTW.

[Coldblackice]

No problem. Got it, didn't know about BOM. You surely know the matter better than I do, but on some cursory searching*, I'm gathering that adding a BOM may cause more issues than it addresses due to non-standardized software interpreting, and that modern software is better able to deduce what encoding is being used automatically. Also, apparently UTF-8 doesn't require a BOM, as software is able to recognize its encoding automatically. Excel, besides saving as a plain no-BOM .CSV, is also only able to save alternatively as UTF-8, so this must be why it doesn't recognize the Unicode BOM.

Again, you know this better than I do, and I realize PL is a globally used software, but if predefined encoding is necessary, would you consider UTF-8 instead? Given the pervasiveness of Excel and its use for displaying + interacting with CSV's.

Not a huge deal if not. I've found some commandline tools that will strip BOMs automatically, it would just avoid an extra step (though I say that acknowledging that I don't know what other potential outcomes would be from removing or changing the BOM).

*https://softwareengineering.stackexchange.com/a/372693/48835
*https://softwareengineering.stackexchange.com/a/372704/48835

[Jeremy Collake]

I think your analysis is correct. The BOM is probably not appropriate for CSVs. I will remove it in the future, and transition to UTF-8, if appropriate, pending further research.

[Coldblackice]

Sounds good, thanks!

Also, not sure if this is a bug or as intended -- I just used the function "Choose alternate log folder", selected a new log location (I'm prepping for an upcoming OS reformat), and immediately after, the entire Actions log pane was wiped. Additionally, the Log Viewer is also completely wiped. I thought maybe this was because a new folder location was starting with a new log file, but I can see that prolass.log.csv is still the same size (40MB) and in the same location (%APPDATA%\ProcessLasso\logs), and it's still saving regularly (once a minute).

Also, the Actions log pane is now all-white, no longer dark-themed like it had been.

EDIT1:

OK, I restarted PL -- the Actions log dark theme is restored. I also see that the default prolasso.log.csv file is still in its default location, however, it's no longer being updated (understandably) -- the new log location now has a fresh log file it's starting from (no issue, just walking through my steps). I closed PL and copied the old/historical .CSV log into the new location and restarted PL. It appears PL is continuing to write to it just fine. However, the Actions log and Log viewer aren't "seeing" the past data, so I'm assuming they don't pull their data from the file on disk?

Now that I mention it, I'm assuming they both get their data "live" as it actively occurs and not from file, so they'll both only show data from the current run of PL (correct me if I'm wrong). I understand why you'd want to have their data come directly from in-process memory as it happens rather than continually accessing from disk, but would it be possible to have them both retrieve their prior data from the on-disk .CSV log file initially when PL starts?

It's useful to see be able to see the history even after PL is restarted. Especially the Log Viewer, this would be most useful there. Not a huge deal, as we can just open the log .CSV on disk in Excel. But if it wouldn't cause any waves or subvert another intended purpose, this would be helpful, if possible.

[Jeremy Collake]

I see there is an issue with the behavior after changing the log path, and it may fail to change the path at all through that menu item. Process Lasso should restart itself, though I may force a log folder change to be made in InstallHelper. I will fix this!

When you change the log folder it starts fresh, instead of migrating existing data.

The data is read from the .LOG files (not CSV) in all cases (Actions pane and LogViewer). The CSV is output-only and an afterthought really. The history should be shown from past instances, may be an issue.

When the logging subsystem gets refactored, all this will be improved. In the interim, I am addressing these issues in the legacy code.

Thanks for the feedback!

[Jeremy Collake]

Pertinent changes to 9.4.0.70:

- Log: Standardize CSV output, adding headers, quotes and removing BOM
- Log: LogViewer: Add full history CSV export in addition to current view export
- Log: Fix main GUI 'File / Export log to CSV' would not export full history
- GUI: Fix to 'File / Choose alternate log folder'

I am evaluating the necessity to save as UTF-8 instead of UTF-16. Excel seems to be able to import UTF-16 properly when the BOM is removed. Modern text editors also seem to auto-detect the UTF-16LE even without the BOM. Breakage of any clients depending on the existing log format is also a concern.

Increasing the depth shown in the Log Viewer is still pending. The limiting factor is actually the max entries in each log file, before it is rotated, or the size of the current log file. It is something I want to time to test more for performance consequences on lesser machines. This may get deferred (below).

There are several more logging related concerns, but they need to wait for the full refactor. Having our own log system was always a poor choice.