Author Topic: Logging problems  (Read 235 times)

Offline Coldblackice

  • Member
  • ***
  • Posts: 86
  • Gender: Female
Logging problems
« on: November 09, 2019, 12:07:28 AM »
Some issues with the logging:
  • In both .CSV logs (prolasso.log.csv + "Export log to CSV" .CSV), there are no column headers on either. This makes it difficult to track data/columns
  • In the "Export log to CSV" function, the outputted .CSV has two bytes in the header/beginning (the very first two bytes) which is causing Excel to trip up on and not process correctly as a CSV table/spreadsheet. The bytes are "FF-FE". See attached
  • In both CSV's, the ordering of columns doesn't mirror the program (main window or Log Viewer). This wouldn't be as bad if column headers were at least available
  • In the prolass.log.csv, there are weird data in columns to the far right, such as random integers, and high enough integers which couldn't be PID's. I thought maybe it could be CPU/running-time (in seconds), but not all rows have it, so it's probably not that
  • In the "More info" column in the Log Viewer, some entries have a comma in the "More info" column separating "Parent" from "PATH". In a .CSV, this is classified as a new column, which means that the columns' data aren't all uniform, since some entries push the columns to the right out further because of the extra column inserted
  • The Log Viewer only goes back a few hours of data. It'd be great if it could go back as far as possible



Offline Jeremy Collake

  • Administrator
  • Member#
  • *****
  • Posts: 5281
  • Gender: Male
  • The Lasso
    • Bitsum
Re: Logging problems
« Reply #1 on: November 09, 2019, 06:17:35 AM »
Thanks, I will address many of these issues going forward. I agree the CSV output needs improved, as does the entire logging subsystem. I plan to move to Windows Event Logs in the near future.

The FFFE bytes are the BOM marker, BTW.
Software Engineer. Bitsum LLC.

Offline Coldblackice

  • Member
  • ***
  • Posts: 86
  • Gender: Female
Re: Logging problems
« Reply #2 on: November 09, 2019, 11:42:17 PM »
No problem. Got it, didn't know about BOM. You surely know the matter better than I do, but on some cursory searching*, I'm gathering that adding a BOM may cause more issues than it addresses due to non-standardized software interpreting, and that modern software is better able to deduce what encoding is being used automatically. Also, apparently UTF-8 doesn't require a BOM, as software is able to recognize its encoding automatically. Excel, besides saving as a plain no-BOM .CSV, is also only able to save alternatively as UTF-8, so this must be why it doesn't recognize the Unicode BOM.

Again, you know this better than I do, and I realize PL is a globally used software, but if predefined encoding is necessary, would you consider UTF-8 instead? Given the pervasiveness of Excel and its use for displaying + interacting with CSV's.

Not a huge deal if not. I've found some commandline tools that will strip BOMs automatically, it would just avoid an extra step (though I say that acknowledging that I don't know what other potential outcomes would be from removing or changing the BOM).

*https://softwareengineering.stackexchange.com/a/372693/48835
*https://softwareengineering.stackexchange.com/a/372704/48835

Offline Jeremy Collake

  • Administrator
  • Member#
  • *****
  • Posts: 5281
  • Gender: Male
  • The Lasso
    • Bitsum
Re: Logging problems
« Reply #3 on: November 10, 2019, 09:25:38 AM »
I think your analysis is correct. The BOM is probably not be appropriate for CSVs. I will remove it in the future, and transition to UTF-8.
Software Engineer. Bitsum LLC.

Offline Coldblackice

  • Member
  • ***
  • Posts: 86
  • Gender: Female
Re: Logging problems
« Reply #4 on: November 11, 2019, 04:02:33 AM »
Sounds good, thanks!

Also, not sure if this is a bug or as intended -- I just used the function "Choose alternate log folder", selected a new log location (I'm prepping for an upcoming OS reformat), and immediately after, the entire Actions log pane was wiped. Additionally, the Log Viewer is also completely wiped. I thought maybe this was because a new folder location was starting with a new log file, but I can see that prolass.log.csv is still the same size (40MB) and in the same location (%APPDATA%\ProcessLasso\logs), and it's still saving regularly (once a minute).

Also, the Actions log pane is now all-white, no longer dark-themed like it had been.

EDIT1:

OK, I restarted PL -- the Actions log dark theme is restored. I also see that the default prolasso.log.csv file is still in its default location, however, it's no longer being updated (understandably) -- the new log location now has a fresh log file it's starting from (no issue, just walking through my steps). I closed PL and copied the old/historical .CSV log into the new location and restarted PL. It appears PL is continuing to write to it just fine. However, the Actions log and Log viewer aren't "seeing" the past data, so I'm assuming they don't pull their data from the file on disk?

Now that I mention it, I'm assuming they both get their data "live" as it actively occurs and not from file, so they'll both only show data from the current run of PL (correct me if I'm wrong). I understand why you'd want to have their data come directly from in-process memory as it happens rather than continually accessing from disk, but would it be possible to have them both retrieve their prior data from the on-disk .CSV log file initially when PL starts?

It's useful to see be able to see the history even after PL is restarted. Especially the Log Viewer, this would be most useful there. Not a huge deal, as we can just open the log .CSV on disk in Excel. But if it wouldn't cause any waves or subvert another intended purpose, this would be helpful, if possible.

Offline Jeremy Collake

  • Administrator
  • Member#
  • *****
  • Posts: 5281
  • Gender: Male
  • The Lasso
    • Bitsum
Re: Logging problems
« Reply #5 on: November 11, 2019, 09:35:41 AM »
I see there is an issue with the behavior after changing the log path. Process Lasso should restart itself, though I may force a log folder change to be made in InstallHelper. I will fix this!

When you change the log folder it starts fresh, instead of migrating existing data.

The data is read from the .LOG files (not CSV) in all cases (Actions pane and LogViewer). The CSV is output-only and an afterthought really. The history should be shown from past instances.

When the logging subsystem gets refactored, all this will be improved. The switch to the Windows Event logs will inherently solve many challenges. Until then, I am not keen to nibble around the edges too much.

Thanks for the feedback!
Software Engineer. Bitsum LLC.