Recording crashes in 6.2.008

[birdman]

The crash log you supplied does not show that as the problem. The bad characters are coming from RecordTimer.py and should be fixed at source, not in RedirectOutput.py.

Might need it in both.
You don't have total control over what is on the file-system in general, so need to be able to cater for reporting anything.

[ocean]

The crash log you supplied does not show that as the problem. The bad characters are coming from RecordTimer.py and should be fixed at source, not in RedirectOutput.py.

If it makes more sense, here what I did to RedirectOutput.py

try:
ePythonOutput(self.line, self.level)
except:
s = ''
for i in self.line:
s = s + str(ord(i)) + ' '
ePythonOutput(s, self.level)
ePythonOutput(self.line, self.level) # Let is crash

After receiving and converting that debug back to string:

[Trashcan] Failed to stat t\udce4htisade.ts: in method 'eBackgroundFileEraser_erase', argument 2 of type 'std::string const &'\n

This is "self.line" that caused crash. That original exception was inside try/except.

[ocean]

@twol thanks for the fix, but I don't think simply ignoring characters is the correct way. It fixes the crash, but does not erase the file.

Question now is why file_eraser.cpp does not accept bytes in python3, if it was bytes in python2?

Can it be modified to take also bytes, I think that would be the correct fix.

It would also make sense to add try/except to RedirectOutput (including some debug output to except block if it happens)

Logging should never crash and there are other places in code where same could happen.

[Huevos]

Can you get the exception from the debug log so we can analyse the problem.

[ocean]

Can you get the exception from the debug log so we can analyse the problem.

Exception in Trashcan or RedirectOutput?

Both are caused by '\udce4' in str.

I tried fixing enigma.eBackgroundFileEraser.getInstance().erase(f n) using bytes, but exception "argument 2 of type 'std::string const &"

Does anyone know reason why, since it was bytes in python2? Any other way to make bytes work here? Only other option is to fix '\udce4' issues in C

[twol]

@twol thanks for the fix, but I don't think simply ignoring characters is the correct way. It fixes the crash, but does not erase the file.

Question now is why file_eraser.cpp does not accept bytes in python3, if it was bytes in python2?

Can it be modified to take also bytes, I think that would be the correct fix.

It would also make sense to add try/except to RedirectOutput (including some debug output to except block if it happens)

Logging should never crash and there are other places in code where same could happen.

" It fixes the crash, but does not erase the file." - I know I am still looking at it!! Just wanted the crash removed.
The issue with your file names is that in python3 we have to use an encoding that will handle the conversion to Unicode string .... and we are guessing ... the "standard" is utf-8 (suits English speaking countries because that's the same as Ascii more or less), but doesn't handle your file names

"Question now is why file_eraser.cpp does not accept bytes in python3, if it was bytes in python2?" - it accepts a string (of bytes) which can be Ascii(py2) or Unicode (py3). Python 2 will actually handle both (ascii and Unicode if used ) transparently.
In py3 you can have either a unicode string or bytes that can be handled by core & I/O routines, where a Unicode byte can be encoded into 1-> 4 bytes in a byte string, depending on the encoding (and visa versa).
Of course, the python3 developers tell you not to touch any byte string unless you know the encoding, which is kind of nice in theory but in our case rubbish, as our inputs come at us from all over --- including helpful Windows file names.

You need to read up on py2 vs py3 - its not that simple. If it was we wouldn't have these and many other issues.... espedcially coming from an initial python2 base where we initially supported both python versions.

I agree on both your last 2 points, but we also need to be able to identify and fix the initial issue.

[twol]

Exception in Trashcan or RedirectOutput?

Both are caused by '\udce4' in str.

I tried fixing enigma.eBackgroundFileEraser.getInstance().erase(f n) using bytes, but exception "argument 2 of type 'std::string const &"

Does anyone know reason why, since it was bytes in python2? Any other way to make bytes work here? Only other option is to fix '\udce4' issues in C

I guess you are pushing a unicode string to FileEraser not bytes.
No - we have to learn how to resolve the unicode string byte ( \udce4 )and others in python

[twol]

...............deleted

[Huevos]

Does anyone know reason why, since it was bytes in python2?

Python2 did not have bytes.

Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 20 2020, 13:25:05) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> b'abc'
'abc'
>>> type(b'abc')
<type 'str'>
>>>

Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> b'abc'
b'abc'
>>> type(b'abc')
<class 'bytes'>
>>>

[birdman]

The issue with your file names is that in python3 we have to use an encoding that will handle the conversion to Unicode string ....

No, we don't . A filename (and pathname) should be kept as bytes. If it is created from a Unicode string it should be decoded to bytes.

Of course, the python3 developers tell you not to touch any byte string unless you know the encoding, which is kind of nice in theory but in our case rubbish, as our inputs come at us from all over --- including helpful Windows file names.

Indeed, they have no encoding at all. They are just a collection of bytes. You can name a file anything with any combination of bytes you wish, in any order (except '/' and '\0'). Hence they should always be kept as bytes and passed on as such.

[twol]

No, we don't . A filename (and pathname) should be kept as bytes. If it is created from a Unicode string it should be decoded to bytes.


Indeed, they have no encoding at all. They are just a collection of bytes. You can name a file anything with any combination of bytes you wish, in any order (except '/' and '\0'). Hence they should always be kept as bytes and passed on as such.

“ If it is created from a Unicode string it should be decoded to bytes.“ ………. how do you (en)code it?
„ You can name a file anything with any combination of bytes you wish, in any order“ - and how is an application supposed to know what those bytes represent?

[Huevos]

They are just a collection of bytes. You can name a file anything with any combination of bytes you wish.

And that collection of bytes was created from a charset. And also if we want to send a "collection of bytes" to the GUI we need to know the charset they were saved from so we can display them correctly.

[birdman]

“ If it is created from a Unicode string it should be decoded to bytes.“ ………. how do you (en)code it?

[gmllaptop]: python3 
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux 
Type "help", "copyright", "credits" or "license" for more information. 
>>> str = "déjà" 
>>> bytes = str.encode() 
>>> print(bytes) 
b'd\xc3\xa9j\xc3\xa0'

„ You can name a file anything with any combination of bytes you wish, in any order“ - and how is an application supposed to know what those bytes represent?

They represent a filename (or rather "an entry in the file-system"). That is all.

[birdman]

And that collection of bytes was created from a charset. And also if we want to send a "collection of bytes" to the GUI we need to know the charset they were saved from so we can display them correctly.

Which is nothing to do with the file-system.
The relevant thing is that the filename has to be kept as bytes. If you wish to display this that is another matter, but you then only convert it for display - you have to retain the original bytes for actual file-system activity.

[Huevos]

Which is nothing to do with the file-system.
The relevant thing is that the filename has to be kept as bytes. If you wish to display this that is another matter, but you then only convert it for display - you have to retain the original bytes for actual file-system activity.

You're welcome to provide a working solution.