Crash when folder contains filenames with Windows encoding

[ocean]

appreciate suggestions....... Huevos and I are still looking at having a resolution ... in python

I'm not sure filesystem issues are solvable with ONLY python changes.

In C++ string (std::string) is bytes and it's not required to be UTF-8 string.

C++ functions should be able take bytes, not sure why it's not working. I think if you know answer to this, you would be closed to solution.

[birdman]

We are working in python. When we access the file system we use os.walk. Are you saying you think we should remove all python abstraction layers and work directly on the byte code on the hard drive?

No.
I'm saying that when you get a filename (or pathname) you must always keep it as bytes. If you ever convert it to a str then you (may) no longer have anythng that corresponds to what is on disk.

[Huevos]

I'm not sure filesystem issues are solvable with ONLY python changes.

In C++ string (std::string) is bytes and it's not required to be UTF-8 string.

C++ functions should be able take bytes, not sure why it's not working. I think if you know answer to this, you would be closed to solution.

Can we try experimenting in Trashcan.py?

				for root, dirs, files in os.walk(trashfolder.encode(), topdown=False):
					for name in files:
						# Don't delete any per-directory config files from .Trash if the option is in use
						if (config.movielist.settings_per_directory.value and name == ".e2settings.pkl"):
							continue
						fn = os.path.join(root, name)
						st = os.stat(fn)
						try:
							import chardet
							encoding = chardet.detect(fn)["encoding"]
							fn = fn.decode(encoding=encoding)
						except Exception as e:
							print( type(e).__name__, e)
						if st.st_ctime < self.ctimeLimit:
							try:
								enigma.eBackgroundFileEraser.getInstance().erase(fn)
								bytesToRemove -= st.st_size
							except Exception as e:
								print("[Trashcan] Failed to erase file after stat.st_ctime selection:", name, "   ", type(e).__name__, e)								
						else:
							candidates.append((st.st_ctime, fn, st.st_size))
							size += st.st_size
					# Remove empty directories if possible

[ocean]

Can we try experimenting in Trashcan.py?

It's possible that could work. I can test it later.

Small note, also "name" is bytes. name == ".e2settings.pkl" -> name == b".e2settings.pkl", and also in that debug print

[Huevos]

It's possible that could work. I can test it later.

Small note, also "name" is bytes ... in that debug print

That's fine.

>>>
>>> print(b"abc")
b'abc'
>>>

[ocean]

Looks like it's still not working. No crash, but it doesn't erase the file.

I modified it to erase always, if filename contains 'test' and added some test files to trash folder.

I guess next step would be to check old python2 image, if it erases the files. (I only know it doesn't crash)

Also if anyone else wants to test, it's easy to create files from terminal. (Just remember to remove st_ctime check from code to make sure it tries to erase)

fn = b'/media/hdd/movie/.Trash/' + b't\xe4hti.mpg'
f = open(fn, 'w')
f.write('')
f.close()

[twol]

@Ocean - I believe that you are using latin-1?? I tried that with file generated in post #36 and it does same as Ascii.... no delete
So I made the mistake of modifying both file_eraser.cpp and .h to take char*, but image build fails because other C++ modules use this module directly and expect string.
Will look at what it will take to handle this change to include modded file_eraser, but starting to think the whole lot will have to be done in python.

[birdman]

@Ocean - I believe that you are using latin-1??

The file-system has no concept of latin-1, unicode, iso8859-16, etc..
It only deals in a byte string.
So what any user is using has to be ignored, except when displaying it to a user, when it might enable it to make sense to a person - but such a translation must not produce an error, which is the problem here.

[Huevos]

@Ocean,

Is it possible to check with the current code in the repo:
1) it works with utf8 filenames that contain single byte chars > 127.
2) it works with utf8 filenames that contain multi byte chars.

[ocean]

So I made the mistake of modifying both file_eraser.cpp and .h to take char*, but image build fails because other C++ modules use this module directly and expect string.

I said it could possible to add overloaded function, not to modify existing one.

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/.. Could someone add some instructions to github enigma2 page?

My initial cpp and h are attached if you like to test those. Don't know if they compile or not.

Also added debug prints to original function where filename is empty or not found. I think you can commit 2 debug lines if they work, erase shouldn't normally be called with empty or non existing file.

If you get debug prints working you should see in log what filename contains in case it does not erase.

[ocean]

Is it possible to check with the current code in the repo:

I'll test later..

[Huevos]

I said it could possible to add overloaded function, not to modify existing one.

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/.. Could someone add some instructions to github enigma2 page?

My initial cpp and h are attached if you like to test those. Don't know if they compile or not.

Also added debug prints to original function where filename is empty or not found. I think you can commit 2 debug lines if they work, erase shouldn't normally be called with empty or non existing file.

If you get debug prints working you should see in log what filename contains in case it does not erase.

Just for your curiosity:

| components/file_eraser.cpp: In member function 'void eBackgroundFileEraser::erase(const char*)':
| components/file_eraser.cpp:87:21: error: declaration of 'std::string filename' shadows a parameter
|    87 |         std::string filename(str(filename));
|       |                     ^~~~~~~~
| components/file_eraser.cpp:85:47: note: 'const char* filename' previously declared here
|    85 | void eBackgroundFileEraser::erase(const char* filename)
|       |                                   ~~~~~~~~~~~~^~~~~~~~
| components/file_eraser.cpp:87:30: error: 'str' was not declared in this scope; did you mean 'std'?
|    87 |         std::string filename(str(filename));
|       |                              ^~~
|       |                              std
| components/file_eraser.cpp:85:47: warning: unused parameter 'filename' [-Wunused-parameter]
|    85 | void eBackgroundFileEraser::erase(const char* filename)
|       |                                   ~~~~~~~~~~~~^~~~~~~~
| In file included from ../lib/components/file_eraser.h:5,
|                  from components/file_eraser.cpp:1:
| ../lib/base/message.h: In instantiation of 'eFixedMessagePump<T>::eFixedMessagePump(eMainloop*, int, const char*) [with T = eBackgroundFileEraser::Message]':
| components/file_eraser.cpp:16:2:   required from here
| ../lib/base/message.h:187:51: warning: unused parameter 'mt' [-Wunused-parameter]
|   187 |         eFixedMessagePump(eMainloop *context, int mt, const char *name):
|       |                                               ~~~~^~
| In file included from ../lib/service/service.h:6,
|                  from ../lib/dvb/idvb.h:12,
|                  from ../lib/dvb/dvb.h:8,
|                  from components/scan.cpp:1:
| ../lib/service/iservice.h: In member function 'virtual void iServiceInformation::getAITApplications(std::map<int, std::__cxx11::basic_string<char> >&)':
| ../lib/service/iservice.h:432:69: warning: unused parameter 'aitlist' [-Wunused-parameter]
|   432 |         virtual void getAITApplications(std::map<int, std::string> &aitlist) {};
|       |                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
| make[4]: *** [Makefile:2703: components/file_eraser.o] Error 1
| make[4]: *** Waiting for unfinished jobs....

[Huevos]

Could someone add some instructions to github enigma2 page?

Example of building the enigma2 binary. Modify paths/machine to suit what is on your server.

Paths:

enigma2 git location = /home/username/6.2/builds/developer/vusolo4k/tmp/work/vusolo4k-oe-linux-gnueabi/enigma2/enigma2-5.2+gitAUTOINC+f3fe89ab92-r0/git
enigma2 binary location = /home/username/6.2/builds/developer/vusolo4k/tmp/work/vusolo4k-oe-linux-gnueabi/enigma2/enigma2-5.2+gitAUTOINC+f3fe89ab92-r0/package/usr/bin/enigma2

Run at command line:

cd /home/username/6.2/builds/openvix/developer/vusolo4k
MACHINE=vusolo4k
MACHINEBUILD=vusolo4k
. env.source
bitbake -f -c clean enigma2
bitbake -f -c unpack enigma2

(transfer your modified files to "enigma2 git location")

bitbake -c package enigma2

(retrieve enigma2 binary from "enigma2 binary location")...

[twol]

I said it could possible to add overloaded function, not to modify existing one.

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/.. Could someone add some instructions to github enigma2 page?

My initial cpp and h are attached if you like to test those. Don't know if they compile or not.

Also added debug prints to original function where filename is empty or not found. I think you can commit 2 debug lines if they work, erase shouldn't normally be called with empty or non existing file.

If you get debug prints working you should see in log what filename contains in case it does not erase.

I have run out of time to run test ...I had
initial crash but that was my issue....
modded c++ files attached...... yes I changed filename for convert

[LraiZer]

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/..

I think different people have different ways to rebuild enigma2. Here is one way!

In the build-envoronment site.conf file comment out to not INHERIT "rm_work" before building.

DL_DIR = "/home/LraiZer/sources"
#INHERIT += "rm_work"

This leaves the enigma2 source code after a build to allow editing and recompile .cpp changes.

EG.
/home/lraizer/openvix/build-enviroment/builds/openvix/release/inihdp/tmp/work/xpeedlx3-oe-linux/enigma2/enigma2-6.2+gitAUTOINC+0cd13deda6-r0/temp

Edit your changes to the enigma2 .cpp source code.

Then execute the script "run.do_compile" file, this will recompile engima2 binary only with the files that have been edited, so is very fast.

Then copy the recompiled enigma2 binary to /tmp on the set top box and run it from there in debug mode to test.

For speedy edit, test, edit, test.. this process only takes about 60 seconds to edit, recompile and transfer the new enigma2 binary to STB /tmp if you put the commands to run the script and ftp binary to STB in a build.sh script