Drive free space vanishing

[speedygonzalez]

After having flashed the box with the Willo build (which was on 4.2.030), I found that my 1TB internal hdd (supplied very recently with the box by WoS) freespace went from 99% to 33%, after having left it on for only a few days for just a few recordings.

In particular many other users are also reporting the problem, including at least one user (aceone in Techkings) running Openatv 6, see:
http://www.world-of-satellite.com/showthread.php?55917-What-is-eating-my-HDD-space
and

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/#post-680916

I have tried many tests and potential fixes, but the problem persists:
1) Redirecting /hdd/movie and timeshift to a 16GB USB stick formatted in ext4
2) Flashed 5.0.002
3) Turning off timeshift
4) Turning off swap
5) Performing a reinitialisation of the hdd, in ext4, via the menu
6) Performing a file system check on the hdd, via the menu. Like a restart this was however effective in returning the lost space, but the problem did not go away

I have found that:
a) Executing fsck -alR on the hdd, and indeed from root, via puTTY, showed no sizeable file that could possibly have explained the lost space
b) Executing fsck or hard restart / reboot from the menu are effective in reclaiming the lost space
c) Recorded programme file sizes are as expected, timeshift file sizes (that are visible by Filezilla or ls-a) do not appear excessive
d) Recording programmes, allowing timeshift to be on, or pressing the pause button on the remote all give rise to hdd freespace lost
e) On the other hand hdd freespace does not change (between restarts e.g.) by leaving the box on without recording programmes as long as timeshift is disabled and not pressing the pause button
f) It is of course not conclusive, from just reading posts and asking other kind users to do tests on their machines, but I have seen no evidence yet that any user is free from this hdd freespace loss phenomenon, if they record and delete programmes, or have timeshift on.
g) Disabling timeshift might have lowered the rate of freespace loss. At one stage I was experiencing ~8GB of loss an hour.

Many thanks for the help in advance!

Cheers!

[Andy_Hazza]

Try setting up without a pre done build. Do a fresh flash of ViX 5.0.1 & setup from scratch and enable logs and see what happens.


Sent from my iPhone using Tapatalk

[birdman]

See other thread:

http://www.world-of-satellite.com/showthread.php?55917-What-is-eating-my-HDD-space&p=442776&viewfull=1#post442776

[speedygonzalez]

Try setting up without a pre done build. Do a fresh flash of ViX 5.0.1 & setup from scratch and enable logs and see what happens.



Andy I have set up a clean 5.0.001 build from scratch without restore as you suggested.

I had 942.78GB free, then I recorded two BBC HD programmes, in total around one hour, which according to Filezilla were just over 2.2GB, but freespace dropped to 911.16GB!

I always read freespace off Menu / Information / Devices

Then I deleted those programmes, waited and waited and yet freespace did not increase at all, remaining steadfast at 911.16GB. The fact that no increase of freespace at all is unexpected, because in previous builds the freespace always increase after programme deletion, but not enough. The difference "might" have been caused by the fact that I stopped those recordings by the red button in the PVR menu, rather than letting those programmes finish recording, but either way the symptom is still not justifiable.

Then I executed the command

find /media/hdd -type f -size +50000k -exec ls -alh {} \;

which member ccs here kindly suggested earlier in the other thread - as expected from my previous exploration, the biggest files found were the recordings, all totalling nothing like the 30GB+ that is missing...

Then I restarted, now freespace is 940.11GB. The difference with the 942.78GB might have been caused by my setting Logging up while the programmes were recording.

I noticed, from monitoring by Filezilla, that recording file size changed only every 10:00 minutes. It would mean the system writes into a temporary workspace in the intervening time and only copy/overwrite the recording at those instances. It is conceivable that the system fails to release those temporary workspace properly.

Now having restarted, logging has started. I will record something, let it finish normally, and see what comes up.

I will also try to figure out how best to run the script:

#!/bin/sh
#
cd /proc
for l in [0-9]*/fd/*; do
( /bin/ls -l $l | grep '(deleted)$') 2>/dev/null
done

birdman kindly provided me with in the other thread.

As I mentioned in the original bug report this morning, a user running Openatv 6 is having the same problem, which probably means the problem is probably not affected by the build, but is something far more fundamental / low level within the system.

Cheers!

[ccs]

Don't forget that deleted programmes go into .trash for n days, which is configurable.

The script can be run via telnet, by simply issuing each command, line by line, starting with cd /proc.

(I tried it, and it found no files on my box.)

[speedygonzalez]

Don't forget that deleted programmes go into .trash for n days, which is configurable.

The script can be run via telnet, by simply issuing each command, line by line, starting with cd /proc.

(I tried it, and it found no files on my box.)

Thanks for letting me know. I have now disabled trash and restarted.

Cheers!

[speedygonzalez]

Folks, I have done more tests as discussed above.

In case there is any confusion, timeshift has never been enabled for this clean 5.0.001 build, and I did not touch the pause button.

Trash is now disabled, which I established to be the case after deleting a programme - incidentally now without trash on after programme deletion the freespace does increase (like I have been experiencing before this build), and the increase took its time (over a few minutes) - another phenomenon which is also consistent with my previous builds. so I suspect my previous builds had trash disabled.

Unfortunately, the increase in freespace after programme deletion is again coming up very short. The hdd freespace gobbling monster is alive and well!! :mad:

After rebooting and before recording, freespace was 911.16GB.

After recording the programme, the size of which I established via both Filezilla and PuTTY to be 2GB, freespace was 889.81GB, i.e. over 21GB gone.

I then ran the script shown above that Birdman kindly wrote for me, it came up with nothing.

After deleting the programme with the red button in PVR, I waited the few minutes for freespace to stabilize (which incidentally coincided with the programme files no longer visible by Filezilla), freespace became 891.74GB. Essentially over 19GB vanished

I am sure if I restart, I will get the lost freespace back. But I am going to wait, so that I do not destroy useful evidence that might help hunting this bug.

I have attached the log, unfortunately, it is gobbledegook to me... :confused:

Cheers!

[bbbuk]

Some commands which may help:

Displays disc usage on flash as well HDD and/or USB
df -mh

Puts box to sleep, unmounts HDD, and forces a check of disc (ensure you don't have recording or timeshift in operation - /dev/sda1 may differ also)

init 4
umount /dev/sda1
e2fsck -p -c -f /dev/sda1


With regards to debug log, the following sticks out for me:

There were a lot of the following messages...

[eDVBRecordFileThread] Waiting for I/O to complete
[eFilePushThreadRecorder] Warning: All write buffers busy

and also I found this:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do

Forgive me as I haven't read any previous threads over this but is your HDD powered itself or via USB?

[birdman]

Observations on the log.

(Unfortunately it has no timestamps - they've been turned back on for future release - so we can't work out when things happened in relation to others).

There are three recordings mentioned in the log.


[RecordTimer] Record RecordTimerEntry(name=\86Building Dream Homes\87, begin=Tue Mar 14 13:57:00 2017, serviceref=1:0:19:1B1C:802:2:11A0000:0:0:0:, justplay=0, isAutoTimer=0)
[RecordTimer] Record RecordTimerEntry(name=ITV Racing Live: Cheltenham..., begin=Tue Mar 14 12:57:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}

The \86 \87 characters are "Start of selected Area" and "End of selected Area". Odd (possibly).


The presence of the first two is a little odd, given that the log appears to show enigma2 stating up at 15:56:46.
The first doesn't seem to have activated a recording.
The second does, as does the third.

There are many entries like this (which isn't good):

[eFilePushThreadRecorder] Warning: All write buffers busy
[eDVBRecordFileThread] Waiting for I/O to complete

Then this shows up as one recording stops:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do.

Plus there are lots of reports about the EPGcache, like this:

[eEPGCache] FS subtable (60280000) version changed (12) now/next (1)
[eEPGCache] FS subtable (60280000) version changed (12) now/next (1)which may, or may not, be significant.

However, there is definitely something amiss with your storage system.
Are you using a local HDD, or a remote file-server?

[speedygonzalez]

The hdd is a Toshiba 1TB drive (MQ01ABD1), supplied inside the box, by WoS, a couple of weeks ago.

Cheers!

[speedygonzalez]

Some commands which may help:

Displays disc usage on flash as well HDD and/or USB
df -mh

Puts box to sleep, unmounts HDD, and forces a check of disc (ensure you don't have recording or timeshift in operation - /dev/sda1 may differ also)

init 4
umount /dev/sda1
e2fsck -p -c -f /dev/sda1


With regards to debug log, the following sticks out for me:

There were a lot of the following messages...

[eDVBRecordFileThread] Waiting for I/O to complete
[eFilePushThreadRecorder] Warning: All write buffers busy

and also I found this:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do

Forgive me as I haven't read any previous threads over this but is your HDD powered itself or via USB?


Observations on the log.

(Unfortunately it has no timestamps - they've been turned back on for future release - so we can't work out when things happened in relation to others).

There are three recordings mentioned in the log.


The \86 \87 characters are "Start of selected Area" and "End of selected Area". Odd (possibly).


The presence of the first two is a little odd, given that the log appears to show enigma2 stating up at 15:56:46.
The first doesn't seem to have activated a recording.
The second does, as does the third.

There are many entries like this (which isn't good):


Then this shows up as one recording stops:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do.

Plus there are lots of reports about the EPGcache, like this:
which may, or may not, be significant.

However, there is definitely something amiss with your storage system.
Are you using a local HDD, or a remote file-server?


I am sorry bbbuk and birdman, I don't think I am adequately knowledgeable or competent to comment, but am of course more than happy to answer any question if I can. The only programme, that was recorded in this session immediately after a restart to ensure a "clean start" with trash can disabled just before 4pm, was this:

[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}

The other recordings are old and there are other old recordings not listed for whatever reason. I am happy to delete them all if and when it helps.

bbbuk what I do know, is if I do a file system check on the hdd, then all the lost space will be recovered.

[bbbuk]

bbbuk what I do know, is if I do a file system check on the hdd, then all the lost space will be recovered.I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?

[speedygonzalez]

I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?

I haven't got a free hdd at the moment, but I do have a 32GB usb stick, but it might be too small? And do I have to rip the internal hdd out?

A data point that might suggest the issue lies elsewhere, is that users at the other thread here indicate their different disks (of 320GB, 500GB and 1TB) are having the same issue?

Is there any other similar angle/tool applicable, apart from birdman's script he kindly provided and reproduced above for "file deleted but not closed", that might be able to identify the "orphaned" hdd space either when it was still being created/filled, or after it was created/discarded?

I guess I am wondering if the space was ever a "file", because if it was, shouldn't it have been spotted by e.g. ls -alR during recording, e.g., while it was still in use and before it was deleted?

I therefore also wonder how many ways are there to "take possession" of hdd space in linux, and what code can spot them all.

My apologies in advance, if my train of thought makes no sense...

Cheers!

[kokojnr]

I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?
Either the HDD or the Receiver, because this issue seems to affect only users of EDISION MEGA, including myself.

Sent from my Moto G (4) using Tapatalk

[judge]

Still don't get the issue? or the paranoia about HDD space being used up?

[birdman]

I guess I am wondering if the space was ever a "file", because if it was, shouldn't it have been spotted by e.g. ls -alR during recording, e.g., while it was still in use and before it was deleted?It's perfectly legal to open a new file and immediately delete it. Since you still have a file-handle to it it still exists in the file system and you can write to and read from it. Several things do this for security (if you cant access the file-handle, you can't access any of the data) and simplicity of tidying up (once the process dies - for what ever reason - the filespace gets freed).

One thing you could try (perhaps you already have?) is just restarting the GUI, rather than restarting the system.

[birdman]

The only programme, that was recorded in this session immediately after a restart to ensure a "clean start" with trash can disabled just before 4pm, was this:

[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}Not according to the log, which has:


[RecordTimer] Record RecordTimerEntry(name=ITV Racing Live: Cheltenham..., begin=Tue Mar 14 12:57:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)
[RecordTimer] activating state 1
[RecordTimer] Found enough free space to record
[RecordTimer] Filename calculated as: '/media/hdd/movie/20170314 1257 - ITV London HD - ITV Racing Live_ Cheltenham___'

and


[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)
...
[RecordTimer] activating state 1
[RecordTimer] Found enough free space to record
[RecordTimer] Filename calculated as: '/media/hdd/movie/20170314 1627 - ITV London HD - Celebrity Home Secrets'


Although since the restart appears to have been at 15:56 it's possible that the first one is a recording that was actually running when you restarted the box, so you should have had a prompt about that - (how do you restart your box?) - and when enigma2 restarted it found that it had a timer recording that hadn't yet reached it's end time.

[speedygonzalez]

It's perfectly legal to open a new file and immediately delete it. Since you still have a file-handle to it it still exists in the file system and you can write to and read from it. Several things do this for security (if you cant access the file-handle, you can't access any of the data) and simplicity of tidying up (once the process dies - for what ever reason - the filespace gets freed).

One thing you could try (perhaps you already have?) is just restarting the GUI, rather than restarting the system.

Thanks for the explanation regarding the file handling. I have just restarted the GUI, it makes no difference to the lost hdd space.


Not according to the log, which has:

and

Although since the restart appears to have been at 15:56 it's possible that the first one is a recording that was actually running when you restarted the box, so you should have had a prompt about that - (how do you restart your box?) - and when enigma2 restarted it found that it had a timer recording that hadn't yet reached it's end time.

Birdman you are good! You are absolutely right regarding the 2nd recording, because now looking at Filezilla I can see the Cheltenham racing one finishing at 4:35pm, I must have programmed it earlier and forgot it was there, and it never crossed my mind that the recording would automatically restart in later sessions (programme officially ran from 12:57 to 16:35 according to the timer record), and it therefore continued to record for about 36 minutes, consuming 2.2GB, after the restart during the logged session. Sorry for forgetting - I was picking random HD channels to make test records, and lost track of them...

There were earlier 'Cheltenham racing' recordings but they were manually deleted before the restart. So for completeness/correctness, 4.2GB associated with 2 actual programmes were recorded during the logged session. I deleted one of the programmes, 'Celebrity Home Secrets', 2GB in size, to find hdd space of around 17GB remained missing.

I have always restarted the box by Menu / Standby and Restart / Restart

Is the code that should have released the lost hdd space likely part of Enigma/OpenVix, or lower level within the system (e.g. in machine code, or part of the disc controller firmware)?

Cheers!

[ccs]

What does the telnet command df -T produce?

There is still the possibility that the hdd is faulty, looking thru' earlier posts.

[ljmramst]

Still don't get the issue? or the paranoia about HDD space being used up?
Having only 32% space left on a 500gb hdd (Seagate installed by WOS) after only setting up epg and making a test recording of approx 2 minutes. The only way to retrieve lost space is to cut power every time.

That's my problem in a nutshell and why I started the other thread. Speedygonzalez stated on that thread he opened this one as he believed it was the proper way to gain official support.

I wouldn't call it 'paranoia' when the box is virtually useless as a recording device in its current state.

[birdman]

I have just restarted the GUI, it makes no difference to the lost hdd space.Then it's not enigma2 that is holding on to the space. It's some other process.

[birdman]

Still don't get the issue? or the paranoia about HDD space being used up?Imagine you get up in the morning, look in the fridge and see that you have 2 pints of milk.
An hour later you look again, knowing that the fridge door hasn't opened in the meantime, and now you only have 1 pint.
Are you saying that you would take this to be normal and just accept it?

[Sicilian]

At the OP can you please try OpenPLi and report back.

[speedygonzalez]

What does the telnet command df -T produce?


root@osmega:/media/hdd# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
ubi0:rootfs ubifs 445324 72884 372440 16% /
devtmpfs devtmpfs 284120 4 284116 0% /dev
tmpfs tmpfs 64 0 64 0% /media
tmpfs tmpfs 284268 156 284112 0% /var/volatile
/dev/sda1 ext4 976559616 63412736 912098304 7% /media/hdd

I am not 100% sure what it shows, but it seems to show the missing 17GB is still missing.



There is still the possibility that the hdd is faulty, looking thru' earlier posts.


You might be right, but isn't it odd that no less than 7 posters are experiencing the same phenomenon from faulty hdds, of different makes and at least 3 different sizes, all on OS Mega, all at the same time?

I suspect we might be able to work out the likelihood, if we knew roughly how many these relatively new machines have been sold, with a guesstimate of typical hdd mtbf. Conversely I am sure many many more zgemma's, e.g., are in use, but has any concurrent mass hdd failure producing the same free space loss phenomenon ever been observed?

[speedygonzalez]

At the OP can you please try OpenPLi and report back.

Fyi another user on OpenATV 6 is suffering the same problem, please see:

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-3#post-681909

and

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-3#post-682295

Also doesn't a GUI restart having no effect on the missing space, hence


Then it's not enigma2 that is holding on to the space. It's some other process.

precludes it has to do with OpenVix/OpenAtv or OpenPli?

Cheers!

[ccs]

On the 2 threads running at the moment, I counted 4 posters with problems, still unlikely I know, but it looks like OS mega is the common denominator.

I don't see what zgemma's have to do with sorting out this issue.


has any concurrent mass hdd failure producing the same free space loss phenomenon ever been observed? :)

[Sicilian]

Fyi another user on OpenATV 6 is suffering the same problem, please see:

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-3#post-681909

and

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-3#post-682295

Also doesn't a GUI restart having no effect on the missing space, hence



precludes it has to do with OpenVix/OpenAtv or OpenPli?

Cheers!

Can you try Openpli please and report back.

[speedygonzalez]

On the 2 threads running at the moment, I counted 4 posters with problems, still unlikely I know, but it looks like OS mega is the common denominator.

I don't see what zgemma's have to do with sorting out this issue.

:)

Sorry I am also counting OS Mega users. reporting the same problem, all within the past few days, at the other website... ;)

I referred to zgemma because there are many more of them, so far more hdds, far more hdd failures, so we should see many more similar whinges, if the phenomenon is associated with hdd failure? I might be wrong, but it seems the common denominator of the problem is the OS Mega box, not hdd failures per se.

[speedygonzalez]

Can you try Openpli please and report back.

Sorry I do not see any Openpli download in your website? Is it supported by WoS? I do not want to step on potentially tricky legal ground here, e.g. in terms of warranty, given my machine is new.

Even https://wiki.openpli.org/installation does not refer to this box, only other boxes, in their installation guide at their website? I am sorry, but I have never even touched one of these boxes until 3 weeks ago... :confused:

Wouldn't someone at or associated with WoS try it instead?

Cheers!

[Andy_Hazza]

ViX don't provide images for OpenPLi etc, google is your best buddy for this.


Sent from my iPhone using Tapatalk

[ljmramst]

There's these links for os mega at https://openpli.org/download/edision/OS+mega/
Appears to be still at beta stage and that makes me uncomfortable as a novice re. warranty issues if I start putting stuff like that on.

[Sicilian]

Openpli is always beta!

There are no warranty issue flashing an image if flashed correctly.

I'm testing OpenViX now for this issue.

[Sicilian]

This is NOT a hardware issue, looks like a software issue bought in with the latest linux kernel. I've been testing this on various receivers, all report more HDD space used than actual recording size. Some more than others, after reboot HDD space is reported back correctly.

Seems to be more apparent on latest kernels. Mega is using 4.10 kernel. I'm reporting to Edision.

[speedygonzalez]

This is NOT a hardware issue, looks like a software issue bought in with the latest linux kernel. I've been testing this on various receivers, all report more HDD space used than actual recording size. Some more than others, after reboot HDD space is reported back correctly.

Seems to be more apparent on latest kernels. Mega is using 4.10 kernel. I'm reporting to Edision.

Hi Sicilian just in case it helps speeding up the bug hunt, it seems two other users, who have older machines, but using the same 4.10 kernel, are not experiencing the problem.

Please see
http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-4#post-683286

On the other hand, their machines appear to be older. Willo got his on 23 December (he posted the fact), and maxal indicated he got his on 20 January.

Doesn't it suggests that the kernel is unlikely to be the issue, while the hardware or associated firmware is?

I just wonder if you have an older machine, back for repairs e.g., that you could test?

Cheers!

[Sicilian]

Hi Sicilian just in case it helps speeding up the bug hunt, it seems two other users, who have older machines, but using the same 4.10 kernel, are not experiencing the problem.

Please see http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/page-4#post-683286

On the other hand, their machines appear to be older. Willo got his on 23 December (he posted the fact), and maxal indicated he got his on 20 January.

Doesn't it suggests that the kernel is unlikely to be the issue, while the hardware or associated firmware is?

I just wonder if you have an older machine, back for repairs e.g., that you could test?

Cheers!

It's got nothing to do with hardware. Even older images with 4.9 kernel have the same issue.

Issue will be reported to Edision tomorrow.

For now reboot the receiver daily.


Sent from my iPhone using Tapatalk

[Sicilian]

I've just spent hours testing this issue with OpenViX, OpenATV and OpenPLi. Issue exists in OpenViX and OpenATV, not in OpenPLi. As i've stated this is not a hardware issue, it's a software bug. Possibly a bug in OE-Alliance core somewhere that both OpenViX and OpenATV share. OpenPLi has their own core.

Issue has been reported to Edision. Anyone bothered by this issue for now I'd suggest a daily reboot timer or use OpenPLi until fixed in OE-Alliance images.

To set a reboot timer, Menu > Timers > Power timers > Press Green to create a reboot timer daily.

Edision normally respond pretty fast to software issues, so hopefully a fix will not be too far away.

@ Speedy, ALL OS Mega in the public domain right now are ALL from same manufacturing batch, this is NOT a hardware issue. It's a software issue.

[Sicilian]

A fix for this issue is in the process of being tested. Cause has been found. Will be in next release build.

[speedygonzalez]

A fix for this issue is in the process of being tested. Cause has been found. Will be in next release build.

Sicilian, 5.0.004 is indeed effective - thank-you and the team for wasting no time at all in tackling it, and 100% proactive in identifying the extent of and fixing the problem!

Cheers!