[FIX] Image Bloat Problem (Images Not Being Deleted...)

For Articles relating to more than one ISC version
Post Reply
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

[FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by CharlieFoxtrot »

Okay, this one has been an annoying thorn in my side from DAY-ONE. There have been many threads on the topic, and (to my knowledge) ISC has never done anything about it. In the months (and years) since... if anyone has already posted a solution, then I overlooked it. (My apologies if someone else has already posted this solution.)

So... I believe I have figured out the problem that causes the "image bloat", whereby the image files for deleted products are not ALSO being deleted.

It's so astoundingly simple, that I wish I had tried to find and fix the problem sooner.

Based on my old 4.07 version of ISC... here's what I found:

In "admin/classes/class.product.php" locate "public function DoDeleteProducts($ids)" --- This is the function that is supposed to deleted products from the database and any associated image files and downloadable files.

Within this function, scroll down until you locate:

Code: Select all

while($row = $GLOBALS['ISC_CLASS_DB']->Fetch($result)) {
     @unlink(APP_ROOT."/../".GetConfig('ImageDirectory')."/".$row['imagefile']);
}
This didn't look right to me. ~ And I didn't like the fact that they used "@unlink"(which suppresses error messages) instead of a straightfoward "unlink".

In any case, I wanted to see exactly what was going on, so for testing purposes, I added the following bit of debugging code inside the "while" loop.

Code: Select all

while($row = $GLOBALS['ISC_CLASS_DB']->Fetch($result)) {
     @unlink(APP_ROOT."/../".GetConfig('ImageDirectory')."/".$row['imagefile']);
     echo APP_ROOT."/../".GetConfig('ImageDirectory')."/".$row['imagefile'] . "<br/>"; // TEMP DEBUG STATEMENT
}
Next, I copied and saved a product... then deleted it. My "echo" statements showed clearly WHY the product images were not being deleted.
/home/myaccount/public_html/store/admin/../product_images/y/mainImage.jpg
/home/myaccount/public_html/store/admin/../product_images/c/additionalImage1.jpg
/home/myaccount/public_html/store/admin/../product_images/t/additionalImage2.jpg
/home/myaccount/public_html/store/admin/../product_images/p/additionalImage3.jpg
/home/myaccount/public_html/store/admin/../product_images/p/mainImage_tiny.jpg
WHAT A MESS!! THERE'S F*CKING THE PROBLEM!! It's looking in the WRONG PLACE when trying to delete product images... AND any error messages are being suppressed with the "@" prefix.

Let's try again... change the "while" loop to be:

Code: Select all

while($row = $GLOBALS['ISC_CLASS_DB']->Fetch($result)) {
     // Notes: Original ISC code used the wrong location. Corrected code follows.
     //@unlink(APP_ROOT."/../".GetConfig('ImageDirectory')."/".$row['imagefile']);

     unlink(ISC_BASE_PATH."/".GetConfig('ImageDirectory')."/".$row['imagefile']); // CORRECTED UNLINK PATH
     echo ISC_BASE_PATH."/".GetConfig('ImageDirectory')."/".$row['imagefile'] . "<br/>"; // TEMP DEBUG STATEMENT
}
Performing the same experiment again (copying then deleting a product) produces the following output
/home/myaccount/public_html/store/product_images/s/mainImage.jpg
/home/myaccount/public_html/store/product_images/f/additionalImage1.jpg
/home/myaccount/public_html/store/product_images/o/additionalImage2.jpg
/home/myaccount/public_html/store/product_images/k/additionalImage3.jpg
/home/myaccount/public_html/store/product_images/m/mainImage_tiny.jpg
Very nice... the "unlink" command is pointed at the CORRECT location of the files... and they are NOW DELETED as they are supposed to be.

After I confirmed that all was performing as intended, I deleted the echo command ("TEMP DEBUG STATEMENT") and re-uploaded, then tested again.

FINAL NOTES: Elsewhere in this script, the erroneous "@unlink(APP_ROOT..." statement appears MULTIPLE TIMES (ie: deleting product downloads, deleting variation images, etc.) ~~ They ALL need to be changed to be "unlink(ISC_BASE_PATH..."

MORE NOTES: These changes were made to an earlier version of ISC. More recent versions may have been fixed already (doubtful) or the current code may vary from the examples I've indicated here (probably).

AS ALWAYS: Take normal precautions. Review changes carefully. Leave comments in your php code so you'll know what you changed. Comment-out the original code instead of deleting it (so that you can easily restore it if needed). Keep backups of your original php and your database in case something goes wrong. Test before going live.

DID YOU SPOT ANY ERRORS? ...
ANY CORRECTIONS? ...
ANY IMPROVEMENTS? ...
Please post them here!
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
Snooper
Posts: 264
Joined: Sat Jun 26, 2010 9:22 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by Snooper »

What a star !! :D

For public interest - In my version of ISC (5.5.4) the class.product.php is located here /admin/includes/classes and yes, it needs this script mod (hoping it works)

What we need now is somebody who can scann the blotted folder and remove the zillion unwanted images (unless you know how best this might be done ??)
Last edited by Snooper on Tue Mar 27, 2012 2:05 pm, edited 1 time in total.
ISC 5.5.4 Ultimate : Being used here -- http://www.kdklondon.com
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by CharlieFoxtrot »

Snooper: I was able to clean up my "product_images" directory... but I had to do it manually.

Unfortunately, there's no easy way to do this... but I will describe the steps that *I* took to clean up the "product_images" and remove the outdated images.

NOTES: These instructions are "concepts" only. They are NOT "step-by-step" cookbook instructions. ~~ Everyone's server is different. Everyone uses a different FTP program. Your ISP's admin interface (and capabilities) will vary... so it's up to you to figure out how to apply these concepts to your situation and admin capabilities.

1) Surprisingly the ISC backup feature appears to work well. I use it to take a daily backup of my database... and ONLY my database. (It would use far too much hard drive space to make daily backups of the "product_images" directory.)

2) However... the ISC backup feature WILL make a clean backup of the "product_images" directory. The backup script does NOT simply make a duplicate of the "product_images" directory... instead, it refers to the database and ONLY copies the product image files that are IN the database. (GOOD!)

3) So, I run the ISC backup routine and make certain to INCLUDE the "product_images" directory. (In my case, I had over 20,000 images... so the backup process takes a LONG time. I feared that the backup had timed-out and failed, but eventually it finished.)

4) The new backup is stored in the "admin/backups" folder. It will be in a separate directory with a name similar to the following:

backup-2012-03-24-09-42-34-a4da6400fb14288c43fc

5) Look in "backup-2012-03-24-09-42-34-a4da6400fb14288c43fc" and you'll see your new (clean) copy the "product_images" directory.

NOTE: THIS IS A "CLEAN" BACKUP COPY... BUT IT'S ALSO INCOMPLETE! ~~ It does contain all of the image sub-directories (a-z) but it is MISSING the following sub-directories and files (and everything contained within):

MISSING: /wrap_images
MISSING: /vendor_images
MISSING: /uploaded_images
MISSING: /configured_products
MISSING: your_custom_header_logo_image.jpg

6) Next, I logged in to the server with my FTP client... and COPIED those folders and files (listed above) to my local hard drive. (This list may be incomplete depending on your ISC version, so you may want to simply copy EVERYTHING in the "product_images" directory except for the a-z folders.)

7) I logged in to my ISP's admin interface and used the "cPanel" File Manager to do the following.

-- a) LOCATED the clean "product_images" subdirectory and RENAMED it to "product_images_clean"
-- b) MOVED the "product_images_clean" subdirectory to the store's root directory. (ie: the same directory that also contains the original "product_images")

8) Returning to the FTP CLIENT:

-- Copy the "MISSING" subdirectories and files into the "product_images_clean" directory. (This step could also be accomplished using the cPanel File Manager... but for me it was just as easy to use my FTP client.)

9) With your FTP CLIENT:

-- RENAME "product_images" to "product_images_old"
-- RENAME "product_images_clean" to "product_images"

10) Give your site a THOROUGH REVIEW... make sure everything is working as it should be.

11) If you're not strapped for hard-drive space, you may choose to leave in-place the "product_images_old" directory. ~~ If needed, you can easily restore the original "product_images" directory by renaming it again. (Assuming, of course, that you have not add any new products during this time.)

12) When you're COMPLETELY SATISFIED that everything is working properly, you can use the cPanel File Manager to permanently delete "product_images_old"
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
Snooper
Posts: 264
Joined: Sat Jun 26, 2010 9:22 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by Snooper »

hi cf..

well.. i have looked at my 'version' of class.product.php and regrettably can not match the lines you refer too. I shall continue to nose around.. I so want this to work for me !!

http://www.ihost4u.co.uk/inter/class.product.zip

but i shall implement the 'clean up' process..
Last edited by Snooper on Sun Mar 25, 2012 1:00 am, edited 1 time in total.
ISC 5.5.4 Ultimate : Being used here -- http://www.kdklondon.com
CharlieFoxtrot
Confirmed
Confirmed
Posts: 413
Joined: Sun Aug 09, 2009 1:23 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by CharlieFoxtrot »

Snooper: I examined the file you provided, and it appears that the product images information is stored in the database in a manner that is different from my version.

As near as I can tell, most of the product image information (ie: filenames) are joined together in an array ("JavaScript Object Notation" format?) which is stored in the database as a SINGLE field (named "vcimagestd") in the table named "product_variation_combinations".

I MAY BE WRONG ABOUT THIS... IT'S JUST A GUESS BASED ON WHAT I SEE.

If I am correct in my assumptions, then this appears to be a quick and efficient way for to retrieve ALL the image information in a faster (and less frequent) query to the database.

All well and good so far. When the cart needs to display the product images, this array of image information is processed internally... and the page is constructed and displayed. ~ Great!

But I've also spotted something in "public function DeleteVariationImagesForRow($row)" that puzzles me.

This function contains the following statement:

Code: Select all

if (!empty($row['vcimagestd'])) {
     @unlink(ISC_BASE_PATH . '/' . GetConfig('ImageDirectory') . '/' . $row['vcimagestd']);
}
... which appears to say:

-- If the field named "vcimagestd" is not empty
-- then assemble together the base-path + the image directory + the field contents
-- assume it's a valid file name (ie: /path/to/product_images/productphoto.jpg)
-- and unlink (delete) it.

HOWEVER... IF "vcimagestd" contains a JSON array of multiple image filenames... then I presume the "unlink" statement will fail.

ALSO... if it fails, we never know about it, because the "@" directive at the beginning of the line causes all error messages to be suppressed.

Once again... IF "vcimagestd" contains information about multiple filenames, then I assume this array would need to be parsed so that the unlink (delete) command could be executed on EACH ONE.

That doesn't appear to be happening here. (And I could be wrong... I have very little info to go on here).

So, even though the "ISC_BASE_PATH" that assembles the "path/to/product_images" is now correct... it appears that it's trying to delete the array contents instead of a valid filename.

And if my observations are correct... this could be the reason that "image bloat" continues. (Different reason... SAME RESULT!)

Please do not take my word on this... someone else who has the full code to review (and a complete database structure to review) will need to analyze this and provide more information.
ISC 4.0.7

"... and let's be honest that whole "by design" thing is getting old too."
Snooper
Posts: 264
Joined: Sat Jun 26, 2010 9:22 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by Snooper »

well.. if it has to happen, it happens to me !!

so now i need to work out what my version of script is doing. but at least the 'rosetta stone' of an idea has been found. so well done, at least a start..

as for back up and clean described in detail.

i tried to export the products database via .csv -
Fatal error: Allowed memory size of 94371840 bytes exhausted (tried to allocate 8388608 bytes) in Unknown on line 0

and then i attempted a back up on my whole sale site and had returned an error ??!!
Notice: Undefined index: backup method in /homepages/10/dxxxxxxxx/htdocs/admin/includes/classes/class.backup.php on line 413

http://www.ihost4u.co.uk/inter/class.backup.zip

and then i tried to backup my retail site it while both sites gave me a folder, nothing was actually saved/ transferred.

so with two sites both using ISC 5.5.4. (ultimate edition) hahahahahahahaha *coughs splutters* :?

i wonder who else might have the same luck as i did and why ISC changed the script. this is as frustrating as standing outside a sweety shop knowing the chance of reward is just in front of you, yet blocked by something like glass. while somebody proving its obtainable once you figure how.. okay, maybe a crap analogy..

blimmin dingally dell :|
ISC 5.5.4 Ultimate : Being used here -- http://www.kdklondon.com
Snooper
Posts: 264
Joined: Sat Jun 26, 2010 9:22 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by Snooper »

I am at the experimental stage of finding a way around this problem..

But my approach is to read the table pointers from within the SQL table called isc_product_images and pull every image down into a folder on my PC and so creating a mirror. That way I assume that this will have ONLY the images relevant to my current data and store. What is held in the original file on the server will be the one holding ‘dead’ images.

Then it will be simply a matter of adding other folders not copied to complete the mirror and upload the result back to the server. A quick integrity check before you remove the old folder and connecting your newest

BUT so far, its a relistic step nearer to clearing bloated images and it works because I have a draft sample working !!! :D

Okay, no error checking or time out prevention. Something I am currently resolving. Plus, there is no visual to allow you to know what is going on..
ISC 5.5.4 Ultimate : Being used here -- http://www.kdklondon.com
joshmods
Posts: 2
Joined: Fri Nov 06, 2009 11:27 pm

Re: [FIX] Image Bloat Problem (Images Not Being Deleted...)

Post by joshmods »

Please don't run your program that just looks at the product_images table. You'll be removing all variation images. Not to mention category images, image manager/uploaded images, brand images, store logos, etc.
Post Reply