Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[benitomuso]

Very cool Pablo, thanks so much!
Now if only we could pack our own files.SFS, that's the single thing that remains...


Mike

Do we know the SFS files structure?

Is there a document I could read somewhere?

Hi Stainless, I haven't seen this question.

Yes, I quite detailed found the structure, that's the way I created these tools.

Much help was analyzing the original SFSExtractor of QTim:

https://www.sas1946.com/main/index.php?topic=7017.0

There were the C++ sources somewhere and I started from there.

Essentially they are a pack of files subdivided in blocks, each one being a Deflate encypted one. The blocks are addressed from some sort of "table of contents".

I never had the time nor the will to document it.

Regards, Pablo.

[ssn637]

Pablo

I just wanted to express my sincere gratitude and appreciation for this wonderful tool. 

I now refer to the SFS Manager as the "little program that could" after having extracted (albeit with another SFS Extractor because the SFS Manager extract tab froze in Windows 11 after loading) every SFS archive in my BAT 4.2.1 installation to a single BATMOD folder, eliminating any duplicates from the original (which still came to a whopping 130 GB of data!) and repacking the lot into a batch of split SFS archives (a total of 488 SFS files numbered from BAT00.SFS to BAT487.SFS!).  Now for the amazing part -- it worked! I replaced the BAT00, BAT03 and BAT03B folders with the BATMOD "clean installation" and loaded up the game for a quick spin, all the while telling myself that the 3 hours I'd spent endlessly clicking the "OK" button while the SFS Manager did its thing breaking up that huge folder into hundreds of split archives would surely be futile.  Incredible!

Just a few questions for you if you don't mind my impertinence:
1. Could you please check the extractor portion of the SFS Manager V4 to see if you can replicate my issue?  I am able to load an SFS archive but cannot extract it to a separate folder -- when I click on the "extract all" command nothing happens.
2.  When splitting a folder into separate SFS archives while repacking, is there any way to get around the current limit of 250 MB per package?  This results in SFS archives that are between 60 - 90 MB in size and I'd love to be able to raise that limit to around 1.5 GB to create repacked archives of around 500 MB each.  This would greatly reduce the time needed to repack such a large folder of data.
3.  Would it be possible to run the repack command without needing to click "OK" each time an SFS archive is generated?  That way if I ever choose to challenge myself with such an immense task again I could split and repack the source folder overnight.

Thanks again for all you've done for this community!

Cheers

Geoffrey

[benitomuso]

Hi Geoffrey

Regarding your questions, I'm really out of business about MODding.

I doubt I would have time to review your points 1, 2 and 3 in the near future. Nonetheless I'm interested in your repacking of the BAT files? How big is your repacking regarding the original size of the bunch of SFSs?

If you make a clean repack of it with all the contents I would be one of the first interested persons on it.

Let me see, perhaps during the weekend I could open the project and going particularly to those issues you mention.

Regards, Pablo

[Knochenlutscher]

Hi Pablo, Hi Geoffrey,

You can split Files with Total Modder? really? I did this manually, crap, I'm dumb.

Yeah, that double click OK, good request,

Pablo? I love this Proggie, it's a pleasure to use and so powerfull. I can't mod/live without. 

Best wishes
Tobias

[ssn637]

Hi Pablo

Wow, thanks for your quick response!  I didn't expect any, to tell you the truth...

Of course I can easily understand that you've moved on to bigger and better challenges, but I'm not sure you realize just how many fans you have out there who admire and respect you tremendously for every improvement you've brought us over the years.  Thank you!

Anyway, I suppose I should explain what prompted me to try to "clean up" the SFS archives.  Carsmaster provided us with an awesome upgrade for the default concrete runways that I loved, so I thought it might be a good idea to replace the original 3do/Airfield/Concrete folder with his to make the modification permanent, rather than adding them via JSGME.  I started searching through the BAT00, BAT03 and BAT03B folders and found, to my surprise, that there are no less than nine SFS archives containing the identical file names that needed replacing!  Of course, you and all the experts are well aware of this, since these WAW and BAT archives have evolved over time, but I wondered how often this might be the case for other 3do, effects, hex files etc.  If duplicate files are scattered throughout the SFS archives, then why not clean them up to save space?

After extracting all the SFS archives and retaining only the latest versions of all duplicates I was able to save approximately 15 GB of space.  There were a lot of effects, for example, that I removed by replacing them with WxTech's V1.4 version wherever possible and creating a separate WxTechEffects.SFS archive from the lot.  I also moved any presets or sounds files to my_presets and my_sounds, since every time I launch a mission the log file shows there are presets NOT FOUND! that I then add manually after searching for their equivalents.  There were also folders like "il8n", "gui", "cache" and "paintschemes" that I deleted because they were already in the STD folder or elsewhere.  I ended up with a zillion hex files, a 3do, cod and maps folder that totalled 130 GB.  Then I simply opened this folder in SFS Manager (it took a while!), checked the option to repack them into 250 MB pieces (the max possible) and let the program run to completion.  The resulting 488 SFS archives, plus the WxTechEffects.SFS came in at just under 40 GB total. 

I'm sure I did something wrong and if anyone were to conduct extensive testing would find errors with what I came up with (for example, about a half-dozen maps failed to load, which I solved by replacing their load.ini file from the XUPMOT archves with the BAT originals), but in theory such a cleanup of the SFS archives does offer the advantage of eliminating duplicate entries and a 25% reduction of the BAT 4.2.1 total, give or take. 

I hope I was able to answer your question and, if not, please feel free to let me know if I can be of any help. 

Take care Pablo and best wishes from all of us who have for so long profited from your incredible efforts here!

Geoffrey

[benitomuso]

Hi Geoffrey.

Try this update, I think that all what you mentioned is addressed:

https://drive.google.com/file/d/1085m8jb9kRXaiPRoDwDOWmfNfGzENykc/view?usp=sharing

Regards, Pablo.

Hi Pablo

Wow, thanks for your quick response!  I didn't expect any, to tell you the truth...

Of course I can easily understand that you've moved on to bigger and better challenges, but I'm not sure you realize just how many fans you have out there who admire and respect you tremendously for every improvement you've brought us over the years.  Thank you!

Anyway, I suppose I should explain what prompted me to try to "clean up" the SFS archives.  Carsmaster provided us with an awesome upgrade for the default concrete runways that I loved, so I thought it might be a good idea to replace the original 3do/Airfield/Concrete folder with his to make the modification permanent, rather than adding them via JSGME.  I started searching through the BAT00, BAT03 and BAT03B folders and found, to my surprise, that there are no less than nine SFS archives containing the identical file names that needed replacing!  Of course, you and all the experts are well aware of this, since these WAW and BAT archives have evolved over time, but I wondered how often this might be the case for other 3do, effects, hex files etc.  If duplicate files are scattered throughout the SFS archives, then why not clean them up to save space?

After extracting all the SFS archives and retaining only the latest versions of all duplicates I was able to save approximately 15 GB of space.  There were a lot of effects, for example, that I removed by replacing them with WxTech's V1.4 version wherever possible and creating a separate WxTechEffects.SFS archive from the lot.  I also moved any presets or sounds files to my_presets and my_sounds, since every time I launch a mission the log file shows there are presets NOT FOUND! that I then add manually after searching for their equivalents.  There were also folders like "il8n", "gui", "cache" and "paintschemes" that I deleted because they were already in the STD folder or elsewhere.  I ended up with a zillion hex files, a 3do, cod and maps folder that totalled 130 GB.  Then I simply opened this folder in SFS Manager (it took a while!), checked the option to repack them into 250 MB pieces (the max possible) and let the program run to completion.  The resulting 488 SFS archives, plus the WxTechEffects.SFS came in at just under 40 GB total. 

I'm sure I did something wrong and if anyone were to conduct extensive testing would find errors with what I came up with (for example, about a half-dozen maps failed to load, which I solved by replacing their load.ini file from the XUPMOT archves with the BAT originals), but in theory such a cleanup of the SFS archives does offer the advantage of eliminating duplicate entries and a 25% reduction of the BAT 4.2.1 total, give or take. 

I hope I was able to answer your question and, if not, please feel free to let me know if I can be of any help. 

Take care Pablo and best wishes from all of us who have for so long profited from your incredible efforts here!

Geoffrey

[ssn637]

Thank you so much Pablo!

All the features I've asked for have been included with Version 4.1 and I'm in the middle of repacking everything again!

I really appreciate the time and effort you've put into this despite your busy schedule. 

Can't wait to see what the resulting SFS archives look like and how much space I'll be able to save along with the elimination of any duplicates!  This time around I'm creating one set of SFS archives from all the Hex files, another from each of the COD, 3do and Maps folders to make it easier to find individual files should I need to update them. 

Thanks again and I'll let you know how it goes when I'm done!

Cheers

Geoffrey


Update : Success! I packed the SFS archives last night, packing the Hex (Class) folder in splitted SFS files of 2000 MB at normal compression for a total of 32 SFS archives, the 3do folder in 1500 MB chunks and the Maps folder in 1000 MB chunks.  The resulting BAT00 folder totalled 39 GB for a 30% reduction compared to the original BAT 4.2.1 combined BAT00, BAT03 and BAT03B folders!  Before packing the lot I removed any unwanted files such as desktop.ini, "old", "backup", "original" and "readme" files, as well as any folders or files beginning with "-" (except for the HEX files located in the COD folder, since I wasn't sure what to do with these so I just left them alone). 

I only noted two issues during the process:
1. When running the SFS Manager V4.1, if you want to extract an SFS archive you must click on the "Extract All" button in the bottom-right corner of the GUI rather than from the drop-down menu at the top, which didn't do anything.
2. I had trouble loading the Guadacanal map (load.ini and eload.ini failing) so I simply removed its folder from the Maps archives sourced from the original BAT and XUPMOT folders, which loaded the original IL2 Maddox map instead without error.

What an incredible little program you've given us, Pablo!  A real gift to this community for which we're surely eternally grateful.  Awesome stuff!