Sdhc Compressed File System?


TitanUranus

Active Member
Joined
Oct 6, 2009
Messages
756
Location
UK
Ok, so far there's been some questions about the best format for SDHC cards. In my eyes, one cruicial problem is space. Face it, there's a gigabyte arms race out there, and with more modern applications and systems, a few gigs doesn't seem as impressive as they once did. Bigger in this case is obviously better, and we wan't our megs and we all want them NOW!

I'm fairly new to linux, but one feature I've always loved about NTFS is the ability to turn drive compression on. Back in the days of poxy little 20mb/40mb HDD's drive compression was virtually mandatory. It also had the bonus that most of the algorythms were... er ... 'zippy' enough that it actually saved time accessing slow drives even on the most piss poor hardware.

This is a simple enough to answer I'm sure for all you clever people, but please indulge me. Will compressed NTFS work with SDHC's used in the Pandora? Is there already a better linux alternative? What's the gains? What's the losses? If less megs have to be read/written to SDHC, won't it increase read/write times as well as life span? Are there any drawbacks? Has it been tested already? If not why not? And why do I have to ask so many questions about something so simple to answer?

EDIT: I recall something about user privelages being a bit of a PITA with FAT file systems, but I'm sure that will dissapear with the next firmware release. Can SDHC be formatted to NTFS compressed anyway? I know they can be formatted to linux ext2 cause someone said so.

EDIT2: I'd swear that I wrote SDHC (as is correct for an anacronym) not Sdhc with 3 lower case letters in the title. Please sort the title filter so it only corrects me when I'm wrong!
 
Thanks for the reply. Has anyone with a pandora tried using an SDHC with compressed NTFS?
 
For a read-only compressed filesystem, the choice is obvious: Use SquashFS.

If you want a read-write compressed filesystem, it's a bit trickier. You may want to look into JFFS2 or fusecompress, but it seems that the performance drop is quite costly.

Most of the time where you want to use compression is where you have a very fast CPU relative to I/O speed and space. You can actually sometimes gain speed with filesystem compression in this situation as less data goes through the slow I/O and while it has to be decompressed, that's taken care of by a fast CPU. That said, the only filesystem I tried that actually pulled this off was ZFS on Solaris running a fast x86 server using mirrored hard drives. You won't be running that on a Pandora!

I think that because there's a lot less CPU to go around on the Pandora, transparent filesystem compression will not be worth it. You'll gain some capacity, but I bet read and especially write performance (from using CPU resources) will drop enough for it not to be practical.
 
As I recall, you can infact create a compressed loopback filesystem (backed with a file).

Then mount the FS on that compressed loop device.

Its called 'cloop' (never used it myself - plain 'ol RAM back loopbacks are STUPIDLY fast... anyways)

Also - does the Pandora use FS labels?

e.g.

/media/my_sd_card

Vs.

/media/mmcblk1p2

(which is stupid as you might as well use the label of the FS to create a mountpoint -- like modern Linux desktops already do...for YEARS)
 
Would a "compressed loopback filesystem" be similar to the loopbacked file I installed Linux on in the xbox? I always kind of assumed that since fatx filesystem on Xbox didn't seem to allow certain things, and the XDS Linux installed its self to a kind of preallocated solid file, I used to think of it as being like a "virtual hard disk", is that the totally wrong or is it roughly how it works?

If this is roughly right, this could be gold! You wouldn't have to dedicate the whole SDHC, but could allocate say the first 4gb of it to a storage block, that would contain the compressed files. The rest of the card could be used for normal every day usage.

The SquashFS sounds interesting. Perhaps a slow SDHC could be used to store all the CD iso's and rom images on it. Would be especially good for emulators that dont allow compressed roms. But how would it affect the actual software? Would it run quietly in the background without impacting on speed, or would things start stuttering as info is decompressed?

With NTFS compression it's never seemed to make much difference to the speed things on any of my PC's, it's maybe not the most efficient compression but it's still a saving, and any data space saving is a bonus. If someone has the time, perhaps they could try formatting an SDHC and trying it on a Pandora to see if it reads/writes properly, and if any slowdown or even speed increase is noticable.

EDIT: With SquashFS, why is it read only? And (maybe I'm being thick here) how do you write the damn stuff onto it in the first place?
 
TitanUranus said:
Would a "compressed loopback filesystem" be similar to the loopbacked file I installed Linux on in the xbox? I always kind of assumed that since fatx filesystem on Xbox didn't seem to allow certain things, and the XDS Linux installed its self to a kind of preallocated solid file, I used to think of it as being like a "virtual hard disk", is that the totally wrong or is it roughly how it works?

If this is roughly right, this could be gold! You wouldn't have to dedicate the whole SDHC, but could allocate say the first 4gb of it to a storage block, that would contain the compressed files. The rest of the card could be used for normal every day usage.

The SquashFS sounds interesting. Perhaps a slow SDHC could be used to store all the CD iso's and rom images on it. Would be especially good for emulators that dont allow compressed roms. But how would it affect the actual software? Would it run quietly in the background without impacting on speed, or would things start stuttering as info is decompressed?

With NTFS compression it's never seemed to make much difference to the speed things on any of my PC's, it's maybe not the most efficient compression but it's still a saving, and any data space saving is a bonus. If someone has the time, perhaps they could try formatting an SDHC and trying it on a Pandora to see if it reads/writes properly, and if any slowdown or even speed increase is noticable.

EDIT: With SquashFS, why is it read only? And (maybe I'm being thick here) how do you write the damn stuff onto it in the first place?
I can answer your SquashFS questions.

It's read-only because it was only designed to decompress at the fs level and to be fixed size , not to compress more data while it's live (I believe the filesystem targets embedded systems with limited storage and no need for persistence). The initial SquashFS image is created by mksquashfs, which works pretty much like making a .tar.bz2 or .zip file. See the man page for mksquashfs here. After the image is created (compression takes a while), the on-demand decompression is pretty fast. If you need to mount it read-write (some apps complain or won't start if they can't write something, etc) but don't need the new data to actually be saved, you can combine the squashfs image with tmpfs (which stores in ram) using unionfs or aufs. If you do that, newly written files will work properly and last until you unmount or reboot at which point they will vanish, just like a LiveCD (A lot of LiveCDs use this very combination, SquashFS + tmpfs + unionfs)

Think of a squashfs image as a .zip that you can mount. If you want to update the data in a SquashFS image, you can extract a squashfs image (Use the unsquashfs command), add to the extracted contents, and resquash it if you have the spare space to do so.

You don't have to use images either. You can use partitions, but growing and shrinking them is much more of a hassle than just having the squashfs filesystem image as a file, which Linux can then mount (use the "-o loop" option).

One more thing. I don't think the Pandora comes preloaded with mksquashfs and unsquashfs, if you don't want to work with the images on a separate Linux PC. There's probably a package in the angstrom repository called squashfstools or squashfsprogs or something like that.
 
Last edited by a moderator:
Thank you for your answer Drack, SquashFS sounds really interesting. I might try it on my PC sometime. If it could be used on the Pandora it could be perfect for storing PSX iso's or CD images used in dosbox, unless of course the emulators support compressed iso's. The MSDOS Buzz Aldrins race into space is a 589mb iso normally, but compressed it goes down to 350mb (less if you use 7zip I guess, but I doubt that level of compression would be useable). This might be a silly question, but will it work on an ARM linux distribution? Have you tried it yourself, and do you think you will use it if/when you get a pandora?
 
TitanUranus said:
Thank you for your answer Drack, SquashFS sounds really interesting. I might try it on my PC sometime. If it could be used on the Pandora it could be perfect for storing PSX iso's or CD images used in dosbox, unless of course the emulators support compressed iso's. The MSDOS Buzz Aldrins race into space is a 589mb iso normally, but compressed it goes down to 350mb (less if you use 7zip I guess, but I doubt that level of compression would be useable). This might be a silly question, but will it work on an ARM linux distribution? Have you tried it yourself, and do you think you will use it if/when you get a pandora?
I'd be surprised if it weren't available for ARM. I know OpenWRT uses it (It's a linux distro for routers), and those routers use MIPS chips.

Personally, would I use it? Well, it really depends on what the apps support. If DOSBox or the eventual PSX emulator support compressed (probably gzipped) isos, then there's no real need to go through the trouble of making a compressed filesystem image.

SquashFS really shines in two circumstances:
1) You are limited in capacity
2) Much of the data you want to store needs to be compressed at the filesystem level - app doesn't support zips, for example

Both of these circumstances depend on two conditions to realize the benefit:
-Data is sparse, ie compressable. For example, putting Zips, JPEGs or MP3s (These formats are already compressed) on SquashFS will not save any space. Things that DO compress well are source code, isos (sometimes. Movies are already compressed. Games are usually very compressable), ROMs, text documents, programs, etc...
-Data does not change frequently. Resquashing is possible but annoying.

And as for when my Pandora comes .. I'm 350-450 on Craig's shop, no email yet and I'm airmail to the USA, so it will probably be a few weeks.

I did use the squashfs + tmpfs + unionfs combo in an encryption experiment. Basically, I had a clean system squashfs image on a USB stick that worked like a LiveCD and didn't save changes, but I had a shell script in there that would update the squashfs with what was currently there so I could update the base if I wanted to. Everything was encrypted except a tiny boot partition with just the kernel and an initramfs that used LUKS to decrypt the system and boot it. I carried it on my keychain and could just plug and play with any PC that could boot from USB, was secure in that I would lose no personal data if the key got stolen, and always had a clean system with me for fixing people's broken Windows boxes. On top of that, it was just a really cool and geeky thing to do. I used Debian for that, by the way.
 
Last edited by a moderator:
I really like this idea, however:

1) PND files can be compressed (SquashFS I think), Ed says he uses it with his (and there is minimal CPU overhead)
2) Most modern media formats are compressed already (MP3, MP4, Jpeg etc.)
3) Many emulators support zipped roms

Although:

1) Not every PND will be compressed
2) Compressed media formats are lossy and you lose quality
3) Emulators might not support zipped roms (as you say)

I'm interested in any solutions. :)
 
I use BTRFS on my netbook as its SSD is really slow. With compression enabled, it takes less time to write to the SSD. I guess it could be used for SD cards.
 
Staying on the topic of compression but extending to PSX ISOs, isn't there a sorta "compressed" format that just throws away a lot of useless stuff that's part of the CD image but the game doesn't use, probably similar to trimming/scrubbing the leading junk off Wii images or throwing away subchannel for games that don't use it or error correction code that don't really matter when not reading off a CD.

Also straight gzipping a .iso would be rather bad. You'd probably want to compress with deflate or something similarly fast on a per-sector/block level. With such small chunks, compression ratio will likely suck, but otherwise, you'd have to decompress up until the point you need to read from if you compress the whole thing solid...

So combining the idea from the first paragraph which I believe I've seen in action but may be only on windows-only emulators, not sure and the compression idea from the second paragraph, you'd probably get something a bit more beneficial to PSX image compression than throwing it in a squashfs.
 
paulguy said:
Staying on the topic of compression but extending to PSX ISOs, isn't there a sorta "compressed" format that just throws away a lot of useless stuff that's part of the CD image but the game doesn't use, probably similar to trimming/scrubbing the leading junk off Wii images or throwing away subchannel for games that don't use it or error correction code that don't really matter when not reading off a CD.

Also straight gzipping a .iso would be rather bad. You'd probably want to compress with deflate or something similarly fast on a per-sector/block level. With such small chunks, compression ratio will likely suck, but otherwise, you'd have to decompress up until the point you need to read from if you compress the whole thing solid...

So combining the idea from the first paragraph which I believe I've seen in action but may be only on windows-only emulators, not sure and the compression idea from the second paragraph, you'd probably get something a bit more beneficial to PSX image compression than throwing it in a squashfs.

It's late, I'm sore and irritable, and I need to go to bed. There's nothing wrong with what you are saying, except that it is not "Staying on the topic of compression". I don't have a problem with this thread going off at an ISO ripping tangent, so carry on. I'm just pedantic, and don't like you trying to do it under the guise of staying on topic.
 
Last edited by a moderator:
paulguy said:
Staying on the topic of compression but extending to PSX ISOs, isn't there a sorta "compressed" format that just throws away a lot of useless stuff that's part of the CD image but the game doesn't use, probably similar to trimming/scrubbing the leading junk off Wii images or throwing away subchannel for games that don't use it or error correction code that don't really matter when not reading off a CD.

Also straight gzipping a .iso would be rather bad. You'd probably want to compress with deflate or something similarly fast on a per-sector/block level. With such small chunks, compression ratio will likely suck, but otherwise, you'd have to decompress up until the point you need to read from if you compress the whole thing solid...

So combining the idea from the first paragraph which I believe I've seen in action but may be only on windows-only emulators, not sure and the compression idea from the second paragraph, you'd probably get something a bit more beneficial to PSX image compression than throwing it in a squashfs.

Almost every part of this post is incorrect.

-It's not on-topic. This thread is about filesystems, not files.

-Trimming has nothing to do with lossless compression. It's removal of padding. These ISOs are the size of the discs because they were ripped straight from the discs, and that's the capacity of the disc it was ripped from. I don't know if a tool exists for removing padding from playstation ISOs, but real compression algorithms do work for this because the padding tends to be just endless zeros, the most compressable data possible.

-What are you even getting at with your gzip hate? gzip uses the DEFLATE algorithm internally. It's also a stream compressor, meaning you can quickly and easily seek to where you need to be without bottlenecking I/O excessively. Yeah, you can perhaps lose an additional 5% or so off the original file size using other algorithms, but they're a lot slower both to compress and decompress, and you'd lose more performance than it's worth imo. Give me a good reason why gzip is not the best compressor for this task.

-Why do you think compressing an iso individually would be "more beneficial" than putting it on a compressed filesystem? SquashFS uses gzip internally. You can use the LZMA version too if you want more compression and are willing to pay the penalty of slower decompression and MUCH slower initial compression. On top of that, if you're not using a compressed filesystem, your app will need to support compressed ISOs. With a compressed filesystem, the compression is transparent to the app.
 
Last edited by a moderator:
Missing the point....

I didn't mean to imply trimming was lossless compression, just a size gain before actually compressing, and the padding isn't always zeroes or even data with a patten so it'd be detrimental to compression. Also I said to just use deflate since gzip has an extra header which would be a bit wasteful in this case and image file level compression would be beneficial since you'd get the gains while obviously not requiring it to exist on a compressed filesystem.

Also what the [CENSORED] is your problem? Your post sounds like you have a personal vendetta against me or some shit and I don't even know you. Otherwise you're just a [CENSORED].

Edit: hey look, someone did this. If this has source available maybe psx4pandora could adopt it.
http://www.fpsece.net/forum2/viewtopic.php?f=8&t=5
 
Thanks for cleaning up that [CENSORED]heads post (ok, I'm only joking, calm down!)...

No one has a personal vendetta against you paulguy, you've just gone a little off topic. I admitted I was tired when I offered my mild rebuke, and Dracks points expressed my sentiments much clearer than I was able to. Please don't start a war. We all go a little off topic occasionally and sometimes it's useful, but this is primarily a thread about filesystems that can be used on Pandora.

If you have yours already, can I request that you test NTFS compressed filesystem for me (and check if reading/writing works). Perhaps you could also try SquashFS (maybe using the loopback technique) and see what performance is like.

If you're determined to discuss ISO ripping (and I'll happily join in) there's probably more appropriate threads in which to do it, though I'd be more inclined to create a new one. Anyway, thanks for taking an interest, the more people who contribute ideas the better.
 
Eh, it was more a random thing that I didn't even expect would really change the topic so much as add to it because the topic of compressing disc images had came up and I figured a more datatype-specific compression scheme would be more beneficial than filesystem-level compression in that case and would be worth consideration.

Anyway, I don't have my Pandora yet so I can't test it, I'm in 350-450. I suppose when I do, I could set up a NTFS partition on a slow SD card and do some operations like compressed/uncompressed copies of text files back and forth to see how it handles highly compressible data, then maybe try something like mp3s to see if less compressible data makes a difference, and whether compression overall makes any beneficial difference to reads and writes given a slow card. I could also test it on a fast card to see if the Pandora's SD bus speed overall acts as a bottleneck in IO versus processing. Or someone who has a Pandora can do this, now, and not have to wait for me.
 
Well paulguy, right now you seem to be one of the few people taking an interest in using a compressed filesystem anyway, so I really hope you get your Pandora soon and can try out a few of these ideas. In all honesty I'm a bit perplexed that there haven't been more of a response to this. I can only guess that most people who've already got a Pandora are linux experts, and know so much about this kind of thing already that they needn't bother looking into it. For me, NTFS compressed seems like a great idea if it works, and I can't see why it wouldn't. It's a fast, well known standard which I use on all my external storage drives. I can access them from Mint Linux so it seems reasonable to assume it would work on other linux devices too. Because most of the stuff stored on my external drives is media and therefore already compressed it only saves me between 3-7%, but thats 3-7% of a 1GB drive so it adds up. More so if you are using limited space SDHC I guess. Amiga ADF's and c64 D64's would probably be best kept unzipped so they easily read/write, along with other computer related files is where I thiink big savings can be made.
 
Heh, I'm mostly fascinated in all things computers and Linux naturally. That little going off before about compressed ISOs is just an example of me kinda going off on a tangent for a while, like I tend to do. :D

We'll see, though. It shouldn't be too much longer, I have express shipping through UPS so when mine does get shipped, which should be soon, it shouldn't be much longer than that. I'd say a week, maybe 2 at the most, I HOPE!
 
paulguy said:
Missing the point....

I didn't mean to imply trimming was lossless compression, just a size gain before actually compressing, and the padding isn't always zeroes or even data with a patten so it'd be detrimental to compression.
Do you have any examples to back this up? I've never heard of padding data that doesn't compress. Trimming can't hurt, though.
Also I said to just use deflate since gzip has an extra header which would be a bit wasteful in this case and image file level compression would be beneficial since you'd get the gains while obviously not requiring it to exist on a compressed filesystem.
A Gzip file adds a 10-byte header and an 8-byte footer to the deflate-compressed data. ZIP, another deflate-compressed format, uses 30-byte headers, a bit more if there are multiple files in the archive. Are you suggesting that getting rid of these double-digit byte headers is worth sacrificing the ease of interoperability these formats provide?

Also what the [CENSORED] is your problem? Your post sounds like you have a personal vendetta against me or some shit and I don't even know you. Otherwise you're just a [CENSORED].
You're making suggestions (in the wrong thread, no less) as if you know what you're talking about when you clearly don't, and you cry foul when your misinformation is corrected. What reaction did you expect?

The right course of action would be to *research* and learn the facts, or if you don't understand it, *ask questions* about a concept (TitanUranus in this thread is a good example of this). Don't make assertions that are only substantiated by guessing or limited experience and present them as absolutely correct.
 
Last edited by a moderator:
Back
Top