Discussion About Additions To The Pnd Format


dflemstr

It's a ball.
Joined
Jul 31, 2008
Messages
2,514
Location
Stockholm, Sweden
Website
Visit site
For some reason, when I and some other people started talking about how the PND format currently is limited and not-so-limited and such things, people got very upset, started complaining, worrying and arguing over pointless things and imaginary issues with the PND system, polluting all of the discussion threads that were going on. I therefore will take some of my recent posts, merge them into this one, and let the discussion continue here. Fell free to continue flaming in the other threads if you so wish, but please stay out of this one.

The whoel discussion is about how the PND format could be extended to not only package music players or games, but also things such as icon themes or (for the really geeky people among us) web servers or desktop environments. Here's my proposed solution to that problem (taken from another post):

What I think would be nice is that you'd have 1 folder inside of the PND (say, "overlay/") that can contain symlinks to files elsewhere in the PND (and only symlinks, nothing else), so that when the PND gets mounted, the PND daemon takes all of the symlinks in the overlay folder, puts them directly on the NAND itself (the NAND writes are not so limited that we can't afford to write 20-byte files) and resolves the symlinks to point to the location where the PND is mounted.

If we don't want any NAND writes at all, an alternative solution would be to create one single UnionFS over the NAND that is combined with a RAM-disk, and create all of the symlinks on that RAM-disk.

Anyways, to show how this works, let's say that you have a PND like this:
Code:
foo.pnd/
|-- libfoo.so
|-- myfile.png
`-- overlay/
    `-- usr
        `-- lib
            `-- libfoo.so -> /libfoo.so #In the PND, the "/" dir means the root dir of the PND since it's isofs
...and it becomes this when mounted:
Code:
/
|-- mnt/
|   `-- pnd019283/
|       |-- myfile.png
|       `-- libfoo.so
`-- usr
    `-- lib
        `-- libfoo.so -> /mnt/pnd019283/libfoo.so

The PND daemon would of course keep track of which symlinks belong to which PND, and remove them when the PND is unmounted. If the system is shut down while PNDs are still mounted, this shouldn't be a problem, since unresolvable symlinks will look the same to applications as missing files, and if an UnionFS with a RAM disk is used for the symlinks, they will of course disappear when the system is shut down.

Discussion arose about wether the PND should be allowed to "override" or "overwrite" files on the NAND, and to that I said:
I don't think that it necessarily is a good idea to allow the "overriding" of files at all. Dependency injection would be a possible exploit if it was allowed, for example (Imagine rewriting libmagic.so and "injecting" (replacing) the original version so that all files are opened by a specific program, for example, or that someone would rewrite libglib.so, thus compromising the whole userspace).

I believe in a more strict rule, that PNDs aren't allowed to "replace" any file in the system, ever, because of the security issues.

Also, since the ld systems (that ake care of dynamic libraries) in all Linux distributions already have versioning built into them (check your own /usr/lib folder!), that would also allow for something like God Ginrai probably was thinking about; imagine that the file structure on the NAND looks like this (This tree was taken directly from my Arch install, btw):
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.3                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.3                        (symlink)
`-- libxml2.so.2.7.3
...and a PND for some reason requires a newer version of libxml2.so (let's say, 2.7.5). It would then, when mounted, create a symlink like this:
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.3                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.3                        (symlink)
|-- libxml2.so.2.7.5 -> /mnt/pnd7273232/libxml2.so          (symlink) <--
`-- libxml2.so.2.7.3
...and applications inside of the PND would of course dynamically link to /usr/lib/libxml2.so.2.7.5, because that's the version they're using.
If and when the "official" NAND firmware then gets the libxml2 version 2.7.5 included, the above will silently transform into this, because PNDs may not override actual NAND files:
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.5                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.5                        (symlink)
`-- libxml2.so.2.7.5
...and all will be well, plus the PND will benefit from any official optimizations that were done in the official version of libxml2. When version 2.7.5 then becomes out of date, or libxml3 is released, the symlink to the PND version will activate again, and will thus still work, despite the fact that the wrong version of libxml is on the NAND.
Code:
/usr/lib/
|-- libxml3.so -> libxml3.so.3.0.0                          (symlink)
|-- libxml3.so.3 -> libxml3.so.3.0.0                        (symlink)
|-- libxml2.so.2.7.5 -> /mnt/pnd7273232/libxml2.so          (symlink) <--
`-- libxml3.so.3.0.0

I am really starting to believe that this could actually work. I'd like to hear WizardStan's comments on it, though, as he seems to have opinions about it still :p

Let your opinions be heard!
 
dflemstr said:
For some reason, when I and some other people started talking about how the PND format currently is limited and not-so-limited and such things, people got very upset, started complaining, worrying and arguing over pointless things and imaginary issues with the PND system, polluting all of the discussion threads that were going on. I therefore will take some of my recent posts, merge them into this one, and let the discussion continue here. Fell free to continue flaming in the other threads if you so wish, but please stay out of this one.

The whoel discussion is about how the PND format could be extended to not only package music players or games, but also things such as icon themes or (for the really geeky people among us) web servers or desktop environments. Here's my proposed solution to that problem (taken from another post):

What I think would be nice is that you'd have 1 folder inside of the PND (say, "overlay/") that can contain symlinks to files elsewhere in the PND (and only symlinks, nothing else), so that when the PND gets mounted, the PND daemon takes all of the symlinks in the overlay folder, puts them directly on the NAND itself (the NAND writes are not so limited that we can't afford to write 20-byte files) and resolves the symlinks to point to the location where the PND is mounted.

If we don't want any NAND writes at all, an alternative solution would be to create one single UnionFS over the NAND that is combined with a RAM-disk, and create all of the symlinks on that RAM-disk.

Anyways, to show how this works, let's say that you have a PND like this:
Code:
foo.pnd/
|-- libfoo.so
|-- myfile.png
`-- overlay/
    `-- usr
        `-- lib
            `-- libfoo.so -> /libfoo.so #In the PND, the "/" dir means the root dir of the PND since it's isofs
...and it becomes this when mounted:
Code:
/
|-- mnt/
|   `-- pnd019283/
|       |-- myfile.png
|       `-- libfoo.so
`-- usr
    `-- lib
        `-- libfoo.so -> /mnt/pnd019283/libfoo.so

The PND daemon would of course keep track of which symlinks belong to which PND, and remove them when the PND is unmounted. If the system is shut down while PNDs are still mounted, this shouldn't be a problem, since unresolvable symlinks will look the same to applications as missing files, and if an UnionFS with a RAM disk is used for the symlinks, they will of course disappear when the system is shut down.

Discussion arose about wether the PND should be allowed to "override" or "overwrite" files on the NAND, and to that I said:
I don't think that it necessarily is a good idea to allow the "overriding" of files at all. Dependency injection would be a possible exploit if it was allowed, for example (Imagine rewriting libmagic.so and "injecting" (replacing) the original version so that all files are opened by a specific program, for example, or that someone would rewrite libglib.so, thus compromising the whole userspace).

I believe in a more strict rule, that PNDs aren't allowed to "replace" any file in the system, ever, because of the security issues.

Also, since the ld systems (that ake care of dynamic libraries) in all Linux distributions already have versioning built into them (check your own /usr/lib folder!), that would also allow for something like God Ginrai probably was thinking about; imagine that the file structure on the NAND looks like this (This tree was taken directly from my Arch install, btw):
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.3                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.3                        (symlink)
`-- libxml2.so.2.7.3
...and a PND for some reason requires a newer version of libxml2.so (let's say, 2.7.5). It would then, when mounted, create a symlink like this:
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.3                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.3                        (symlink)
|-- libxml2.so.2.7.5 -> /mnt/pnd7273232/libxml2.so          (symlink) <--
`-- libxml2.so.2.7.3
...and applications inside of the PND would of course dynamically link to /usr/lib/libxml2.so.2.7.5, because that's the version they're using.
If and when the "official" NAND firmware then gets the libxml2 version 2.7.5 included, the above will silently transform into this, because PNDs may not override actual NAND files:
Code:
/usr/lib/
|-- libxml2.so -> libxml2.so.2.7.5                          (symlink)
|-- libxml2.so.2 -> libxml2.so.2.7.5                        (symlink)
`-- libxml2.so.2.7.5
...and all will be well, plus the PND will benefit from any official optimizations that were done in the official version of libxml2. When version 2.7.5 then becomes out of date, or libxml3 is released, the symlink to the PND version will activate again, and will thus still work, despite the fact that the wrong version of libxml is on the NAND.
Code:
/usr/lib/
|-- libxml3.so -> libxml3.so.3.0.0                          (symlink)
|-- libxml3.so.3 -> libxml3.so.3.0.0                        (symlink)
|-- libxml2.so.2.7.5 -> /mnt/pnd7273232/libxml2.so          (symlink) <--
`-- libxml3.so.3.0.0

I am really starting to believe that this could actually work. I'd like to hear WizardStan's comments on it, though, as he seems to have opinions about it still :p

Let your opinions be heard!

I would rather not use up part of the RAM for this, because I could see some applications needing all the RAM, and this would also limit multitasking. I think writing the symlinks to NAND should be fine. As long as they don't delete themselves (we don't want the overlay to have to write to the NAND every time that PND is remounted) and can be replaced by the actual file when a system update adds it.

-God Ginrai
 
Last edited by a moderator:
God Ginrai said:
I would rather not use up part of the RAM for this, because I could see some applications needing all the RAM, and this would also limit multitasking. I think writing the symlinks to NAND should be fine. As long as they don't delete themselves (we don't want the overlay to have to write to the NAND every time that PND is remounted) and can be replaced by the actual file when a system update adds it.

And when the user wants to first use SomeAwesomeEmulator-1.2.3.4.pnd and then unmounts it and instead mounts SomeAwesomeEmulator-1.2.3.5.pnd, you wouldn't be able to because the symlinks exist (assuming that two versions of something contain roughly the same files)... Oh well, I guess that the daemon could check for broken symlinks, as well, and allow overwrites in such cases.

BTW, a symlink is about ~20 bytes, so an UnionFS wouldn't really take up much space. But it would technically be easier to just write the symlinks to NAND, so yeah, it's probably better that way, anyways.

I think I'll actually fork pandora-libraries and add this; would anyone be interested to join me? I acually can't run pandora-libraries (aka the daemon) on my main PC for some reason so I would have to add various wrappers and unit tests and so on to debug my additions, so it would be cool if anyone who already has pandora-libraries up and running on his/her system could help testing it.

Oh, and I still don't have skeezix opinion about this so don't expect this to be merged into the "core" without him knowing of it, first.
 
Last edited by a moderator:
dflemstr said:
I think I'll actually fork pandora-libraries and add this; would anyone be interested to join me?
Not yet. I am interested in making it as feature complete as possible (even if I don't agree that some of the features are needed), hence the heavy discussion, but I'd rather hold off on actually making any changes until I can see the thing first hand, with a chance to experiment a little. Keep things academic for now. Also, I don't have Pandora libraries running either, so it would be a pain to test anything.
 
Last edited by a moderator:
WizardStan said:
dflemstr said:
I think I'll actually fork pandora-libraries and add this; would anyone be interested to join me?
Not yet. I am interested in making it as feature complete as possible (even if I don't agree that some of the features are needed), hence the heavy discussion, but I'd rather hold off on actually making any changes until I can see the thing first hand, with a chance to experiment a little. Keep things academic for now. Also, I don't have Pandora libraries running either, so it would be a pain to test anything.
I usually try to implement something in order to discover faults with it, but maybe you're right; there's always more discussiong to do, and I wouldn't want to write a complete solution that then simply gets trashed.

I'm now thinking about: would this system render some parts of the PXML unnecessary, or would there still be uses for them?
I'm thinking about the "categories" section, the "osversion" section and the "associations" section. If you don't know what I'm talking about, clone the git://git.openpandora.org/pandora-libraries.git repo and read the documentation I've written. Basically, with this new model, the PND could provide its own .desktop files in the right locations to make it's executable show up in all standard Free Desktop compatible menus. But maybe it's good to keep those sections around, just to be on the safe side?
 
Last edited by a moderator:
dflemstr said:
WizardStan said:
dflemstr said:
I think I'll actually fork pandora-libraries and add this; would anyone be interested to join me?
Not yet. I am interested in making it as feature complete as possible (even if I don't agree that some of the features are needed), hence the heavy discussion, but I'd rather hold off on actually making any changes until I can see the thing first hand, with a chance to experiment a little. Keep things academic for now. Also, I don't have Pandora libraries running either, so it would be a pain to test anything.
I usually try to implement something in order to discover faults with it, but maybe you're right; there's always more discussiong to do, and I wouldn't want to write a complete solution that then simply gets trashed.

I'm now thinking about: would this system render some parts of the PXML unnecessary, or would there still be uses for them?
I'm thinking about the "categories" section, the "osversion" section and the "associations" section. If you don't know what I'm talking about, clone the git://git.openpandora.org/pandora-libraries.git repo and read the documentation I've written. Basically, with this new model, the PND could provide its own .desktop files in the right locations to make it's executable show up in all standard Free Desktop compatible menus. But maybe it's good to keep those sections around, just to be on the safe side?

I think it would be good to keep those other sections around. That way, people don't have to use the overlay and write to the NAND when it can be done without doing so. ;)

-God Ginrai
 
Last edited by a moderator:
God Ginrai said:
I think it would be good to keep those other sections around. That way, people don't have to use the overlay and write to the NAND when it can be done without doing so. ;)
The writes will happen anyways; it'll just be the pnddaemon that creates the .desktop files instead. I guess that this would be preferred, however, since some devs aren't used to the file system layout of Linux and might not put the .desktop files in the right place.
 
Last edited by a moderator:
I have some questions on this.

When are you going to apply the overlay, as soon as the card is mounted and the pnd daemon actually checks it?
Are you going to look for every pnd on the card and apply the overlay?
If we do this for every pnd or just for one pnd whats the processing time going to be?
What happens if you start hot swapping the sd card?
Since your touching the nand, what if something bad happens, low battery, can we corrupt the nand?
In the above example power is lost, so now you have symlinks on the nand that should have been removed, but would the daemon now think they were part of the nand to begin with, so it wouldnt remove them, nor allow an overlay since the file exists.
Doesnt this assume your running the daemon as root? Is everyone going to want that?

I think touching the nand on a regular basis is bad especially if your going to be playing in sensitive areas. Personally I be more open to the unionfs approach if it could be guaranteed that the nand would remain the same after a failure situation.
Also in the case of the libs, cant the paths LD searches be modified to just include the pnd path? Just update the ENV var and the system should find the libraries on pnd pat, right?
 
First, nothing wrong with discussion.

Second, we already know that we're not adding many features right now, since that would only get in the way. Of course, things can go willy nilly once Pandora ships :) (And folks can fork all they like, too.)

Those things considered, discussion on :)

We'd thought about this sort of thing (though symlinks I don't think came up, but whatever); for my part, I always wanted to keep it safe and simple so my though twas to have a couple of searchpaths (everythign is on searcvhpaths now), say one in nvram and maybe one on the SDs, that when the machien boots up are scanned; anything in those paths is mounted into various places, depending on location or perhaps pxml flags or other conventions. If you want to add a few sahred libs, could do it this way. My thought was to do it in /usr/local/ or /usr/pandora/lib or /opt/pandora/lib or the like .. always out of the normal chain. ie: Never touch /usr/lib (say), except with dpkg firmware updates.

The run environment could be patched easily (pnd_run.sh pretty much, say) to LD_LIBRARY_PATH= .... to the various newly mounted paths, and good to go.

Where apps are found is already done.

(Sure, there are cases you'd want to modify system directories .. but coudl those cases be relegated to dpg's for the hardcore?)

ie: Let us define some actual use cases and try to figure out if tyou can do them on the side, or if they _need_ to modify system directories (and if things can be tweaked to avoid that.)

Before inventing neat ways of doing things, lets come up with reasons to do so ;)

jeff

(I'm away again for a week, so won't reply much .. man, life has been funny the last 9 months or so :(
 
Pickle said:
I have some questions on this.

When are you going to apply the overlay, as soon as the card is mounted and the pnd daemon actually checks it?
Are you going to look for every pnd on the card and apply the overlay?
If we do this for every pnd or just for one pnd whats the processing time going to be?
What happens if you start hot swapping the sd card?
Since your touching the nand, what if something bad happens, low battery, can we corrupt the nand?
In the above example power is lost, so now you have symlinks on the nand that should have been removed, but would the daemon now think they were part of the nand to begin with, so it wouldnt remove them, nor allow an overlay since the file exists.
Doesnt this assume your running the daemon as root? Is everyone going to want that?

I think touching the nand on a regular basis is bad especially if your going to be playing in sensitive areas. Personally I be more open to the unionfs approach if it could be guaranteed that the nand would remain the same after a failure situation.
Also in the case of the libs, cant the paths LD searches be modified to just include the pnd path? Just update the ENV var and the system should find the libraries on pnd pat, right?

I believe that it would probably overlay when the PND is mounted.

The reason I suggested not removing the symlinks when unmounted was because then you would be writing to the NAND a lot more than you might want.

And I have no idea if we can corrupt the NAND.

Isn't the PND daemon already running as root? Where does it put the .desktop files?

Also, the unionfs idea is great, but my problem is that it could stunt multitasking.

-God Ginrai
 
Last edited by a moderator:
I think that touching NAND is bad idea (IMHO). I like keep NAND with firmware that I choice (if any option came up) or with updated original one. All writing may be done on SD cards only.

One way is manual scan. User run "scan card for PNDs or so" and some tool do this. The tool place symlinks to some folder (even create it) on card. The tool remove previously created symlinks for files that no longer exists.

Other way is full automatic scan and creating/deleting symlinks on background. Same process as described above, but started automatically after inserting card.

Symlinks created by the tool or by background process then can be displayed on desktop or in some menu or so.

--

I personally plane to have more than one card and swapping them as I need. Some emulators on one, some productivity tools on other, and some really personal things on another. So I curious how to dflemstr's idea can work. I insert card and some symlinks are created in NAND. I remove 1st card, insert 2nd and now what? Symlinks for unavailable files goes to be deleted, new symlinks will be created? This sound as unreasonable wearing of NAND even by few bytes per cycle.
 
peca said:
I think that touching NAND is bad idea (IMHO). I like keep NAND with firmware that I choice (if any option came up) or with updated original one. All writing may be done on SD cards only.

One way is manual scan. User run "scan card for PNDs or so" and some tool do this. The tool place symlinks to some folder (even create it) on card. The tool remove previously created symlinks for files that no longer exists.

Other way is full automatic scan and creating/deleting symlinks on background. Same process as described above, but started automatically after inserting card.

Symlinks created by the tool or by background process then can be displayed on desktop or in some menu or so.

--

I personally plane to have more than one card and swapping them as I need. Some emulators on one, some productivity tools on other, and some really personal things on another. So I curious how to dflemstr's idea can work. I insert card and some symlinks are created in NAND. I remove 1st card, insert 2nd and now what? Symlinks for unavailable files goes to be deleted, new symlinks will be created? This sound as unreasonable wearing of NAND even by few bytes per cycle.

That's why I suggested the symlinks shouldn't be deleted, to get rid of the excessive wear.

-God Ginrai
 
Last edited by a moderator:
@skeezix,
First of all, it's worth mentioning: I wouldn't be having this discussion if it wasn't for the fact that people have asked (either in threads or in PMs to me):
"How do I install apps ont he Pandora?"
>"You download a PND and drop it in a special folder, done."
"Ok, I've heard that certain codecs won't be included in the Pandora distro, how will you provide those?"
>"Err... through IPKG, I guess. Or codecs might also be included in every single music player deployed ever, individually"
"So, what if I as an artist want to create an icon theme for the Pandora that there isn't enough space for on the NAND?"
>"Err, drop it in the home directory? Oh, right, /home is on NAND..."
"Dude, I want to create levels for Quake3 (à la OpenArena) that won't fit on NAND, how would I distribute those to users?"
>"Err..."
Ergo, this discussion started. I've had many of them before and you are probably used to them by now :p

I've understood how you've tried to approach this "problem" before but IMHO, it has always involved adding more and more stuff (like /usr/pandora/lib) etc and I like "keeping it simple" in situations like this.

Oh, and even if system directories are touched, nothing is ever changed or overwritten in them; PNDs may only add things.

Things that we simply cannot allow to happen:
1. The PND should never ever be able to execute any code without the user giving permission for it.
2. It shouldn't be necessary to add this and that extra directory or this or that extra daemon when the normal stuff will suffice just fine (→ bloat)
3. It shouldn't be necessary for someone making a PND to learn how to write config files or something similar; KIS,S.

@the general UnionFS idea:
I would really prefer to use this idea, but it will be difficult to actually implement. For example, when the user for some reason decides to add something to the NAND (when something is added to the user's home directory, which will reside on NAND, for instance), that something is going to end up in the RAM file system due to how UnionFS works. We'd have to somehow chroot the pnddaemon into, say, /pndroot/, add symlinks to /usr/, /lib/, /bin/, etc in that folder (AVOIDING /mnt/, or else we'll overlay the SD cards too, and /home/, or else we'll overlay home -.-), and then UnionFS (it's a verb now ;) ) that folder with the RAM fs. It would be complicated.

It would really be much easier to just write the symlinks to NAND. It's more writing-to-disk, but otherwise, nothing is different really (both operations are equally safe)

Oh, and to give my answer to some of Pickle's questions (God Ginrai already answered a few too):
1. We don't care about SD cards here; to us, the file system is a file system is a file system, so that's how it is. If we create an UnionFS (which is complicated as heck, as shown above), it'll be constructed when the pnddaemon boots.

2. Only one overlay (aka UnionFS) required, and all PNDs will share it. The symlinks are copied (aka "cp"'d) when the PND is mounted/loaded, which happens when the pnddaemon discovers it (I think?).

3. Every PND; the time required will be about the same time that it takes to execute "find . -type l -exec cp '{}' somedir ;" in a directory with 500 or so symlinks (that should be the correct number for the maybe 50 PNDs an user will have). In my /usr/lib folder (with 2094 symlinks + other files), this took 0.310 seconds (according to "time")

4. Ehm, nothing, I guess? What do you expect to happen? Everything that would happen with the current pnddaemon implementation would also happen with this feature added to it; it doesn't require anything active to be done by the daemon or anyone else (yes, my daemons are people :rolleyes: ).

5. It's maybe 10 à 20 bytes symlinks that we're writing (200 bytes) per PND. That will happen only once per PND, even after unmount/remount. The process is safe against power failure, because no system files or existing files are touched (there are only file creations) so if worst comes to worst, the daemon will simply have to create one extra symlink (overwriting a corrupt one) the next time a PND is mounted.

6. Those symlinks will be broken (aka without target; it's guaranteed due to how the system works) so the PND will simply overwrite them (that should be the default; overwrite broken symlinks). The overwriting will of course only happen if a PND requests the creation of a symlink that would overwrite the old one. I'm sure you won't be missing any broken symlinks, even if they were part of the NAND to begin with, now would you?

7. The daemon runs as root already (And so do most daemons as you know :p)
 
Last edited by a moderator:
I think it's safer to reserve a fixed directory in the root of each SD card for the symlinks (let's call it NAND-Overlay for now). It will partially recreate the linux FS hierarchy as needed and contain all the symlinks for the PNDs on that card.

Inserting an SD card creates a UnionFS between NAND-Overlay on the SD card and / on the NAND. Removing the SD card thus removes all the symlinks as well.

When mounting a PND the symlinks are created in NAND-Overlay pointing to the contents of overlay/ within the PND. If they already exist (and are still valid) nothing needs to be written.

-Caine.
 
God Ginrai said:
peca said:
... I curious how to dflemstr's idea can work. I insert card and some symlinks are created in NAND. I remove 1st card, insert 2nd and now what? Symlinks for unavailable files goes to be deleted, new symlinks will be created? This sound as unreasonable wearing of NAND even by few bytes per cycle.
That's why I suggested the symlinks shouldn't be deleted, to get rid of the excessive wear.
-God Ginrai
But what if I delete some files on my desktop PC (little reorganization) as no longer needed and place some new. Then old symlinks stay in NAND? I guess that creating symlinks on card is better due to their "this card" dependency. I case that symlinks will be created on card we have stable system that need no care when user swap cards.
Caine said:
I think it's safer to reserve a fixed directory in the root of each SD card for the symlinks (let's call it NAND-Overlay for now). It will partially recreate the linux FS hierarchy as needed and contain all the symlinks for the PNDs on that card.
...
-Caine.
+1
 
Last edited by a moderator:
Caine said:
I think it's safer to reserve a fixed directory in the root of each SD card for the symlinks (let's call it NAND-Overlay for now). It will partially recreate the linux FS hierarchy as needed and contain all the symlinks for the PNDs on that card.

Inserting an SD card creates a UnionFS between NAND-Overlay on the SD card and / on the NAND. Removing the SD card thus removes all the symlinks as well.

When mounting a PND the symlinks are created in NAND-Overlay pointing to the contents of overlay/ within the PND. If they already exist (and are still valid) nothing needs to be written.

-Caine.

What would be the constant RAM usage for such a method?

EDIT:
peca said:
God Ginrai said:
peca said:
... I curious how to dflemstr's idea can work. I insert card and some symlinks are created in NAND. I remove 1st card, insert 2nd and now what? Symlinks for unavailable files goes to be deleted, new symlinks will be created? This sound as unreasonable wearing of NAND even by few bytes per cycle.
That's why I suggested the symlinks shouldn't be deleted, to get rid of the excessive wear.
-God Ginrai
But what if I delete some files on my desktop PC (little reorganization) as no longer needed and place some new. Then old symlinks stay in NAND? I guess that creating symlinks on card is better due to their "this card" dependency. I case that symlinks will be created on card we have stable system that need no care when user swap cards.

Broken symlinks do not make a system unstable.

-God Ginrai
 
Last edited by a moderator:
Caine said:
I think it's safer to reserve a fixed directory in the root of each SD card for the symlinks (let's call it NAND-Overlay for now). It will partially recreate the linux FS hierarchy as needed and contain all the symlinks for the PNDs on that card.

Inserting an SD card creates a UnionFS between NAND-Overlay on the SD card and / on the NAND. Removing the SD card thus removes all the symlinks as well.

When mounting a PND the symlinks are created in NAND-Overlay pointing to the contents of overlay/ within the PND. If they already exist (and are still valid) nothing needs to be written.

-Caine.
The location of the overlay is actually moot if you don't care about ~500 symlink creations in RAM on each boot (which, btw, only takes up 10kb of RAM ideally, note God Ginrai :p ). But yeah, that would be a sustainable solution for PNDs on SD-cards. Also remember that SD cards might have FAT32 file systems, and you can't create symlinks on FAT32, so you'd need to create a FS image imstead of a directory for the overlay.

Also, the problem of the user writing to the NAND manually (or by accident - most Linux applications need to write to an user's home dir sometimes) still remains; those writes will end up on the overlay too if we don't find a good way to avoid that problem (heck, if done incorrectly, even IPKG will write to the overlay).
 
Last edited by a moderator:
dflemstr said:
Caine said:
I think it's safer to reserve a fixed directory in the root of each SD card for the symlinks (let's call it NAND-Overlay for now). It will partially recreate the linux FS hierarchy as needed and contain all the symlinks for the PNDs on that card.

Inserting an SD card creates a UnionFS between NAND-Overlay on the SD card and / on the NAND. Removing the SD card thus removes all the symlinks as well.

When mounting a PND the symlinks are created in NAND-Overlay pointing to the contents of overlay/ within the PND. If they already exist (and are still valid) nothing needs to be written.

-Caine.
The location of the overlay is actually moot if you don't care about ~500 symlink creations in RAM on each boot (which, btw, only takes up 10kb of RAM ideally, note God Ginrai :p ). But yeah, that would be a sustainable solution for PNDs on SD-cards. Also remember that SD cards might have FAT32 file systems, and you can't create symlinks on FAT32, so you'd need to create a FS image imstead of a directory for the overlay.

Also, the problem of the user writing to the NAND manually (or by accident - most Linux applications need to write to an user's home dir sometimes) still remains; those writes will end up on the overlay too if we don't find a good way to avoid that problem (heck, if done incorrectly, even IPKG will write to the overlay).

Sounds messy. Oh, and thanks for the RAM usage estimate. ;)

-God Ginrai
 
Last edited by a moderator:
dflemstr said:
Also, the problem of the user writing to the NAND manually (or by accident - most Linux applications need to write to an user's home dir sometimes); those writes will end up on the overlay too if we don't find a good way to avoid that problem (heck, if done incorrectly, even IPKG will write to the overlay).
I suppose that home dir will be mounted from SD card (and tmp and other working directories too) :-O I don't see any reason why write to NAND but firmware upgrade.

Other thought. Standard card use FAT and as dflemstr say, FAT doesn't have symlinks. But can't it be replaced some file stub (as *.lnk file in old Windows98) which can contains path to the original file?

EDIT: Anyway, I plane use some real FS, like ExtFS or NTFS which both can Windows fully access.
 
Last edited by a moderator:
I see no benefit and much to the detriment of such an arrangement. PND and NAND should never cross beyond the mount point, it's off to a good start, hate to see it screwed up with useless feature creep. If you need a special version of something not part of the up to date angstrom image then you should have statically linked it in there already. No PND should ever modify NAND, chisel that in fucking stone.
 
Back
Top