A possible PND-system replacement

[_wb_]

Let's just use Prolog syntax for the metadata, so we have a Turing complete language to express things and everything is in a logical notation that can be natively parsed provided you are programming in Prolog

(I'm only half kidding here, it would actually not be a bad idea. But I still like simple .desktop files more.)

 

[Neelix]

Xml brings benefits I.e. You can validate xml against a schema xsd file, you can transform xml into various formats (probably even into a .desktop) using xslt.


Both xsd and xslt are also written in xml which is neat.


I can only think of these a positives over plain .desktop files.


Plus xml is very well suited to structure data in a logical format.

It may be neat if you happen to like reading and writing raw XML,  less so otherwise.

A few months ago, I exported a bunch of data to an XML file with a view to importing it into another program. (XML was the default export option and I figured it was a fairly well defined format so importing the data shouldn't be a problem)

Then I discovered it wasn't that easy.  I couldn't find any ready made tools to automatically parse it and in order to transform the data with xslt (which was the destination program's recommended way of importing it) I would have had to first write another XML document describing how to translate the first one, which would have required a considerable amount of work.   I gave it up as more trouble than it was worth, and  re-exported the data as CSV instead.

In any case I'm struggling to see how that's an advantage in this context.   The format the data needs to be translated to for use in this context is .desktop.

Case for XML:  there are cumbersome to setup translation tools

Case for .desktop: it doesn't need to be translated.

That's a clear win for .desktop if you ask me.

While the inbuilt validation scheme may be a point in XMLs favour it's not enough to convince me that XML is better for this use case, especially given the simplicity and readability of the .desktop format.

- Neelix

 

[milkshake]

I take it the repo uses a PXML schema to validate submissions, but I don't actually know where to find it, and wouldn't have anything installed that I know of to test a PXML file against a schema currently.

Yes there is an official schema which the repo does use but it wasn't written for the repo.


There is a while bunch of information on the wiki for the pxml specification.

http://pandorawiki.org/index.php?title=PXML_specification


Validation and schema:

http://pandorawiki.org/index.php?title=PXML_specification#Validation


I'm not trying to say xml is the greatest thing ever but it's mature and already used.


Using a schema to validate xml is not hard just Google "validate xml with xsd <your language here>".


Anyway this is just an idea and experiment not official (that I am aware of), but if it was and xml or json or similars wasn't chosen I wouldn't complain I was just get on with it and deal with any fallout (if any) later.

 

[slaeshjag]

The dbp-system keeps track of which executables are exported by which packages, so it'll clean those up as packages comes and goes.

As it is right now, that would be kind of annoying to implement copied binaries. As part of the start-up routine for dbpd, I nuke all stale wrappers. For that, I depend on a magic comment on line 2 to determine if it's created by dbpd or not. I guess a workaround would be to have it on a tmps and assume dbpd will never crash or be restarted, but that's kinda ugly.

I was expecting more like a flat-file db on tmpfs of the exported wrappers. An alternative would be to have a tmpfs folder that contains all of them and only all of them and just nuke it on daemon start.

--

But I'm okay with you not breaking that meta / binary (=zip/sqfs) barrier

Alternatively, are you looking to eventially 'cache' the mounts for a little while (atleast when using these wrappers)?

These wrappers will also be used by the desktop files. However, I could add an umount-delay. It'll be a bit messy and probably involve another thread though... (Already got 4 of them <.<) But I'll consider to do it later. A reason against would be that you can't umount your SD-card for X seconds after you've closed all your programs running from it.

 

[slaeshjag]

So, I've worked some more on this. The core functionality is now more or less in place. I still need to do some more in-depth testing of them, but so far it seems to be working just fine.

run-dbp is still a bit hackish, but a friend said he wanted to rewrite it his way, so I'm going to leave it be for now and let him do that. It's not a massive part of the system anyway. The needed functionality in it is there, though.

So, a current feature-list:

* Packages are discovered both when media is inserted (or if it was already there,) and are discovered when they are downloaded to a watched directory.

* .desktop-files, icons and executables are exported and cleaned up as packages are added or removed

-> Apparently, .desktop-files in /usr/share/applications have their mime-types automatically added to the DE, so mime-type registration is theoretically already in place. Need to test it myself though, next time I'm in a graphical DE.

* The union is mounted, and writes to cwd is automatically redirected to a userfolder "appdata/<appdata>", the subdirectory is either the package ID, or a different name defined in default.desktop.

* Packages can be executed from both exported executables, and launching the package-file by path. When the latter is done, the desktop entry default.desktop is used to determine what it'll run.

* Argument-passing to executables works just like you'd expect.

* Executables with enviroment-override set will automatically get their output logged to a configurable directory. stdout and stderr will still go out into the terminal unmodified if it's logged.

Dependency-checking is probably the next big thing on the list, but I want to iron out any remaining bugs in the base first. I will probably also work on other minor features in the meantime.

 

[sebt3]

Dependency-checking is probably the next big thing on the list, but I want to iron out any remaining bugs in the base first. I will probably also work on other minor features in the meantime.

Understoud, especially since there are 2 kinds of dependencies to support :- OS packages deps

- other dbp (like the java.pnd, mono.pnd and alike. If it had been built in, it would have been way more used /me hides as he feels guilty here)

 

[slaeshjag]

dbp dependency checking is really easy to do, and I will include that when I add dpkg-deps.

 

[_wb_]

Keep up the good work!

(cheering from the sideline from my holiday in the German Mosel region)

 

[KickAss]

Dude im also in that region right now. Where you at?

 

[slaeshjag]

I suppose it makes the most sense to do the dependency checking at runtime? Since deps can come and go after the package has been registered.

 

[Cralex]

Even though I don't know enough to understand all the technical details, it's exciting to see this project develop! Keep up the good work!

 

[ekianjo]

(cheering from the sideline from my holiday in the German Mosel region)

excellent wine there I mean in the Mosel valley.  

 

[Wally]

I hope you support highres icons... because that's a teething issue with PNDs now

 

[slaeshjag]

Well, I haven't put in any limitation on icon size

 

[skeezix]

Ooh, fun thread but I didn't notice it till now .. gotta read up.  Great to see everyone carrying the flag!

Keep in mind compatibility or migration strategy/requirement (ie: pnd's coexisting, meaning libpnd/etc running same time.. or forcing everyone to migrate.)

BTW.. zip was not used as metadata on purpose (feel free to ask why we did things the way we did.. it was a huge crazy rush of course, but we did try a dozen variations first.. we had some reasons .. at the time (possibly today even), the zip management had a huge memory overhead (and also at the time, zip didn't work at all for mounting, but did later in firmware life; but recall we had only a couple weeks to put together a system, so thats why ; Various mount strategies tended to put the entire thing in RAM at once, which meant including a couple png's in there ate up a lot of memory in a worst case. At the same time, it was considered to use an unpacked zip since the content should be small and the large content wouldn't compress well anyway, and that could permit manual scanning through anyway; but in the end, we've all written out own unzip handlers before and just jumping to the piece in question and unzipping it is probably okay too. Just be ware, that using a mount against a zip chunk is usually a bad idea for performance

Before nailing down anything, try it out on your desktop and check the performance in terms of memory usage, kernel support, feature parity, raw usage performance .. though with todays hardware its probabyl a very different situation than last round!

jeff

 

[skeezix]

Feature list above is so far more or less what pnd is .. good to reevaluate the foundation, but the hard stuff is..

- package dependancies? Is that an inclusion, or an exclusion to worry about? (probably better to include it this round!)

- package exports .. like a package that can export shared libraries, dictionaries, etc; packages that dump stuff in unusual places (outside of their own tree.. like JVM.)

   -> still need to run installer pnd and let it muck with fileystsems/paths, or can mount at boot and go .. and dependancies on RAM if lots of these, or order of operation loadups, etc

jeff

 

[Bosbeetle]

hehe the pndfather speaks  

 

[Neelix]

BTW.. zip was not used as metadata on purpose (feel free to ask why we did things the way we did.. it was a huge crazy rush of course, but we did try a dozen variations first.. we had some reasons .. at the time (possibly today even), the zip management had a huge memory overhead (and also at the time, zip didn't work at all for mounting, but did later in firmware life; but recall we had only a couple weeks to put together a system, so thats why ; Various mount strategies tended to put the entire thing in RAM at once, which meant including a couple png's in there ate up a lot of memory in a worst case. At the same time, it was considered to use an unpacked zip since the content should be small and the large content wouldn't compress well anyway, and that could permit manual scanning through anyway; but in the end, we've all written out own unzip handlers before and just jumping to the piece in question and unzipping it is probably okay too. Just be ware, that using a mount against a zip chunk is usually a bad idea for performance

Some time ago in another thread _wb_ suggested using tar as the metadata container format.  If there are still performance or memory issues with zip, could tar be a viable replacement?

- Neelix

 

[EvilDragon]

Well, tar is not mountable out of the box, afaik.

Maybe that would work: http://avf.sourceforge.net/

 

[_wb_]

Well, tar is not mountable out of the box, afaik.

Maybe that would work: http://avf.sourceforge.net/

We can still use squashfs for the main content, I was just talking about the glue to add the metadata to the content. What we're doing now is just concatenating first the squashfs file, then the PXML file and then the PNG icon. Concatenating stuff is not a very clean way to do it, and it has its limitations. It would be better to glue things together with e.g. tar, so if at some point we need more metadata files (e.g. extra icons for multiple applications in one PND), there's an easy way to do it.