How to sign up for apps.openpandora.org?


well the important thing is that we have somewhere for people to go to get there apps which is easy to use, and hopefuly and with the introduction of PNDstore and Panorama+MilkyTest UI things will get even easier for the End User.
 
Can't sign up? If you think that's the major issue, I already have an account at the App Store and I can't even sign in! :lol: (Yeah, I know it could just be odd terminology at play here)


It's broken beyond belief and needs to go, but first, something needs to be done about the software that can only be found in PND form there. Audacity is one of them. Perhaps a good solution would be to first move the ones that obviously won't cause issues (like ones under the GNU GPL and other free software licenses), and then make an announcement that the App Store is going to be closed after some amount of time (maybe two months after the announcement) and encourage the developers of the remaining software to move their stuff to the repo, maybe also offer to help with or do the move. That would avoid the issues with restrictive licenses.
 
Last edited by a moderator:
Personally I don't want a.o.o to go away. Craig spent a lot of time working on it, and has been hyping up the successor which is supposed to solve all the problems that the repo has been solving, but in his own unique way. He always seemed so excited about his solution but he never wants to discuss it, so I'm actually really curious as to how it is going to pan out.
 
I would agree with WizardStan here, if craig would actually talk about it or would announce it officially or come up with some kind of plans for it.


Edit: To clarify: All my statements about it are about the CURRENT status of the appstore.
 
Last edited by a moderator:
The current version of apps.openpandora.org is so horribly outdated that it hurts more than it helps. Unfortunately there is still quite a bit of software on there which is not in the repo. I have my doubts about Craig making a newer version, though I would certainly like to see (or hear) what he had in mind.


I would still prefer to see devs upload their missing applications to the repo instead, but we can't force them to do so.
 
Gruso, the obvious thing to do (at least to my mind) would be


1) Download everything unique from the apps.o.o to your own machine. Keep the relevant info (i.e. uploader and stuff). Now everything is safe.


2) Start contacting authors.


3) Upload things to the repo when the author replies with an OK.


Apart from that, I agree with mcobit and WizardStan. Everything would be better if apps.o.o were better. Unfortunately, it isn't.
 
I could create a crawler that finds all PNDs on apps.o.o, parses them for info, then lists them all in the repo format. That would at least make them a little more accessible to users. Would that be an okay stopgap solution?
 
problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail, and also if you thinking parse the info from the app pages within the app store ... well even the version details are messed up, but maybe you can do it ;) I have some code for pasrsing the appstore download links if you need anything from that i can find it from my old code.
 
problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail
Don't worry, I got plenty of experience with malformed PXMLs while writing PNDstore ;) Parsing them will never be perfect, but it should be enough to give the right idea to end users.
I have some code for pasrsing the appstore download links if you need anything from that i can find it from my old code.
I think I've figured out how to crawl through the download links, but your code could be useful. Post it up wherever, and I'll look over it. Assuming people think this will actually be useful.
 
Last edited by a moderator:
problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail, and also if you thinking parse the info from the app pages within the app store ... well even the version details are messed up, but maybe you can do it ;) I have some code for pasrsing the appstore download links if you need anything from that i can find it from my old code.
I wish taht more Progamms follow these basic standards, would really help the Pandora Useres and it's a important addition to the usability of the Pandora IMHO.


Is Some kind of "auto-correct-PND-Files-tool" possible? :D
 
problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail, and also if you thinking parse the info from the app pages within the app store ... well even the version details are messed up, but maybe you can do it ;) I have some code for pasrsing the appstore download links if you need anything from that i can find it from my old code.
I wish taht more Progamms follow these basic standards, would really help the Pandora Useres and it's a important addition to the usability of the Pandora IMHO.


Is Some kind of "auto-correct-PND-Files-tool" possible? :D
Large parts of the spec have been changed since the beginning of the PND system, older PNDs dont follow these specs : they were created before the changes ;)
 
OK this is for tempel it maybe useful, its only basic and I create the filename on the fly assuming all links are PNDs (which im sure they are not).


I have got extra stuff to actually get content-disposition as getting the actual file name/type from app-store's headers is almost impossible (I haven't included it here though as it wasn't part of the original parser but came later as part of the "side-loading" addition to the repo).


If you find you need it, I could drag that out too.


simple app store parser class:

Code:
<?PHP

//Basic app store URL parser class by Jake (aka milkshake)


class parse_app_store{


    public $links_array;


    private $category = array('Emulators','Games','Applications','Other');

    private $cat = array('Emulator','Game','Application','Other');

    private $downloadURI = "http://apps.openpandora.org/cgi-bin/dl.pl?";

    private $_error_list;



    function __construct(){

    	$this->reset_error_list();

    	$this->parse_links();

    	$this->links_check();

    }




    //returns array of all pnd filenames on the appstore website

    public function parse_links(){

    	$i=0;

    	foreach($this->category as $value){


        	$url = "http://apps.openpandora.org/cgi-bin/viewarea.pl?".$value;

        	$code = file_get_contents($url);

        	$doc = new DOMDocument();

        	@$doc->loadHTML($code);


        	//search for li elements as they contain the name of the app

        	//stored within the onclick attribute - populate $link array

        	foreach ($doc->getElementsByTagName('li') as $element) {

        		if ($element->hasAttribute('onclick')) {

        			$link[] = $element->getAttribute('onclick');

        		}

        	}

  	  	//for app name store within link array

  	  	foreach($link as $val){


        		$cat = array('Emulator','Game','Application','Other');

        		//regex magic to get just the app name - \/([^.]+)/ -

        		preg_match("/".$this->cat[$i]."\/([^\&]+)/",$val,$match[]);

        	}

        	$i++;

        	//unset variables to be on safe side

        	unset($link,$val,$code,$doc);

    	}


    	//for each match create new array called $links_array with name of app

        //as array-key and URL as array-value.

    	foreach($match as $app){

        	$name = substr($app[1], 0, -6);

        	$this->links_array[$name] = $this->downloadURI.$name.".inf";

    	}


    	//unset variables to be on safe side

    	unset($match,$value);

    }


    //simple, check to make sure we have links in array

    private function links_check(){

    	if(empty($this->links_array) && !is_array($this->links_array)){

        	$this->_error_list[]='no links found';

    	}

    }


    //check whether any errors have occurred

    //returns Boolean

    public function is_error(){


   	if (sizeof($this->_error_list) > 0){

    		return true;

    	}else{

    		return false;

    	}

    }


    //return the current list of errors

    public function get_error_list(){

    	return $this->_error_list;

    }


    //reset the error list

    public function reset_error_list(){

    	$this->_error_list = array();

    }


}


?>




test code



Code:
<?PHP

include('class/parse_store_test.php');


//instantiate our object

$p = new parse_app_store();


//check for errors

$err = $p->is_error();


if($err==true){

    $err_array = $p->get_error_list();


    foreach($err_array as $error){

    	echo $error."<br/>";

    }

    exit();

}


//if no erros get array

$url_array = $p->links_array;


echo count($url_array)." links parsed<p>";


//to test we have our downloads links lets generate a list.

foreach($url_array as $k=>$v){

    echo "<a href='$v'>$k</a><br/>";

}


?>




result


Hopefully you find it useful, sorry for the late post.
 
Last edited by a moderator:
Thanks milkshake, that was really helpful! Your code was much more straightforward than what I originally had in mind.


The parser took a little while to write, but that seems to be working too. The whole program was written fast and dirty, but it looks like it's doing its job. Here it is for posterity's sake; consider it to be WTFPL:

Code:
#!/usr/bin/env python2


import os, urllib, json

from lxml import html, etree



download_uri = 'http://apps.openpandora.org/cgi-bin/dl.pl?%s.pnd'


def download_apps(storage_dir):

    categories = ('Emulators', 'Games', 'Applications', 'Other')

    directory_uri = 'http://apps.openpandora.org/cgi-bin/viewarea.pl?%s'

    destination_path = os.path.join(storage_dir, '%s.pnd')


    for cat in categories:

        page = html.parse(directory_uri % cat)

        for i in page.xpath("//div[@class='itemlist']/ul/li/@onclick"):

            name = i.rsplit('/',1)[1].rsplit('.',1)[0]

            print 'Getting', name

            urllib.urlretrieve(download_uri % name, destination_path % name)



def generate_json(storage_dir, output):

    repo_data = {'repository':{'name':'Pandora Apps', 'version': 3.0},

        'packages':[]}


    pxml_start_key = '<PXML'; pxml_end_key = '</PXML>'


    # Bytes of PND to check at a time, so we don't have to load the whole thing

    # into memory at once.

    window = 4096

    # But maintain some overlap between each window to ensure the '<PXML'

    # doesn't get cut off at the edge of one.

    seek_jump = -window + len(pxml_start_key)*2


    for fname in (i for i in os.listdir(storage_dir) if i.endswith('.pnd')):

        print 'Parsing', fname


        fpath = os.path.join(storage_dir, fname)

        f = open(fpath, 'rb')

        f.seek(-window, os.SEEK_END)

        # Seek backwards to start of PXML.

        pxml_start = -1

        while pxml_start == -1:

            text = f.read(window)

            f.seek(seek_jump-window, os.SEEK_CUR)

            pxml_start = text.rfind(pxml_start_key)

        f.seek(pxml_start-seek_jump, os.SEEK_CUR)


        # Read remainder of file, then remove everything after end of PXML.

        pxml = f.read().split(pxml_end_key, 1)[0] + pxml_end_key

        # Strip out the stupid goddamn namespace stuff that I hate.

        pxml = '<PXML>' + pxml.split('>',1)[1]


        # Parse PXML through lxml.etree.

        parser = etree.XMLParser(recover=True, remove_comments=True)

        root = etree.fromstring(pxml, parser)


        # Extract relevant data.

        package = {}

        package['id'] = root.xpath('/PXML/*/@id')[0]

        package['uri'] = download_uri % fname[:-4]


        package['version'] = {'major':'0', 'minor':'0', 'release':'0',

            'build':'0', 'type':'release'}

        try:

            version = root.xpath('/PXML/*/version')[0]

            package['version']['major'] = version.xpath('@major')[0]

            package['version']['minor'] = version.xpath('@minor')[0]

            package['version']['release'] = version.xpath('@release')[0]

            package['version']['build'] = version.xpath('@build')[0]

            package['version']['type'] = version.xpath('@type')[0]

        except IndexError: pass


        titles = {}

        for title in root.xpath('''/PXML/package/titles/title | /PXML/package/title

            | /PXML/application[1]/titles/title | /PXML/application[1]/title'''):

            lang = title.get('lang')

            if lang is not None and lang not in titles:

                titles[lang] = title.text

        descriptions = {}

        for desc in root.xpath('''/PXML/package/descriptions/description |

            /PXML/package/description | /PXML/application[1]/descriptions/description

            | /PXML/application[1]/description'''):

            lang = desc.get('lang')

            if lang is not None and lang not in descriptions:

                descriptions[lang] = desc.text

        package['localizations'] = {}

        for lang in titles.iterkeys():

            l = {}

            try:

                l['title'] = titles[lang]

                l['description'] = descriptions[lang]

            except IndexError: pass

            package['localizations'][lang] = l


        package['size'] = os.path.getsize(fpath)


        # TODO: Calculate MD5.


        try:

            author = root.xpath('/PXML/*/author')[0]

            package['author'] = {}

            for i in ('name', 'website', 'email'):

                j = author.get(i)

                if j is not None:

                    package['author'][i] = j

        except IndexError: pass


        # TODO: Icons, previewpics from apps.o.o.


        package['categories'] = list(set(root.xpath('/PXML/application/categories//@name')))


        # Add package to list.

        repo_data['packages'].append(package)


    json.dump(repo_data, open(output, 'w'))



if __name__ == '__main__':

    storage = '/media/disk/pnd_storage'

    outfile = '/home/randy/www/extra/repo.json'

    download_apps(storage)

    generate_json(storage, outfile)


I've started it running now, and it's currently in the process of downloading every PND from apps.o.o. I don't intend to run it often, since downloading everything is the only way to be up-to-date (I couldn't come up with an easy way to only download new/updated PNDs). But apps.o.o doesn't seem to have been updated since July, so I don't think it's a big deal.


Once it's finished running and I've had a chance to test it for real, I'll post usage details for everyone.
 
whats the plan for PND's with invalid/missing PXML data and or missing icons?


also do you plan on downloading all the files to do this? as there are litterally gb of files on the server, also there are some downloads that are not PND at all.


and how did you get around grabbing the "ACTUAL" file name? I can show you how I did it if you like?
 
Everyone. It's up! The repository is at randy.heydon.selfip.net/extra/repo.json


If you're using PNDstore, you can add it as a second repository. In it's appdata, open pndstore.cfg. Under "repositories", add the URL given above. It should end up looking like this (note the presence of the comma):



Code:
"repositories": [

    "http://repo.openpandora.org/includes/get_data.php"

    ,"http://randy.heydon.selfip.net/extra/repo.json"

]
Note that, if a package is available from both, PNDstore will always take the one with the highest version number. If they're the same, it'll take from the first listed repo.


If you're using Milkyhelper/MilkyTest, I know extra repos are supported, but I don't know how to set them. Hopefully Cloudef or someone can show how it's done.


This repo isn't perfect (some PNDs didn't parse, a couple did parse but apparently don't have titles, etc.), but the majority seem to be fine. Let me know if you spot any glaring errors.

whats the plan for PND's with invalid/missing PXML data and or missing icons?


also do you plan on downloading all the files to do this? as there are litterally gb of files on the server, also there are some downloads that are not PND at all.


and how did you get around grabbing the "ACTUAL" file name? I can show you how I did it if you like?
Missing data was basically handled by hoping for the best and falling back on defaults if need be. For example, at least one PXML had no version number, so it defaults to 0.0.0.0. And a couple didn't parse titles properly, but I haven't come up with a clever solution for that yet. I also don't have icons or screenshots, though I could eventually just provide the URLs to the ones on the appstore.


And I did indeed download them all, since there's really no other way to get their PXMLs. My server stores and parses them, but I leave the download links pointing to the appstore. And for that reason, I didn't need to get the actual filename. But I have PNDstore read the Content-Disposition header to determine it if it's available. The appstore must use that, right?
 
Large parts of the spec have been changed since the beginning of the PND system, older PNDs dont follow these specs : they were created before the changes ;)
Yes, one more reason to update all the old Files, I would like to see the entire Pandora PND Archive from ED AND Craigs App Store AND the rest of the missing PNDs, found often in the Wiki list in the Repo. :) This really would be an improvement, you can't expect from the average User to seach 4 sources for new Pandora Software, this is Madness.


And I'm not even count the Forums to the list. ^^""
 
Yes, one more reason to update all the old Files, I would like to see the entire Pandora PND Archive from ED AND Craigs App Store AND the rest of the missing PNDs, found often in the Wiki list in the Repo. :) This really would be an improvement, you can't expect from the average User to seach 4 sources for new Pandora Software, this is Madness.
And I'm not even count the Forums to the list. ^^""

Sure, we all know that. Now the question is, who's up for the task ?


Repackaging something is some painfull works yet very low on the rewards side. I for one haven't finished this work (most is done, but not all).
 
But I have PNDstore read the Content-Disposition header to determine it if it's available. The appstore must use that, right?

yes but not directly, after you click install it takes you to another page for 3 seconds before redirecting you to the file, i.e. rockbox redirects you to: http://apps.openpand....pl?rockbox.inf


then at that point only do the headers contain content-disposition.


there does appear to be a direct corralation between what is contained within the onclick on the "Emulators,Aames,Applications,Other" list pages to what the end redirect URL looks like for example



Code:
<li onclick="location.href='../cgi-bin/viewapp.pl?/Application/rockbox.inf';" style="cursor:pointer;">

//rockbox.inf is what is contained within the redirect URL

<META HTTP-EQUIV="Refresh" CONTENT="3;URL=dl.pl?rockbox.inf">

so when I assumer .pnd as part of my class I should have replaced it with .inf then read the content disposition for the correct filename but I suppose that didn't matter for the actual download to your server?


EDIT:


also the only other thing that worrys me about this is all the assumptions you are making as part of the PXML parser are great to allow downloads but it means you may be supplying the end user with a PND that is very badly made.


but apart from that... good work :)
 
Last edited by a moderator:
Back
Top