Some Repo statistics for you!


milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,748
Age
39
Location
Rotherham, UK
Hi Guys some people were asking in IRC about some download statistics from the repo so I though I would share them with you...

please bear in mind this may not be 100% accurate because I only store the current app filesize in the database not the historical filesize and all filesize and bandwidth stats are based on that fact but its as as good as it will get.

DATE | TOTALDL | DISTINCTAPPS | AVGFILESIZE | TOTALBANDWIDTH
--------+---------------+---------------+---------------+----------------
Sep '11 | 6519 | 353 | 11.72Mb | 72.45Gb
Oct '11 | 9141 | 374 | 11.70Mb | 100.00Gb
Nov '11 | 7920 | 398 | 13.01Mb | 115.41Gb
Dec '11 | 8100 | 419 | 14.58Mb | 133.54Gb
Jan '12 | 6064 | 440 | 15.20Mb | 93.23Gb
Feb '12 | 7162 | 452 | 13.44Mb | 94.24Gb
Mar '12 | 12813 | 472 | 14.19Mb | 167.29Gb
Apr '12 | 15095 | 510 | 14.12Mb | 200.88Gb
May '12 | 15053 | 573 | 13.07Mb | 224.21Gb
Jun '12 | 14622 | 598 | 13.77Mb | 202.50Gb
Jul '12 | 13086 | 610 | 12.24Mb | 151.22Gb
Aug '12 | 14082 | 634 | 13.25Mb | 176.66Gb
Sep '12 | 13664 | 663 | 14.71Mb | 219.48Gb
Oct '12 | 15333 | 670 | 15.43Mb | 220.04Gb
Nov '12 | 12910 | 703 | 18.13Mb | 375.63Gb
Dec '12 | 16308 | 716 | 16.83Mb | 303.42Gb
Jan '13 | 19953 | 756 | 17.85Mb | 422.19Gb
Feb '13 | 19338 | 808 | 18.07Mb | 376.26Gb
Mar '13 | 27783 | 837 | 17.18Mb | 517.09Gb
Apr '13 | 21095 | 876 | 17.05Mb | 378.23Gb
May '13 | 8699 | 726 | 19.14Mb | 180.12Gb
The repo is growing with new apps/games daily (907 now) and I like to thank all the devs in our community that are making this possible.
Looking at the bandwidth stats looks like our niche handheld is getting out to more people all the time (or our small community's hunger for downloading apps is increasing).

Go OpenPandora!  :)

Here is a chart from _wb_ (thanks):

The blue and yellow lines correspond to the left y-axis.

The orange and green lines correspond to the right y-axis.

repostats.png

Here are is the top 10 grouped by package_id for 2013 (so far) ordered by ttl_downloads:

rank | package id | title | total dl | filesize | total bandwidth
--------+-------------------------------+-----------------------------------+-----------+-----------+------------------
1 | ppsspp_ptitseb | PPSSPP | 2827 | 9.87Mb | 27.25Gb
2 | firefox.hdonk.500 | Firefox 20.0 | 2813 | 34.18Mb | 93.89Gb
3 | package.pcsx_rearmed.notaz | PCSX ReARMed | 2461 | 2.07Mb | 4.98Gb
4 | DraStic | DraStic | 1792 | 0.61Mb | 1.07Gb
5 | minislug_ptitseb | Mini Slug Project | 1266 | 5.22Mb | 6.45Gb
6 | sysinfo.wb | Pandora System Info | 1247 | 1.70Mb | 2.07Gb
7 | microbes.wb | Pandora Microbes | 1134 | 13.30Mb | 14.73Gb
8 | libreoffice_ptitseb | LibreOffice | 986 | 159.75Mb | 153.82Gb
9 | mupen64plus | Mupen64Plus Nintendo 64 Emulator | 917 | 1.84Mb | 1.64Gb
10 | szupdater1.openpandora.org | SuperZaxxon 1.54 offline updater | 917 | 31.92Mb | 28.59Gb  For anyone wanting to graph this or just because they want more information here is a CSV file which includes ALL apps.

data.zip
 

Attachments

  • data.zip
    31.4 KB · Views: 165
Last edited by a moderator:
Nice to see those numbers :)


You can rename your topic yourself btw.
 
Here's the above table in a graph. The blue and yellow lines correspond to the left y-axis, the orange and green lines correspond to the right y-axis.

repostats.png
 
Looking at the massive bandwidth use, I wonder if using a diff to update a pnd instead of downloading the whole pnd would be viable
 
Looking at the massive bandwidth use, I wonder if using a diff to update a pnd instead of downloading the whole pnd would be viable
If the PND is an ISO file, maybe, but a squashfs file not a chance, it's binary compressed and even a single byte in the wrong place can change the entire file. The best that could be done would be for the server to open the archive and send only new and updated files which would then be put in the appdata directory, but that makes things extremely complicated and error prone, among other things. There isn't really an elegant way to solve the problem, it's just easier (and cheaper) to keep throwing more bandwidth at the problem.
 
Looking at the massive bandwidth use, I wonder if using a diff to update a pnd instead of downloading the whole pnd would be viable
Something like this should work:

Server-side:

When a PND gets uploaded that already existed (so it's an update), if the PXML contains something specific to allow incremental updates, then unpack both the current version and the most recent archived version. Create a list of all modified, added or deleted files, and put the modified or added files, together with the list of deleted files and maybe some other meta-data, in a .tar.bz2 (you could also do more detailed diffs but since most files will be binaries, that will probably not be worth the effort). You could include a checksum of the full PND in this meta-data, as well as instructions on how the PND has to be created (compressed or not? what icon? basically the parameters to pnd_make.sh). Make both the full PND and the incremental update .tar.bz2 available for download.

Client-side:

Make a shell script that takes an old PND and an update .tar.bz2, unpacks the PND, copies the files from the .tar.bz2 and removes the files to be removed, repacks the PND according to the instructions in the metadata, and checks the checksum to verify that everything went well.

Add an automated feature in PNDManager to download the incremental update instead of the full PND and call the shell script.

You could also allow authors to upload the diff instead of the full PND, which would be really useful if you want to do stuff like edit the PXML a bit or add some screenshots without having to reupload the entire thing again.
 
rsync the files? But probably some work for the pndstore to ssh in and syncing.
 
Oh, looks like I misinterpreted something: the middle column ("DISTINCTAPPS") is not the number of PNDs in the repo at that time, but it is the number of distinct PNDs that actually got downloaded in that month. Are different versions of the same application considered to be distinct or the same? (I assume the latter)

In that case, this column says something about both the growing number of apps in the store, and the quality of the apps. If out of ~900 apps, over 800 are actually downloaded each month, then there is probably not a lot of crappy stuff on the repo that nobody bothers to download.

Unless these numbers are influenced by people like DREDD who maintain a mirror of the entire repo and perhaps download everything on the repo on a regular basis.

Would it be possible to extract a histogram from the database counting the number of apps in groups defined by their number of downloads in 2013? It would be interesting to see how that distribution is.
 
Would it be possible to extract a histogram from the database counting the number of apps in groups defined by their number of downloads in 2013? It would be interesting to see how that distribution is.
Could you explain further? I can group by app if I need to.
 
For each app, compute the number of downloads in 2013. Then count the number of apps with 0-5 downloads, the number of apps with 5-10 downloads, 11-15 downloads, and so on.
 
Here are is the top 10 grouped by package_id for 2013 (so far) ordered by ttl_downloads:

rank | package id | title | total dl | filesize | total bandwidth
--------+-------------------------------+-----------------------------------+-----------+-----------+------------------
1 | ppsspp_ptitseb | PPSSPP | 2827 | 9.87Mb | 27.25Gb
2 | firefox.hdonk.500 | Firefox 20.0 | 2813 | 34.18Mb | 93.89Gb
3 | package.pcsx_rearmed.notaz | PCSX ReARMed | 2461 | 2.07Mb | 4.98Gb
4 | DraStic | DraStic | 1792 | 0.61Mb | 1.07Gb
5 | minislug_ptitseb | Mini Slug Project | 1266 | 5.22Mb | 6.45Gb
6 | sysinfo.wb | Pandora System Info | 1247 | 1.70Mb | 2.07Gb
7 | microbes.wb | Pandora Microbes | 1134 | 13.30Mb | 14.73Gb
8 | libreoffice_ptitseb | LibreOffice | 986 | 159.75Mb | 153.82Gb
9 | mupen64plus | Mupen64Plus Nintendo 64 Emulator | 917 | 1.84Mb | 1.64Gb
10 | szupdater1.openpandora.org | SuperZaxxon 1.54 offline updater | 917 | 31.92Mb | 28.59Gb  For anyone wanting to graph this or just because they want more information I have attached a CSV to the original post which includes ALL apps.
 
Last edited by a moderator:
Is there a way to count the number of installs? The number of downloads gets inflated because of updates, which is probably part of the reason why PPSSPP gets more downloads than DraStic: PPSSPP had more updates.

It would be interesting to have a column for number of downloads of the most recent version only; that number should be roughly equal to the number of distinct people who have installed the PND.
 
If the PND is an ISO file, maybe, but a squashfs file not a chance, it's binary compressed and even a single byte in the wrong place can change the entire file. The best that could be done would be for the server to open the archive and send only new and updated files which would then be put in the appdata directory, but that makes things extremely complicated and error prone, among other things. There isn't really an elegant way to solve the problem, it's just easier (and cheaper) to keep throwing more bandwidth at the problem.
As I have no real knowledge about compression implementations this is probably a naive view, but would this really be a problem ? I guess by utilizing checksums, you can clearly determine wether the file was downloaded correctly or not. So if it isn't give it another try, and if that fails too, try it (depending on the filesize, to circumvent that getting the diff eats more bandwidth then a full download) several times more. If it is still a faulty download, try getting the complete file.
Something like this should work:
Seems too complicated (for me,) to be of practical use.
 
Back
Top