ER-301 hub BETA - An attempt to organize community presets/packages/units

ER-301
#1

The ER-301 hub - beta version:

I’m moving this project in its own thread, so the discussion can continue if necessary. It started here: Package management, distribution, organization - V0.6

When the feature list is somewhat solid, we can start filling this thing up with content. For now Accents, Lojik and Extras serve as guinea pig projects.

I’m keeping a very informal road map and collaborator’s docs in the GitHub Wiki:

CHANGELOG

2021-06-23 v0.5

  • FEATURE: Add navigation
  • FEATURE: Add firmware overview and download page

2021-04-3 v0.4.02-beta

  • FEATURE: Add direct GitHub release link

2021-03-27 v0.4.01-beta

  • FEATURE: Optional dedicated page per unit mardown docs
  • IMPROVEMENT: Better category search and category color coding to categories
  • IMPROVEMENT: refactor frontmatter: merge categories and units in one array with key:value objects

2021-03-24 v0.3.00-beta

  • Replace yaml content with markdown (yaml is still used in the frontmatter)
  • Declutter project page to make room for markdown docs

2021-03-21 v0.2.00-beta

  • FEATURE: Source user avatars from O|D forum
  • IMPROVEMENT: search in unit names, categories, project titles author names and description

2021-03-19 v0.1.00-beta

  • FIXED: direct links to projects pages did not work, now they do
  • FEATURE: filter projects by clicking categories
  • FEATURE ROLLBACK: sorting by “last updated” field does not make sense without a persistent database: dates are overwritten for all projects with every new build

frontplateloop_m

30 Likes
Package management, distribution, organization - V0.6
#2

It’s a wiki post now!

6 Likes
#3

:clap::clap::clap: looks great. Really love the minimal design.

1 Like
#4

Now we just need someone(s) to create a v0.6 packaged preset to see how that fits into the hub.

1 Like
#5

Yes! I haven’t made any yet, but will get to it shortly. I still have to see how it all works honestly.

I think it I will color code the different files available in a project on the project overview, so you know what to expect on the download page. One color for pkg, zip, unit, something along those lines.

#6

It’s really straightforward. I can see the need for file types other than .pkg diminishing as we head into the future. Why not use the package manager for sharing? It takes a lot of the guess work out of installing. Maybe the need for .chain files and some variation of quick saves might still be needed?

I’ve created and shared 2 packaged presets so far. Rather than being building block type units, they were more like example/tutorials. Does it make sense to reference packages like that on the hub and if so, does it make sense to have a way to easily filter/separate them from packages that are more like building block units?

#7

You’re probably right, I will get some hands-on experience with the packaging system first before jumping into features like color coding file types.

The way the category filtering works in the hub at the moment inspires towards having many single presets in the overview I think, so I would definitely add them. It’s really easy to drill down to the type of unit you are looking for by selecting single or multiple categories and have a direct overview. Hm, maybe it would make sense to just show a number of units next to the project name?

Since the firmware is still not at 1.0 we will probably re-invent the hub at least once or twice :stuck_out_tongue: But that’s ok.

#8

I’m playing around with the package manager on the ER-301 and found out the pkg’s are just zip files with a toc.lua file that holds very interesting data about the units and the package (now I know what @tomf was talking about…) so now I’m thinking: maybe we aim for the future with the hub, and allow pkg files only that will be parsed with no or very little extra work for the contributor, and maybe we can even extend the lazy approach with automatic release updates, if there is a GitHub repo available. That sounds rather sweet, but will be v0.6 only. But since the app is not ready yet, and the work at v0.5 compatible units will be lost anyway, this is probably a good call. What do you think?

1 Like
#9

Yea unfortunately though I don’t think we can just parse lua files, since they can contain arbitrary code. Perhaps packages can include some raw data format, say hub.json (or yml whatevs) that contains parseable information. With a github reference you can pull this data directly in js and start mapping it onto a page without having to download and unzip any files. Then for private contributors they could just provide a static hub.json file.

Edit: Packages already have a somewhat structured directory pattern:

./assets -- Lua files
./*.so -- DSP Library

Could add a hub structure to search for:

./hub/metadata.json -- Hub data

Then you give a github root path to the hub to try and find that file and parse it.

#10

Isn’t the toc.lua always present, in the same place and readable? It seems very straight forward? We can use node-stream-zip - npm I think to only read that file from the pkg file, and then leave the file alone. Am I not seeing something?

#11

Yea unzipping is a possibility as well, it’s just a lot of IO. If we could just get the file directly then there’s no need to stream the entire zip file.

As for reading the toc.lua, check out the core mod toc. The only way to properly read that file is to actually run the lua code inside it. So having a simple data format would make that way easier, but it may be possible to run lua dynamically in a browser (there’s no way it’s safe though ha).

Edit: This line is also particularly problematic, since how do you even determine that.

#12

Ah yes I see what you mean. Darn. Not worth the security risk and I/O overhead.

Do you mean this as a feature request for O|D? I mean you could manually add it to the source repository, but I’m looking at the ER-301 generated package format at the moment since that would be the main way users will exchange presets.

#13

For generated packages, users could write their own json file and PR it to give the hub a static link. This might be better than trying to auto-generate it since the creator can provide better detail as to what their unit is.

So for packaged presets / private repos:

  1. User builds preset on their device.
  2. They export a package file.
  3. They write a json file for their package, describing what it does.
  4. They make a PR to the hub to add the package and json file.

For lua/c++ mods on github:

  1. User writes their code and creates a new release artifact on github.
  2. User creates a hub.json file in their repo.
  3. They PR the hub to add their repo information and a link to their hub.json

So now when the hub page loads, it knows where to find all these hub.json files, be they static or dynamic. It loads them all (or as needed) into working memory and renders the page.

Edit: If users can’t / don’t want to create their own json file describing their stuff, someone else could do it for them as long as it’s simply pasting a description and a few links:

> hub.json
{
  "name": "My-Cool-Package",
  "description": "It does things that sound good",
  "links": [ "my youtube page", "something else" ]
}

Edit2: Come to think of it, might need to be wary of CORS when requesting json from github, certainly need to POC it first

1 Like
#14

So basically, we keep the two file approach that’s in place already (package + metadata) but fine-tune/standardize further.

yes it’s likely CORS will be a problem. I will do some tests.

#15

Yea I think two files is probably the only way, like who’s going to unzip their package, add a file, and rezip it? That would be annoying, and the only way to get it into the package before hand is to do a bunch of typing on the ER-301, which is even worse ha.

Edit: It certainly seems that optimizing for preset creators is the way to go. I may really want my github integration, but there’s many more people making more interesting / musical stuff on their devices.

#16

FYI, I’m open to moving the package meta data from the toc.lua to another more web-friendly format.

4 Likes
#17

Meanwhile the text search function has improved. Now we are searching in categories, project desciptions, titles, authors and unit names. Oh and I stole the O|D forum avatars (by author name). Not what I would call an “integration”, but it looks good :slight_smile:

@odevices I don’t think moving the package meta data to a web friendly format is necessary at this point, because we would have to unzip the files on the fly anyway and that would probably be a serious performance hit. But it’s good to know this is possible.

1 Like
#18

I replaced the yaml files with markdown. The yaml data is still in use as frontmatter, so structurally it’s a minor change, but now it is possible to start writing docs underneath. Also images can be used, linked to the static folders. I added some example markdown in the projects.

2 Likes
#19

The link in “How to collaborate” on Github (https://github.com/bparticle/er301-hub/blob/master/content/projects/accents.yaml) is throwing a 404 error.

#20

Nice time to get started writing some docs :slight_smile:

Still dreaming of hosting them on my own repo, separated even per unit would be magical.