Observations with Alecks

rss

The beginning

For just under a year, Podping has been available and in use — through dedicated podcast hosting companies and also self-hosting individuals — to efficiently notify interested parties of updates to RSS feeds that define podcasts.

It was a bit rocky in the beginning, mostly because understanding the design simplicity offered by a decentralized message bus and defining a software interface to write to it efficiently are two different tasks. But it worked all along and we largely got through it without incident, thanks to the transparency and resiliency of the Hive blockchain.

Unfortunately, since it's on a blockchain, some of the initial experiments proving out the concept of the project at large are still around. One of the most important concepts of a project like this is people can depend on a defined schema, even if it's implemented in a schemaless manner (JSON).

Learning to communicate intent of the Podping project overall was helped by collaborating and putting out a stable release. The 1.0 release of podping-hivewriter, the current primary software we use to manage writing “podpings” to the Hive blockchain, came out several months after the original vision of the project had been laid down in a lowly Podcasting 2.0 developer roundtable.

Intent

While a stable release contributed to our understanding the stability of the system, we had only just begun to realize the potential we had beyond merely a simple podcast update system.

After all, the original scope of Podping was to help reduce unnecessary polling of RSS feeds when they had no change in content. We had already accomplished this by allowing anyone to announce the following data on the blockchain within a given a “podping” event:

{
    "version": "0.3",
    "num_urls": 1,
    "reason": "feed_update",
    "urls": ["https://example.com/super-great-pod.xml"],
}

Simple! This tells a user that an update to the RSS feed that defines a fictitious “super great pod” podcast occurred.

This alone is already more valuable than it would first appear. Not only can one reasonably assume they can stop polling feeds that get submitted via Podping; they also automatically obtain access to an entire history of podcasts. Without compromise.

That would be enough for most people.

Evolution

But a few things had happened since the Podping project had been started.

While the Podcasting 2.0 community had largely agreed that the movement was for more than podcasts, no one really understood how to make that work.

Eventually, I had postulated that the easiest way to do this was to tell the consumer and shortly after wrote the podcast:medium specification which was eventually finalized in The Podcast Namespace.

In parallel, the Podcasting 2.0 community had been throwing around ways to formalize live streams within RSS feeds.

Coincidentally, both had been finalized in Phase 4. It just so happens that the type of the live stream depends on the medium.

Our intent had evolved.

Since the intention of having a medium is to tell the consumer, and it's to be expected that not all consuming applications will care about all types of mediums, we had decided it was important enough to include in the basic event types of Podping.

We had also realized decentralized live feeds weren't of much use to anyone without the ability to instantly notify consumers when a live feed actually starts with an indication of priority beyond normal feed updates.

podping-hivewriter version 1.1

Given the above information, in addition of some new context about how the particularities of how Hive functions internally, we made the decision as a team to improve upon the Podping events by including the above metadata directly in the event names (known as operation IDs in Hive).

These event names are changing from podping to the format of pp_{medium}_{reason}, prefixed with pp_ to denote podping.

Where {medium} can be one of, as of this time of writing, the following:

  • podcast
  • music
  • video
  • film
  • audiobook
  • newsletter
  • blog

And {reason} can be one of, as of this time of writing, the following:

  • update
  • live

Importantly, the podping-hivewriter project will default to the podping medium and update reason to remain compatible with the current scope of users. Official documentation for the above reasons and their meaning will be available on the podping-hivewriter Github project by the time 1.1 stable is released.

One may replace the pp_ prefix with pplt_ for “podping livetest,” which is what we use during development and continuous integration of the podping-hivewriter project. You can use these “livetest” events to test these changes as a consumer before anyone officially adopts them.

Definition

In addition to the event name changes above, we also decided to change the on-chain Podping format to continue to communicate intent.

In short, the new schema will use version “1.0” to help compatibility and is defined as follows:

{
    "version": "1.0",
    "medium": "<ex: podcast>",
    "reason": "<ex: update>",
    "iris": ["list", "of", "iris"],
}

Most noticeably, urls is being changed to iris. This indicates given RSS feeds can be identifiers besides HTTP URLs — perhaps IPFS CIDs or magnet links, for example — and the character set is “internationalized,” supporting any UTF-8 character. Note that this has been assumed by podping-hivewriter since the 1.0 initial release and this is merely a name change.

The addition of the medium and reason slugs to this schema is primarily for portability of data and flexibility of filtering. It is redundant to have it both in the schema and the event name, and that is intentional.

Given the above additions, it's safe to say the following definition of Podping holds true and is identified by the intent of the given data:

Podping is a mechanism of using decentralized communication to relay notification of updates of RSS feeds that use The Podcast Namespace. It does so by supplying minimum relevant metadata to consumers to be able to make efficient and actionable decisions, allowing them to decide what to do with given RSS feeds without parsing them ahead of time.

Looking forward

We have some more ideas to expand upon the Podping update reasons listed above. However, many of these will require new Podcast Namespace features as outlined here by Brian of London.

For example, we want to be able to allow hosts to use Podping as a way to tell consumers when a feed is changing hosts. In order to prevent abuse, we want to be able to tell consumers to expect this type of event to come from a known Hive account set within the RSS feed.

After all, feeds already get polled to oblivion. Anyone announcing a feed update via podping is relatively harmless, even if it's not their feed. A host change, on the other hand, is another story altogether. We are trying to be cognizant of that for new features.

The <podcast:podping> proposal also allows consumers to actually know when a feed is set to update via Podping, as opposed to guessing, helping to remove ambiguity.

Conclusion

In the last year we've turned the Podping project around from an experiment that happens to work well to a full-fledged project with defined scope.

Podping doesn't just send URLs around to applications in hope that they know what to do with them, nor to funnel a user into clicking on something. It provides context as to why they were sent and how relevant the changes are to applications.

Because we don't need new ways to send people URLs. People have been trying that for the last 16 years.

Discuss...

#podcasting20 #rss #podping #hive #blockchain #podcasts #music #films #audiobooks #videos

Diving into controversy

If you know anything about me, you know I'm a very technical person. I am often the first person to point out when someone is merely arguing semantics that have no impact on a technical solution.

Perhaps stirred by the news that Anchor is only creating RSS feeds for its podcasts if users request one, James Cridland of Podnews recently responded about the semantics of “What is a podcast?

An argument about the benefits of an open ecosystem certainly helps.

However, it’s probably not too helpful to tell people who have just spent an hour listening to their favourite podcast on YouTube that, in fact, they’ve not been listening to a podcast. Because they have.

My last post even led with the exact same question. I'd go as far to say and agree that a podcast doesn't require an RSS feed. From a technical perspective, it's all semantics; one could implement the exact same functionality without RSS feeds and still keep it open.

The fact is an RSS feed is a standardized data interchange format. The tradition of using data within RSS feeds to supply audio files and other contextual information within an open ecosystem has worked well for the last nearly two decades, and there's no reason to stop.

But something phenomenal happened.

Falling for fallacy

For better or worse, the cultural phenomenon known as “podcasts” has outgrown its original technological implementation.

Reasons and motivations for this aside, it means podcasts have won. They've become more than an implementation detail. They are a modern representation of a cultural idea which has transcended what any single entity can control; that's the point.

Arguing about whether or not an RSS feed is required for this talk/theater-oriented, on-demand audio experience is merely a distraction. Organizations will either use it or they wont.

It's on us to keep the general art of podcasting open as a platform for free speech, available for all when the closed platforms ultimately fall apart. The proposition that podcasts don't require RSS feeds — while an effort I otherwise disagree with — wholeheartedly validates the Podcasting 2.0 movement by affirming RSS is for more than podcasts.

But how do we add more than podcasts? What if I want to create an application that utilizes RSS for films or music?

Overcomplication

If you're like me, many of you are thinking about nerdy tech-details of hosting or client-side implementation of new features. I get it. These concepts are new and non-trivial.

But we need to slow down. Before we can figure out what kind of data or user experience is needed for something like films, we need a way to know what's classified as a film. It's more than just a video file, and it's often a different user experience from watching YouTube or a television show.

Ask anyone if the film “Pulp Fiction” (pretend it's in an RSS feed) is a video podcast and you're likely to get either very confused looks or laughter, even if the delivery mechanism is fundamentally the same. But the same person knows films aren't limited to distribution on VHS tapes or DVDs.

Similarly, the move away from a definition of a podcast that mandates the use of an RSS feed makes the problem easier for us.

A long-solved problem

Users already know what they want. Even if they can't give you a definition of what it is, they know how to find it. Long before the internet — even electricity — people had stories around fires, cave paintings, town squares, theaters, books, newspapers... Currently these experiences are primarily, or at least most obviously, realized through different levels of applications or social media.

TikTok is the most obvious recent example of this phenomenon. It's not the company that made it popular, either; video shorts have been around for years, perhaps most notably from Snapchat.

It turns out semantics have already solved this problem for us. The distinction between the application and the medium they represent is a subtle, yet, significant point.

Medium

The medium is the message because it is the medium that shapes and controls the scale and form of human association and action. The content or uses of such media are as diverse as they are ineffectual in shaping the form of human association. Indeed, it is only too typical that the “content” of any medium blinds us to the character of the medium.

— Marshall McLuhan (Understanding Media: The Extensions of Man, 1964, p. 9)

Podcasts are a medium in their own right. Why not embrace it?

The construct of podcasts as a medium offers us a new, powerful opportunity: We can, at once, define a podcast beyond its technological implementation and use this concept to logically distribute other mediums within the same system of podcasts while not adversely affecting existing podcast applications.

We can use this idea to offer applications discoverability of mediums, existing or new, and give said applications hints at how to handle the content distributed by these mediums beyond only knowing how to handle an audio or video file.

Films and audiobooks might just be one item per RSS feed, because that's how users expect these mediums to behave. You don't typically search for the studio that creates a film when you want to watch the film — you search for the film itself.

Music, which technically plays fine in any existing podcast player, is often grouped into singles or albums but a user might prefer to follow the artist for new content.

The above user experience scenarios are only possible with knowledge of the medium — something which is already happening within the ecosystem of applications and websites but has yet to be called out explicitly within a content distribution framework.

Inversion of control

This isn't only for films, music, podcasts, or audiobooks. The mere standardization of data in an extensible, open, decentralized ecosystem with a method of practical real time updates opens up a whole new world of applications.

If someone has a new idea for a medium that doesn't exist yet, or they want to take an existing medium that's been locked into proprietary applications, it's as simple as defining it and showing the world. Why not take the concept of “video shorts” and open them up to the floodgates of reasonable competition?

As unlikely a scenario as it is, TikTok, Instagram, or even Spotify could just be hosting companies for video shorts, photographs, or music, respectively.

The inversion of control for applications to implement or define a medium without worrying about economies of scale within a decentralized ecosystem is possibly the most disruptive concept in technology since the creation of the World Wide Web.

For all we know, Adam Curry may well one day be clutching the last podcast of its kind within his cold hands, but a medium-agnostic, adaptable, open ecosystem laid out by Podcasting 2.0 will live on forever.

It's Space-Grade!

Stop worrying

Podcasts have grown beyond their original technological implementations that solved very specific problems into a medium in their own right.

Now is the time to extend the same courtesy of the open ecosystem to other mediums and take our own advice:

Stop worrying about the closed systems. We've already won.

Discuss...

#podcasting20 #rss #podcasts #music #films #audiobooks

What is a podcast?

Whether you listen to them or not, it goes without saying you have some idea of what a podcast is. Maybe you make one yourself. Maybe you use a podcast app to listen to a few regularly. Above all else, you've likely heard something along the lines of “did you hear about that podcast?” It's part of our culture.

Yet, while there is general agreement that podcasts exist, is there agreement on the definition?

Podcast (noun):

An audio programme produced on a regular basis, delivered over the Internet in a compressed digital format and designed for playback on computers or portable digital audio players, such as the iPod. — WiktionaryInterestingly, users removed “RSS feed” from the definition in 2014

a program (as of music or talk) made available in digital format for automatic download over the Internet — Merriam-Webster

a digital audio or video file or recording, usually part of a themed series, that can be downloaded from a website to a media player or computer — Dictionary.com

A series of digital media files distributed over the internet to which a user can subscribe by means of a syndication application. — The American Heritage® Dictionary of the English Language, Fifth Edition

All of these touch on technically relevant descriptions of how a podcast is distributed or the medium in which it's distributed and still completely fail to describe the culturally disruptive phenomenon podcasts have become. When you casually converse about a podcast with someone chances are low that you're talking about the particularities how you listened to it — unless you're deeply involved in the surrounding ecosystem of podcast creation, hosting, or application development.

Even more, none of these definitions are in agreement. Sometimes podcasts are regularly produced, themed, audio, video, subscribed to, automatically downloaded, or... music?

Categories

Like anything else in the world, people like to assign podcasts to categories. The current state of the podcast category system is sort of like Yahoo's web directory, which was from the mid-1990s.

Some applications have come up with a few methods of search, but usually nothing is super reliable without a lot of curation. Otherwise you are either given a specific podcast name to look up, or a creator, having put a keyword on their podcast, might get lucky and have their podcast trending under the keyword they used. Maybe you're looking for something new and use the category to discover it during browsing.

Take music, for example. Listen Notes has a music category for music podcasts. It's great at first glance, but go through a few to find something new and you'll realize there's a problem.

The Is-About problem

Categories given on podcasts only tell you what the podcast is about, not what the podcast is. Browsing through the podcast “music” category, you will find podcasts both about music and podcasts that are music.

Tailored user experiences

When you set out to do an activity, you typically expect that activity to remain consistent until you're done with it. Music listening and podcast listening affect your brain differently, after all.

Mixing the two activities is relegated to specific types of playlists, like Spotify's automatically generated “Your Daily Drive.” When the content is mixed, you expect it.

Spotify is only able to do this by requiring creators to identify their content as music or podcasts up front. Even some existing podcast music experiences depend on specific RSS feeds.

No wonder it's confusing to define what a podcast is! The assumption is — unless you have the ability to specifically force people onto your platform and categorize your content appropriately — any RSS feed serving audio media is just a “podcast.” But, given the Is-About problem, we know that's not true.

If we had a way to identify music separately from podcasts in a decentralized manner, we would automatically provide a way to search for music without getting it mixed up in our podcasts. One might even develop an application specifically to listen to music served from RSS feeds.

Podcasting 2.0

How does this fit into Podcasting 2.0? Dave Jones, from Podcast Index, states the following:

Podcasting 2.0 is a set of forward looking ideas combined with the technology to realize them. It's a vision for what the podcast listener experience can and should be. That experience has stagnated for over a decade, with almost all of the improvements coming in isolated sections of the ecosystem. There hasn't been a single, unified vision from the podcasting community acting together with one voice. So, we've ended up with fragments of innovation across the podcasting landscape with no central driving goal in mind. Podcasting 2.0 is the expression of what that goal could be.

The original movement of “Podcasting” took the concepts of radio-oriented audio programs and disrupted an industry. While RSS has been used successfully for other things like blogs and news, efforts beyond radio-oriented audio have largely stagnated while the internet ecosystem moved forward.

Podcasting 2.0 is the second wave of this disruption. Documentarians can release new documentaries without worrying about censorship for promoting controversial ideas or lost revenue from simply being unglamorous. Video creators can work with hosts to distribute their content without covering it in unwanted, unrelated advertisements. Authors can get their work out to their audience without being stomped on by big tech. Independent musicians can take back control of their content and their income streams.

Your audience wants your content. They do not care how they get to it as long as it's a reasonable experience and will even reward you for your content if they find it valuable.

In Podcasting 2.0, this content is available to any application. Not just the ones the “platforms” allow you to use.

Source of truth

But, Podcasting? Some may be asking, what about [insert other standard or platform here]? I'm a big fan of ActivityPub, for example. Blockchain tech is very promising to solve otherwise hard global problems.

But the key differentiator is RSS XML has remained an open, valuable, and uncomplicated way to to define what content you have available to any application that wants to see it. Any person with a computer can reasonably create an application that reads RSS.

The fact that RSS features have largely stagnated over the past decade or more isn't due to any fault of RSS itself — no one has stepped up nor had the momentum to unify the community. After all, doing things in the open for everyone requires a certain level of agreement on how to make new features available. It just so happens enough of us have decided we're not gonna take this anymore.

RSS remains a source of truth for content — not just podcasts — but we're missing a way to identify non-podcasts.

The Podcast Namespace

If you haven't kept up — or, perhaps, just haven't heard — part of the Podcasting 2.0 movement is to extend RSS to make new features available. Finally, after nearly two decades.

Given the Is-About problem identified above, one thing we need to add to The Podcast Namespace is a way to identify content outside of its category. I've written a basic proposal to get the process started, but we need much more discussion on such a critical idea.

It's prudent for us to identify content for the purpose of a user experience but also for the application. The fact is, some content will use more/different Podcast Namespace features than others. For example, some in our community are looking at how to tell an application that a feed is an audio tour.

An “audio tour” is a type of “podcast” that has location markers for a user to follow along while listening. The only way to identify if something is an audio tour right now is to, well, scan all of the millions of available podcasts to check or simply guess. This works but is unreasonable to any user or developer.

Developers

Particularly, we need developers to get engaged. We already know content creators exist for the content discussed, many of whom are struggling to get by working on their passion projects.

Once we can come to a conclusion about how to identify this content, applications can be created to use it. Podcast applications already exist, but using them to listen to anything outside of a “podcast” is a poor user experience. There's nothing inherently wrong with that, podcasts are the main thing RSS has been used for.

Music and audiobooks have gained a little more traction because they “fit” into existing applications, but mixing them into “podcasts” is often just messy. It's a different experience even if they are all just “audio.”

The concept isn't limited to audio, either. Video has been available as “vodcasts” for a long time, but it tends to be an afterthought in podcast applications. That's partially because there simply aren't a lot of video RSS feeds, and hosting video is more difficult than hosting audio; but there's also no standard way to identify if a feed is primarily for videos without guessing.

Some have already talked about how to create feeds for cooking recipes, too. This is a completely new concept. Imagine having a recipe application that can use content from other people anywhere in the world without you having to browse through a search engine and loading questionable advertisements... all while hoping the recipe is displayed in a reasonable manner. With value4value, you could even reward these chefs for their recipe from the app.

Telling the application what a feed is for is arguably more important than telling the user. The user either already knows what they are looking for or are looking for ways to discover something new. They don't care if your parmesan chicken recipe is distributed in the same way as their favorite comedy podcast.

Conclusion

Whether it's limited to a culturally significant idea of people talking about things sort of like an old-time radio show, or a method of distribution of digital media, we're not here to decide what “podcast” means.

Importantly, with Podcasting 2.0, we finally have a way forward to identify other types of content in a socially-relevant, meaningful way while allowing creators to control and distribute said content to their audience.

Right now, most of this content consists of typical podcasts. We are working together to change that. Let's stop waiting for others to do it for us.

Discuss...

#podcasting20 #rss #music #audio #video #bitcoin

RSS Delivery

No one likes polling.

Ever since the dawn of the open RSS ecosystem, applications poll RSS feeds. That's how it works, right? They check a feed for updates on some kind of schedule. Maybe, if they feel like tackling an unsolvable problem out of sheer curiosity, they add some learning algorithm to figure out when the feed actually changes to save resources.

Polling, most of which is done in regular intervals, is the most reliable method of knowing when a feed is updated within a decentralized ecosystem no one controls. Even today this is how most RSS feed aggregation runs because it's the only way to avoid missing anything. This isn't a problem in its own right, but it's inefficient.

It becomes an issue when:

  1. A feed gets extremely popular, which can easily cause unwanted bandwidth costs or even take down smaller operations.
  2. A single host manages a significant amount of RSS feeds. They add up!
  3. Time-sensitive content is released. Say one wants to publish a breaking news story or a live steam. Polling will catch it late, and increased polling frequencies are frowned upon (often rate-limited) due to the above issues.

Until now, users and services have often relied on other options for updates, particularly for time-sensitive content:

  1. Outside systems like social media have always offered a means to notify users of updates, but it's not a great user experience for anyone. This helps #3, but still requires polling. Often it causes more polling with the user refreshing rapidly.
  2. Centralized systems, usually implemented by podcast services as back-ends for client applications — especially on mobile — offer a way to hide polling from users. It helps #1 & #2, and works, but it's still polling and hiding this from users can cause confusion.
  3. Otherwise entirely closed systems with their own notifications (YouTube, Twitch ... the list goes on). These usually solve the issues for users, but it's no longer open! I'm not going to discuss the philosophical implications of closed systems, but they still get real time updates wrong, because real time updates are hard.

What about WebSub?

Yes, WebSub helps with some of the above issues. It has two main problems:

  1. WebSub attempts to move the problem from polling feeds to maintaining WebSub subscriptions. It's great in theory. In practice it's just another problem to solve and it's ultimately just another unreliable link in the chain. Applications frequently fall back to polling.
  2. WebSub requires a server to obtain updates. This isn't an issue for podcast applications that run their own services, but what about standalone applications as intended by the spirit of open RSS? What about the last mile? Standalone applications still end up polling, and especially get left out of time-sensitive content.

The Last Mile

The last mile is a relatively new problem within the scope of the open RSS ecosystem. Every application used to poll for content. Live streams, outside of obscure internet radio, didn't even exist or weren't widespread enough to be relevant.

Fast forward to today, most podcast applications run their own services and/or depend on large aggregators to do the heavy lifting. Many trusted Apple for this task, until they couldn't, and the Podcast Index only stepped up as an alternative because it's still such a difficult problem. Even more, the aggregators still have to be polled! They don't have the resources to become a notification system en masse.

Podping.cloud

Enter Podping.cloud:

Podping is a blockchain based global notification system for podcasting. Feed urls are written by the publisher to the blockchain within seconds of a new episode being published. Anyone can monitor for those updates and only pull a copy of that feed when it shows up on the chain.

Podping.cloud architecture

What does that mean? It consists of two parts:

  1. An on-boarding mechanism for hosts to migrate to with ease — something similar to WebSub but simpler to manage.
  2. A standardization of data interchange mechanisms — the “podping” namespace — to broadcast podcast updates onto the Hive* blockchain.

*Note: This is really just a specification of a JSON schema. Hive is an implementation detail.

Decentralized podcast updates!

Given Podping, a standardized way to communicate podcast updates into Hive, anyone can listen for them on the Hive blockchain. An important characteristic of what the Hive community has developed is a standardized HTTP API for applications to utilize, as opposed to having to download and manage the Hive blockchain themselves (though they could if they wanted to).

What does this look like for an application? Below are a few real transactions from the Hive blockchain:

{"_id": "cabd5cc591662b4cb131fb546ae9189d104a00ee",
 "block_num": 54014222,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":3,\"reason\":\"feed_update\",\"urls\":[\"https://feeds.buzzsprout.com/981862.rss\",\"https://feeds.buzzsprout.com/1722601.rss\",\"https://feeds.buzzsprout.com/1287197.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:06Z",
 "trx_id": "ebd5b45772e119c38f75e246d4b70d60ee527716",
 "trx_num": 23,
 "type": "custom_json"}
{"_id": "6b7fe74973283bf99d76ef9b81ced193c43ebe84",
 "block_num": 54014229,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":2,\"reason\":\"feed_update\",\"urls\":[\"https://media.rss.com/njb/feed.xml\",\"https://feeds.buzzsprout.com/1749995.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:27Z",
 "trx_id": "394ecb5449f6a7d9472432915ed2241628e0716e",
 "trx_num": 20,
 "type": "custom_json"}
{"_id": "c6ed805969a3c86634f34352e44a232e907336e1",
 "block_num": 54014236,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":1,\"reason\":\"feed_update\",\"urls\":[\"https://feeds.buzzsprout.com/1575751.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:48Z",
 "trx_id": "b3a97ec7bf97604194755913953b28b6de21c403",
 "trx_num": 12,
 "type": "custom_json"}
 {"_id": "73c63d127e040049f0d6c2f118c1c00e46da17da",
 "allow_curation_rewards": True,
 "allow_votes": True,
 "author": "<redacted>",
 "block_num": 54015435,
 "extensions": [{"type": "comment_payout_beneficiaries",
                 "value": {"beneficiaries": [{"account": "hiveonboard",
                                              "weight": 100},
                                             {"account": "tipu",
                                              "weight": 100}]}}],
 "max_accepted_payout": {"amount": "1000000000",
                         "nai": "@@000000013",
                         "precision": 3},
 "percent_hbd": 10000,
 "permlink": "<redacted>",
 "timestamp": "2021-05-19T03:39:54Z",
 "trx_id": "7d4012b4b7e4901bbaf910a71166a78f5ac9b186",
 "trx_num": 22,
 "type": "comment_options"}
{"_id": "502ddccc9a1e4bb86b7379497e6b17e943c869d4",
 "block_num": 54014474,
 "id": "sm_find_match",
 "json": "{\"match_type\":\"Ranked\",\"app\":\"steemmonsters/0.7.89\"}",
 "required_auths": [],
 "required_posting_auths": ["nandito"],
 "timestamp": "2021-05-19T02:51:45Z",
 "trx_id": "b41cf36a88d798d15e31b1d2b718cd7fa4176b1b",
 "trx_num": 49,
 "type": "custom_json"}
{"_id": "1dc0e544c25978fcf868e40623e930d5bbdc97fa",
 "block_num": 54014474,
 "id": "sm_submit_team",
 "json": "{\"trx_id\":\"8a965173a51cf0457264196066febf3040d9c720\",\"team_hash\":\"946c2011bd2ec8844fcdc2bbf946a738\",\"summoner\":\"C1-49-SM0LILHETC\",\"monsters\":[\"C1-64-TPHW58AV34\",\"C1-50-1AMDZHJ7OG\",\"C1-47-S6HDP0EA5S\",\"C1-62-AQOTC401J4\",\"C1-46-1BP1BKEY3K\"],\"secret\":\"TZte3wNaM7\",\"app\":\"steemmonsters/0.7.24\"}",
 "required_auths": [],
 "required_posting_auths": ["postme"],
 "timestamp": "2021-05-19T02:51:45Z",
 "trx_id": "8ff17c754ff1d9621e9e0014439e720086ab5e85",
 "trx_num": 50,
 "type": "custom_json"}

A few interesting things to note here. It's nice to have all of the information, but it's a bit of a fire hose. How can we clean this up? We can hide it within our application, which might be acceptable for some servers... however it's still all unnecessary bandwidth. Particularly the last two non-Podping operations as well as the non-custom_json type.

Walking the last mile

In order for this to be efficient for client applications — especially mobile applications with limited resources — there's a couple things we need to address. Note: I am only familiar with the beem python library at this moment in time, so it's possible these two points are moot.

  1. Server side filtering of custom_json operations. Currently the python implementation, beem does this on the client side, hiding it from the user. It still pulls in every block even if you tell it to only look for the custom_json operation.
  2. Server side filtering of the custom_json id — even if we pulled in only custom_json operations, we're still getting other data from other applications. Really, we only care about id='podping'.

Given those two changes, we can size down the fire hose to the following podping operations:

{"_id": "cabd5cc591662b4cb131fb546ae9189d104a00ee",
 "block_num": 54014222,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":3,\"reason\":\"feed_update\",\"urls\":[\"https://feeds.buzzsprout.com/981862.rss\",\"https://feeds.buzzsprout.com/1722601.rss\",\"https://feeds.buzzsprout.com/1287197.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:06Z",
 "trx_id": "ebd5b45772e119c38f75e246d4b70d60ee527716",
 "trx_num": 23,
 "type": "custom_json"}
{"_id": "6b7fe74973283bf99d76ef9b81ced193c43ebe84",
 "block_num": 54014229,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":2,\"reason\":\"feed_update\",\"urls\":[\"https://media.rss.com/njb/feed.xml\",\"https://feeds.buzzsprout.com/1749995.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:27Z",
 "trx_id": "394ecb5449f6a7d9472432915ed2241628e0716e",
 "trx_num": 20,
 "type": "custom_json"}
{"_id": "c6ed805969a3c86634f34352e44a232e907336e1",
 "block_num": 54014236,
 "id": "podping",
 "json": "{\"version\":\"0.2\",\"num_urls\":1,\"reason\":\"feed_update\",\"urls\":[\"https://feeds.buzzsprout.com/1575751.rss\"]}",
 "required_auths": [],
 "required_posting_auths": ["hivehydra"],
 "timestamp": "2021-05-19T02:39:48Z",
 "trx_id": "b3a97ec7bf97604194755913953b28b6de21c403",
 "trx_num": 12,
 "type": "custom_json"}

This on its own is a huge improvement, but we can do more...

One step further

See the json attribute? That's our Podping schema! What if we only subscribe to one or two podcasts? We don't need the rest of the results.

Thankfully, there's a straightforward way to handle this: JMESPath

JMESPath is a query language for JSON.

Let's say we only want to get results from the podcast “Quality during Design” in the above list. We can test this in a quick way with the jp cli tool and the JMESPath contains function.

echo '{"version":"0.2","num_urls":3,"reason":"feed_update","urls":["https://feeds.buzzsprout.com/981862.rss","https://feeds.buzzsprout.com/1722601.rss","https://feeds.buzzsprout.com/1287197.rss"]}' \
| jp "contains(urls, 'https://feeds.buzzsprout.com/1722601.rss')"
true

Behold! We just defined a mechanism to query Podping json from a custom_json operation. In this example, the contains function being run by jp is returning true, telling us that this example indeed includes the feed URL we care about.

Why is this important? Couldn't we just parse the json object in our code and check for the URL?

Well, yes... but JMESPath provides us a way to run this query on the API server, without the API server requiring knowledge of our dataset. Meaning, given an implementation of this query on the Hive API, we would happily only get the results we need from the API.

If we only subscribed to one podcast we could get update notifications for it without having to parse through all new Hive blockchain events.

This is truly potential for a whole new generation of a decentralized notification system, and Podcasting 2.0 is driving it.

Hopefully some of the above ideas can be incorporated into the Hive ecosystem in a way that will benefit everyone.

Discuss...

#podcasting20 #podping #jmespath #rss #hive