sapient [they/them]
Autistic queer trans²humanist and anarchist. Big fan of dense cities, code, automation, neurodiversity, and self-organising resilient networks.
Pronouns: they/them, xe/xem, ze/zem
Favourite Programming Language: Rust
Alt-Account Of: @[email protected]
- 3 Posts
- 58 Comments
English- •
- 2Y
- •
In particular, I’ve figured out a way to specify sentiment/interests efficiently and combine it reliably over federation, and the data structures required to do that.
I’ve also provided some ideas for sensible defaults (automatic selection of instances, and accounting for instance load), with incremental enhancements to specificity for more advanced users ^.^, as well as a general search mechanism that can be derived from this - though for efficiency, it might be worth trying to develop some sort of probabalistic reverse index to avoid a linear scan, if we’re talking about discovering entities like users or groups where there may be very large numbers.
I hope that if people are interested they will boost the post onto Mastodon, which afaik is where the devs and ActivityPub standards people are, and try and get the ball rolling, because my focus is elsewhere right now, and the social aspects of developing things like this are much more difficult for me than the algorithmic and architectural parts ;3
I have made a post with some work I did to make getting into the federation without requiring users to understand it much easier here, by providing a sensible automatic way to select instances randomly but weighted :)
My idea is meant to allow for a spectrum from simply “pick an instance for me” using the weightings for an assumed “the user is interested primarily in general discussion”, to “search for an instance for me related to xyz topics as a search query”, to fine-tuned discovery ^.^
The weighting is always necessary to use because it allows instances to have more control over who they accept and avoid overloading smaller instances. But you can make the default UI very simple.
This is actually kind of a general activitypub thing. I might do that but that feels like I’d have to make it was more refined and go through some formal process, and I hate doing that kind of thing and find it quite difficult, especially since my attention is elsewhere now.
I kinda just want to put it out here so people with more attention and time and knowledge can push it forward or e.g. boost it onto mastodon. Though if it really goes nowhere I might do something? Idk ;p
Step 4 - Term Merging
Each instance has provided subject trees of what it’s community is meant to be like. Moreover, it has provided the terms it believes to refer to various concepts within their subject tree.
This step is where all those terms get merged together to then be used later via some kind of search algorithm, for the more sophisticated cases.
The steps are as follows.
- Collect all the subject trees from each instance into some way of iterating over them.
- Construct a BTree-based map of topic paths plus associated term information, merging in new values for every level from every federated server ^.^. Much more sophisticated versions of doing this efficiently are documented in the Common Interest Algorithm snippet, even if not for the terms, so just look at that :)
Step 5 - Common Interest Weighting
Apply Common Interest Weighting via the Common Interest Algorithm between the user and each possible instance.
There may be a way to use Heaps or some hierarchical datastructure to sort the instances to do this more efficiently, but as long as the implementation of the Common Interest Algorithm uses BTrees and pre-calculates lexicographically ordered maps of data it can be ensured that the cost of this kind of commonality assessment only grows with the size of the tree specified by the user and the single instance to be compared, rather than all instances (for an individual instance/user comparison ^.^).
There may also be ways to compare the user against all instances at once more efficiently that I don’t know of. But the point is, we can use the Common Interest Algorithm to assign weights for each instance/group/etc. relative to each user.
We could also use some way to convert a user search query into their Common Interest Algorithm tree weights, using the list of known terms. This is for slightly more advanced terms or people perhaps searching for communities or other groups too.
Step 6 - Elimination of Anti-Aligned Instances
Any instances/groups/communities/etc. with alignment <0 should be immediately eliminated from the list of suggested instances/groups/communities/etc. to the user.
Step 7 - Combining Sentiment Alignment Weights & Other Ranking, plus Final Selection
We already have some ranking information based on how willing and able an instance is for new users, plus we have information on how aligned each instance is with this hypothetical new user - now all a fraction from 0 to 1, as we cut out instances that have a negative alignment with the user ^.^. Then I suggest we find some simple way to join those two values together. For now, I suggest simply multiplying the alignment fraction with the weights for each instance, and then use probabalistic selection to direct the user to an instance that aligns with what they want ^.^
It may also be desirable for instances to prioritise somewhat older instances with better uptime, or more trustability (e.g. using some kind of heuristic to detect bot instances or similar), and modify the weightings based on that, or eliminate some instances ^.^
For non-instance searching or discovery, we can use the alignment ranking directly as a form of search ranking :)
Step 8 - Redirection
Redirect the user to the “final” signup page as listed in the instance metadata, along with the parameter for their desired username. Perhaps it would be worth using webfinger to make sure the username isn’t taken on any selected instance, and automatically selecting different instances from the list until you find one without the username taken already, with a warning.
If we’re talking about discoverability of communities or similar, you just put those in order of their direct sentiment alignment rank ^.^
Common Interest Algorithm
The weighting system indicates how much interest (or avoidance) an instance has for a topic as specified by the subject tree. The value of weight for each subject tree should be a value from -1 -> 1 (inclusive), and applies to the deep-most component of the tree. We’ll call this the sentiment of the instance towards that specific level of the tree.
The common interest algorithm specifies a rough way to estimate how “aligned” in sentiment a given pair of entities are using an incomplete collection of nested topic paths ^.^ and then using heuristics to fill in the “gaps” needed for direct comparison. It takes the partially specified trees - along with estimated polarisabilities - from federated instances, combines them together, then uses that to “complete” the sentiment weights specified by users and instances so they can be directly compared to determine the common interests of each to contribute to directing users to instances correct for them.
The default option should be that users are assumed to want “general sentiment/general topic/root topic” instances (i.e. with path /), and then they can specify much more refined interests using various methods, like taking search terms and using the collected known topics for them in various languages to construct a user-friendly search function based off the common interest algorithm heuristic, or allowing direct specification of interests, for more advanced users ^.^.
The full (but slightly incomplete) details of my approximate proposed Common Interest Algorithm are in this gitlab snippet, written in poorly-organised Rust code.
Tagging the Willingness for New Users
Different instances have a different level of desire (and gatekeeping) for new users.
Some don’t allow any new users at all. Others require filling out a form and waiting for approval. Many require an email or captcha, and some don’t require anything whatsoever.
Some don’t want any new users, some do accept new users but only can handle a small number, and others are free-for-all open registration.
Many users will want the ability to create communities without needing to seek approval. For defaults on the “maximum” level of “inconvenience” an instance presenting other instances should show to the user, it makes sense for an instance to use it’s own level of “inconvenience”.
nodeinfo2 (also see here for all keys) already exists to provide some basic information, but it’s not enough for this feature ;p
As such, I suggest we instead construct a property on the main server actor, for now called instance_onboarding_meta. This is an object of the form:
{
"accepting_new_users": bool, // if this is false, no other references need be present
"capacity_used": float (>= 0), // Must be present, represents one-minus the remaining amount of users it can take as a fraction of total estimated capacity. Alternatively, represents an approximate fraction of resource usage. If it's >1, this implies the server is over-capacity.
"preferred_max_users": integer (>= 0), // If present, represents the approximate maximum number of users this instance wants to host. If unset, assume unlimited but perform estimates based on the fraction.
"signup_requirements": {
"captcha",
"email",
"approval",
}, // Must be present, a list of the signup requirements. May need more options as new authentication and validation mechanisms are added to the various Fedi servers ^.^
"signup_uri": "https://example.com/signup/finalized" // "final" signup page, rather than one providing alternate instance suggestions. Should take e.g. a `?username=<new username>` parameter.
}
Instance Signup Redirection Algorithm
Now that a system has been proposed for giving instances to describe how much effort it takes to sign up, how much they can really take new users, and what kind of community they’re interested in, we can use this data to construct a method to split signup across the fediverse.
We’ll describe things in terms of what happens either as the list of instance values is changed while they are polled, or finally what happens when a user actually looks for an instance ^.^. Though, a lot of the ideas are also mentioned in the Common Interest Algorithm Snippet, which also at least partially discusses some other things.
Step 1 - Candidate Instance Collation
The first step is to collate information about potential candidate instances, by making requests to the endpoints described above to instances the current instance is federated with - including itself! (it might be useful to combine all the metadata into one endpoint as well, but that’s all bikeshedding):
instance_software- the software of each instanceinstance_focus- the list of weightedsubject-trees that indicate what the community is oriented around - see the algorithm snippet for efficiently merging in information from instances without having to recalculate the full weights every time, via use of BTrees/BTreeMap.instance_onboarding_meta- Information about how the instance accepts new users, and it’s resources to do so.
Instances shouldn’t poll this very frequently - certainly not on every attempted user signup! - and instead should cache it and poll periodically (say, every hour or so ^.^). This avoids slamming large portions of the network.
Step 2 - Software Filtering
The next step is filtering out candidate instances running different fediverse software than ourselves.
Step 3 - User Acceptance Filtering & Weighting
Our instance should then filter out instances that aren’t accepting users, and perform the following steps to assign weights to instances (may be configurable if the user is ok with accepting more effort than our instance requires - as most users are likely to use the default settings it should be cached too):
-
For each instance, if it requires more things to sign up (email when we don’t need it, etc.), then remove it from the list.
For captcha, mark that instance with a “0.5” weight multiplier rather than eliminating it, if we don’t also require captcha.
From a user-configurability perspective, each possible requirement to signing up can either:
- Eliminate from the list (a user doesn’t want to deal with forms) - this is the default for things required by another instance that aren’t required by ours, except captcha
- Reduce it’s chance of selection (as in captcha) - this is the default for instances if the respective instance has captcha but the current instance doesn’t.
- Have no effect - this is the default if we also have a requirement.
-
For each instance, if it has a preferred max user count, then calculate the current approximate user count by multiplying it by the resource usage capacity.
Then, calculate the approximate available user slots by subtracting the approximate user count from the preferred maximum. Note that this value may be negative in the case of an overloaded server.
-
Find the instance with the largest preferred max user count (if none exists, then use the current server’s user count instead, though remember that if your server does have such a preferred max count, it should be in the list). If any server has an estimated total user slots consumed greater than the maximum preferred user count, use this instead.
Then, assume that the preferred maximum for servers with no specified maximum is approximately 2x that value. Calculate the approximate available user slots of instances without an existing preferred maximum, using this estimate in combination with the resource consumption fractions.
-
For any instance with available user slots <0 - that is, overloaded servers - divide those (negative) available user slots by some value such as 4.
If any instance has a negative number of available user slots, add the most-negative number back on to every instance’s count of available user slots, so that the smallest value is zero.
The division by 4 (or some other number) means that all overloaded servers are avoided more than they would be if we just added the most-negative value back directly.
-
Assign weights to each instance depending on their proportion of available user slots compared to the total. If the instance has already been tagged by a weight (from e.g. having captcha), then multiply by that weight.
You do not have to leave everything. I think lots of instances will be defederating :)
If you don’t want to be manipulated by the algorithms that the Threads instances use to surface content, then don’t subscribe to people on Threads, it’s really not that complicated. If Meta leaves later and you find yourself desperately missing content, then guess what? That’s not Meta killing the fediverse that’s Meta having kept the fediverse alive for a while.
You do realise they can work on social groups right? It’s not just individuals, but they can propagate and push for trends even outside their direct users and communities. And if you think you’re immune to social manipulation, you’re not (insert Garfield YOU ARE NOT IMMUNE TO PROPAGANDA image here ;p)
Even if it’s harder to use, Facebook has the ability and the means to run campaigns to promote their own stuff even if it’s worse.
Federating doesn’t change that.
It makes it much much harder to do it on our network, which is the risk ^.^
Furthermore, it’s not just about that, it’s also about the fact that federating with them entwines us with their communities, and given their size it will not take long for our organisation and communities to be entirely stuck to theirs.
Oh no, we’ve recreated Reddit with millions of users and a thriving community, what a nightmare!
The nightmare is that then they can kick us around at will, buy us off, or destroy us easily. Also, their users likely would not be very aware they’re part of a federation. Did you not read the article I posted and the general concept of EEE and EEC? >.<. Furthermore, it means that their algorithms and mechanisms for pushing things become dominant and we become yet another userbase to be assimilated and farmed for manipulability by the Facebook Monolith.
Seems pretty alive to me, actually.
Then go check whatever instance you’re on three times throughout the day and do the same on Reddit and notice the distinct lack of change and movement on Lemmy/Kbin.
Lemmy/Kbin is a little less active than Reddit. This doesn’t mean it’s dead, far from it! Have you looked in new or hot?, or made sure you’re looking at the All or Subscribed feed?
Lemmy is pretty damn active.
Quesion: I don’t know if the tech limits this, but if an instance owner flips to the dark side- could past posts and content be opened up for Meta mandated data scraping? Or would any code change like that not be retroactive? Aka if we select an instance that turns bad could we be feeding the machine in the future without knowing it today?
Most of our posts are public already - that is part of how ActivityPub works. But internal data, any logs that might be kept, and a centralized repository of this data would be more accessible to Meta/FB if an instance were to be bought out.
However, this is true of essentially any entity capable of being bought, you can’t really avoid it without going full p2p and even then…
Your points boil down to “Threads will be easier to use and more attractive so people will use that”, congrats, that’s the case regardless of whether or not you federate. That’s not a result of federation, that’s a result of meta having a lot of money to make good apps.
They boil down to much more than that. Even if it’s harder to use, Facebook has the ability and the means to run campaigns to promote their own stuff even if it’s worse. Furthermore, it’s not just about that, it’s also about the fact that federating with them entwines us with their communities, and given their size it will not take long for our organisation and communities to be entirely stuck to theirs.
This entire argument hinges on the idea that the Fediverse is filled with great content that Meta will just steal and present to their users when quite frankly that’s just untrue. The fediverse is still a pale imitation of Reddit that is severely lacking in content and is still likely to die from never entering the virtuous cycle required to get a social network off the ground.
Seems pretty alive to me, actually. And the risk is not just Facebook/Meta taking our content, but more us being sucked in by theirs and having their algorithms and strategies used to manipulate us and make us too dependent on their own infrastructure to sustain our own communities again, especially if they cut us off after ^.^ (the threat of which can then be used as leverage or to outright subsume large instances).
I did, if you read it. Past and continued malicious behaviour + open manipulativity means that what they say cannot be trusted.
Trust is not required in this equation. The fediverse exists as a technical system and we can see how it operates. Within the context of those bounds I see no path for meta to break it and no one has been able to explain one beyond vague generalities like “they can’t be trusted”.
I gave examples in part 2 in my post of various routes to destroy activitypub, or nore importantly, destroy or consume the existing network of people.
Even if there aren’t formal Communities (as in, like Lemmy), there are still communities of people.
Yes, but the point I was responding to was saying that communities like [email protected] would lose all its users when they went to [email protected] when that’s simply not even possible
It is absolutely possible if fediverse content is presented as-if it’s just from threads, and then the majority of posters in communities become threads users, and then they either subsume or defederate.
You can post to lemmy communities from in mastodon via @-ing, which Threads could easily add as another feature later. And referring to more general communities the same principle applies.
Presumably people using Threads want that. Or they’ll tolerate it.
They will do it to us, not just Threads users.
Fediverse instances aren’t just providers, they’re communities.
Just like email list serves. Should a listserv block gmail subscriptions? I would again argue not.
Its more like email lists blocking people from other email lists. If there is a massive email list that has continually and specifically coordinated to destroy or consume other email lists and spent massive resources learning specifically how to do this via social manipulation, yes, I would think blocking people from that email list is a very good idea ^.^
It’s tempting to believe the email issue really is some conspiracy to keep the little guy down, but it really is just that a new domain, with low volume, is a strong signal for abuse
Perhaps if it wasn’t already corporate agglomerated, this wouldn’t be so true. But fediverse isn’t email, we have easier indicators for abuse because most content is public and we can guesstimate how much of an instance is “real” users ^.^
Atleast having the option to communicate with threads is enough for the people on mastodon to stay on mastodon and have the choice to do so. Blocking them off will just cause mastodon users to have to make a seperate account just to merely communicate with their friends on Threads. Defeating the purpose of the fediverse entirely and going back to square one.
It doesn’t defeat the purpose to prevent a known-hostile actor from interacting with everyone on Fedi.
It’s not just your friends, it’s Facebook, with algorithms specifically designed to manipulate you and the communities you are part of - including your friends - and by engaging with them you end up locked back into FB’s reach and make it easier for them to EEE or EEC us or do even worse >.<
If you want to talk to your friends, use another app (including threads if you really feel that much need to interact with people who are only on there), or post links promoting Fedi on other platforms, to your friends.
Defederation exists to protect our network from groups like Meta/FB. It doesn’t defeat the purpose of federation to choose who to federate with.
That’s because they’re essentially defederating entities they don’t trust; exactly what’s being proposed here. The solution to defederation is not pre-emptive defederation.
They’re defederating smaller entities because the network got consumed by corpos. And abuse, but lots of that comes from big services and they don’t defed those.
Fediverse instances aren’t just providers, they’re communities.
That’s a real problem for the Fediverse too, because there’s almost nothing that stops someone from spinning up infinite numbers of instances and spamming other instances.
This is in essence what FB/Meta is doing, all the time, except it’s not individual spam it’s an algorithmically backed manipulation mechanism using it’s users as tools ^.^
This is complicated and brand new to all of us but if people realize they’re a part of something bigger than them. They would want to be part of that too!
That’s not how this works. The overwhelming majority of Threads users just saw whatever thing FB put on the instagram accounts and clicked it. They have probably never heard of the fediverse and even if they like the idea they’ll just go “oh, I’m already on it, no need to bother”.
We can get exposure without letting a company specialised in manipulation and astroturfing straight through the door.
Did you read my post? Meta/FB is a well known threat. We already know they are continuously engaging in information warfare towards their own ends and federating with threads just saps our momentum and redirects it towards them >.<
Defederating doesn’t stop people who want “exposure” from creating an account on Threads or even starting a masto instance. I highly doubt FB will make it obvious to Threads users that Mastodon even exists, which you would know if you read my comments on how their app acts as a silo.
ISPs are at a different level of the stack and already have an oligopoly.
We can see the end state of email from when they let big corps take over - its very difficult to selfhost without permission from them lest you get marked as spam.
We have an opportunity to prevent that before it happens here, too. ^.}^
Point being, Threads doesn’t need any other communities. People using Threads are those people who have never used reddit, and never would have signed up for lemmy. These people are also the same ones who don’t care about if their content is coming from a federated source, or just Threads.
And hence, defederation is a good idea.
Defederation is to protect us from them. You are absolutely right that they aren’t comparable to beehaw in size - now imagine if people here start joining the communities on Threads (not formal ones cus threads doesn’t have those), and we later decide to defederate, as some have proposed? Beehaw alone caused a massive clusterfuck, now imagine an instance with 10000x more users and power and concentrated community being let in?
I’m not here to make excuses for Meta, but not a single one of these “sky is falling” posts actually articulate any real danger with federation. They just list all the bad things that have come out of Facebook to imply that surely something bad must happen here too then.
I did, if you read it. Past and continued malicious behaviour + open manipulativity means that what they say cannot be trusted.
Except that Threads isn’t organized around topics / communities, it’s organized like Insta / Twitter around following people, so there’s no communities to flock to.
Even if there aren’t formal Communities (as in, like Lemmy), there are still communities of people.
… you can still spin up your own instance if you really want to expose yourself to facebook’s social manipulation? Or you could make an account on there.
But you can also show people it’s fine over here by posting links, talking about it, sharing lemmy memes, etc.
Letting them in on any large scale is a losing proposition, as explained in my Original Post ^.^
https://github.com/wescode/lemmy_migrate
For now, this is the best I’ve found. I think there’s work on implementing this in actual lemmy, but this python script is the best option I know of ^.^ - you basically put your user account name and password for each account in a config file and run the script - though I’d make sure to delete the config file after you’ve done it, preferrably with some kind of secure-delete tool like shred on linux.
However, we don’t know yet if lemmy.world will defederate from Meta/FB, until the admins make some kind of statement on it nya. So I wouldn’t leave yet. Though making an account on a smaller instance might be a good idea anyway for simple performance reasons :)
Vote with your dollar means rich people get way more votes.
<insert explanation of the fundamental contradiction vetween capitalism and democracy here> ;p