Conversions: keys and other types

Ok I'm going to post something here I wrote while working on converting Yapeal from old userID ApiKey system to new custom keys. I've only glanced at it over last couple weeks so it's a little outdated maybe on some points but should be interesting anyway and helpful I hope.

Hi all been doing some some thinking about all changes are going to be needed in
Yapeal because of the new custom keys that are coming to the Eve APIs. I also
decided to take a little time and do some brainstorming and see if there are any
other changes that could be made based on my mostly mental list of development
ideas and the list of suggestions I've receive from all of you on things you'd
like to see done with Yapeal as well.

Before getting into the ideas all this thinking has generated I'm going to do a
little review for us of how things work in Yapeal now so I can show how some of
the new ideas maybe possible now that weren't before and how some of the ideas
could greatly change how Yapeal works and it's scalability for large and small
projects while also giving it more versatility and maybe make it easier to use
in a wider range of applications.

Okay let's start with how Yapeal decided which APIs to get each time it runs. It
starts by making a list of 'sections' (account, char, corp, eve, map, server) by
looking for files in class/ with names starting with 'Section'. It then puts
them in a random order so any bugs in any one section that might crash Yapeal
can't end up totally blocking other sections. Using the list it check if the
section has been enabled at the Yapeal wide level by seeing if it's 'active'.
For each active section another list of 'allowed' APIs is made which is passed
to the class in class/Section*.php when it is called. The section class then
makes another 'available' list this time of the per API class files found in
class/api/ where the first part of their names match the current section. The
available list is then used to filter the allowed list to come up with a section
level active API list. At this point the different sections start splitting into
two different groups with account, char, and corp doing some extra stuff and
the eve, map, and server sections just check which APIs from the active list
have expired 'cachedUntil' times and Yapeal should now try to retrieve it from
the Eve API server. The account, char, and corp section also do this last
check but they also have a couple of more things they need to do first.

So the extra stuff that account, char, and corp sections need to do is make yet
another couple of lists of course. The first list is a list of the 'registered'
users (accounts), character, or corporations which are check to see if they are
'active' and then a new active 'owner' list for each user/char/corp is made.
This list is filtered by the section level active list to come up with the list
that is then checked to see if it has an expired 'cachedUntil' time like the
eve, map, and server sections did above.

After re-read the above I've realized I finally written some of the
non-techinical overview documentation that people have always been looking for
and asking where they can find it for a long time. The irony is I write it
without intending to just so I can propose making some changes that could end up
invaliding several parts of it depending on if people like some of the ideas
from below.

Now to understand parts of the ideas I have I will have to get a little more
into the technical side of how Yapeal implements some things in the account,
char, and corp sections but I will try to keep it as non-technical as I can so
the ideas come through hopefully without getting bogged down in all the details
that will be need to make them work.

First I'll go over here how the current API two key system works and how it can
cause many issues for CEOs and Directors in any corporations plus how the new
custom key system tries to improve on it. The current two keys can be thought of
as account level keys since each of them works for all of the characters and the
corporations those characters belong to without separating corporation access
from personal access. The first problem in the two key system that the new
custom key system tries to overcome is separating corporation level access from
personal access to the APIs for CEOs and directors. In the current system if one
of the directors or the CEO of a corporation gives someone their full API key
they are trusting that person with full access to all the APIs for the
corporation as well. They also have to trust people with access to their
personal data like assets etc so the other person could have access to the
corporation API data as well which made some of them unwilling to do so for some
applications that could have been very useful to the whole corp. Now they have
the option to just give someone access to their personal stuff like everyone
else has always had an option to do or just corp data without worrying about
what personal stuff they have also given away to someone. The custom key
system also lets you have more than two keys and you can have very fine grain
control over which APIs each key allows access to for both personal and
corporation keys. One of the things that was tried in the current two key system
to keep CEOs and directors from having to give away everything like above was to
let people with certain roles to also be given access to some of the corp APIs
but it was buggy so you couldn't use titles but always had to assign the roles
directly through the 'wonderful' in game interface that CCP provides. It wasn't
use much in most corporations because of that and other reasons but did allow
for some interesting Meta-gaming chances for people that choose to do so as part
of their game play on unsuspecting corps. The custom API system limits creating
corporation keys to only the CEO and directors which hopefully are a little more
security aware than say the people you give access to a POS so they can
refueling it or have pickup stuff from corp buy orders. So now that I've covered
the main differences between the old and new keys and the main improvements the
custom system gives us I'll next need to get into how Yapeal currently manages
keys and all the account, char, and corp APIs.

First we need to go into a little more detail about the 'registered' database
tables that are used to store and manage the keys plus the account (user), char,
and corp stuff so Yapeal knows how and which APIs to get. To make the following
a little easier to follow I will use the term 'owner' in place of 'user, char,
and/or corp' and probably also in a few places for 'account, char, and/or corp'.
As state above there are several 'registered' tables in the database and in fact
there is one for each type of 'owner'. In the 'user' table is stored the keys
and some info to let you determine which user you are talking about. The 'char'
table mostly has information about the characters but also a link to the
associated user as well. The 'corp' table has information about itself plus a
link to an associated character from 'char' table as well. The reasons they are
setup this way are that all 'owner' APIs require one of the API keys and the
'userID' (link) as parameters in the call to the Eve API server and the 'char'
and 'corp' ones also need a 'charID' (link) parameter. Some of the 'char' and
'corp' APIs also have other required and optional parameters but they aren't of
any concern to us for our needs and will be ignored. Since in the two key system
the keys are account level keys it makes the most sense to store them in the
'user' (account) table. By linking 'char' to 'user' and from 'corp' to 'char'
it's relatively easy to make sure the correct parameters get sent to the Eve API
server when new/updated data is requested. Now that the relationships between
the tables are understood which is the main reason used in determining the
current structure of all the owner tables we'll go over some of the
characteristics those relationships and structures force in how Yapeal works.

The first thing to notice is that both 'char' and 'corp' are dependent on 'user'
which is ok and done for good reasons but does put some limitations on the corp
stuff which isn't really absolutely required by the APIs. To highlight one of
the main problem let me use an example.

Lets say you have the information for the main character on someone's Eve
account in the 'user' and 'char' tables of Yapeal. Lets also have that character
be a director in a player corporation you have in Yapeal as well. So you have
their keys and user information in 'user' and one or more of the characters from
the account in 'char' included the director which links back to 'user' via
'userID'. You also have the corporation info in 'corp' which links to 'char' via
'charID'. Everything works great until one day the player decided he's tired of
doing all the extra work required of him as a director and is going to leave the
corporation to go do something else for a while and have some fun without all
the cares he has as a director. Now there is a problem since his info is needed
to get all the corp data in Yapeal from the Eve APIs. If there is another
director's or the CEO's info registered in Yapeal then it easy to fix by just
changing the link to the different charID and all is well but what if no one
else's info was in Yapeal? Still not hard to solve you just have to add one of
the remaining directors or the CEO to Yapeal and replace the charID info with
the new character's stuff. Does this sound like a mostly manual process to you?
It is by default in Yapeal. It could be automated at the application level but
as far as I know it hasn't been done fully automated because it requires lots of
calls to multiple classes inside Yapeal plus a fair amount of extra code in the
application to do. A system to update the corp info with a new existing
character that is started by an administrator as needed isn't to hard and I'm
sure has been done but automating it to detect these type of changes and doing
it on its own would be very very hard.

Another example would be where someone decides to sell one of their characters
and both the user and corporation info has to be changed. CCP makes this example
a little easier because the characters can't have any roles in the corporation
at the time of transfer I understand but it'll still cause some problems at the
character and user level. Now the changes to Yapeal as I foresee them wouldn't
include a way to directly do any of this either but it should require a lot less
code and should change automating it from a very very hard problem to just a
very hard problem which might be enough for someone to try to do it at least.

Now that we have at least a little better understanding of how the registered
tables work the question becomes how could they be changed to better take
advantage of the new custom keys and all the other changes CCP has made to the
APIs so they work with the custom keys. One thing that has change is the keys
aren't 'owned' by an account any more but by characters. You can still make keys
that work for all the characters on an account but the relationship between the
account (userID) and a key is much less important now and largely gone for all
but the 'all' keys. Yapeal doesn't really care about the 'owner' on a key it's
only really interested in which APIs any one key gives it access to. The only
reason it have to know anything beyond what APIs a key gives access to is so it
can make sure to include the proper 'ownerID' when storing the retrieved data
into the database. The only other place it really needs to associate something
to the key is in the case of 'all' keys because the API server needs to know
which character we are using it for on the 'char' APIs. I was one of several
developers asking CCP to keep the idea of 'all' keys alive because I thought it
would make things easier but now after working with the changed APIs I wonder if
we really need them. Okay since keys can be associated with 1-3 characters or a
corporation/character combination and the unlimited number of keys per available
having the keys stored in the 'user' table doesn't work anymore and moving them
to 'char' does work either. The keys also now have the associate vCode which is
per key. The best thing to do is have a new 'registered' table that stores the
keys and their vCode together. There also is a new API to find out additional
information about the keys account/APIKeyInfo which lets us know things like
which type of key it is 'Character'|'Corporation' and you can figure out
indirectly from it if the key is an 'all' key as well but currently not get a
full list of all the characters associated with them.